Hao:

I have the impression MPI's limitation is 2^31 elements, and elements can
be large (limited by buffer sizes?), although I haven't tested that.
Perhaps registering a type of 1kB and 'chunking' can be enabled behind the
scenes.
  Fixing the ragged end might not cost any real performance, probably the
communications buffers are that large or larger anyway(?)

   OTOH, it's not fast to bcast a GB anyway, so the speed of explicitly
breaking/assembling buffers in the R environment isn't much of a real
concern. The real downside is more copies of the data in system mem, which
might be an issue when this much storage is involved.

   As an observation, working with more than a GB in a PE is the province
of high-memory nodes or small clusters. Large HPC is in the couple GB/PE
range, and my recollection of projections is that will be going down (as
core count rises faster than total memory). How to address this might
depend on the target application; using filesystem to buffer large data
transfers might make sense for relatively small numbers of PEs.

  --j

On Thu, Aug 8, 2013 at 7:53 AM, Hao Yu <hyu@stats.uwo.ca> wrote:

> To all,
>
> Look like the max buffer size in any MPI transfer is restricted to 2^31-1.
> Whether XLENGTH is used or not will not change it. This is the restriction
> of MPI-2. MPI-3 will address some of those issues. Until then we either
> split the data in R before transferring or define a user-defined type
> (e.g. a contiguous type like 10 double as one unit so we can transfer max
> 10*(2^31-1) elements of double).
>
> Hao
>
>

	[[alternative HTML version deleted]]

