Dear Dr. Yu,

Good suggestion.

After I double check my system, the MPI library actually is not 64 bits and
the integer pointer size is 4.......So, I went back to check my code that I
take a
unique() call on the object after bcast call ... The results show that the
buffer is not
really overwrote for all memory (00 and 01 occur on rank 1 which should all
00.)
This means XLENGTH works, but MPI did not send right amount of data.
After I upgrade to 64 bits MPI library (interger pointer size is 8) and
R-3.0.1,
then I got

  long vectors not supported yet: memory.c:3100

In short, the previous pbdMPI result is not correct. Need more study to
figure
out how to send such a long object in once.

No, I did not change any further code. I also tested both Rmpi and pbdMPI
in SPMD,
and I got the same memory error above for both 2147484070 or 6e9.

Sincerely,
Wei-Chen Chen


On Fri, Aug 9, 2013 at 11:34 PM, Hao Yu <hyu@stats.uwo.ca> wrote:

> Hi Wei Chen,
>
> Somehow I cannot make Rmpi use 64bit integer. I recompiled openmpi 1.6.5
> according to
>
> http://wiki.chem.vu.nl/dirac/index.php/How_to_build_MPI_libraries_for_64-bit_integers
>
> So I have openmpi with 64bit integer by default. After recompiling Rmpi, I
> still get
> > library(Rmpi)
> > mpi.spawn.Rslaves(nslaves=2)
>         2 slaves are spawned successfully. 0 failed.
> master (rank 0, comm 1) of size 3 is running on: cloud
> slave1 (rank 1, comm 1) of size 3 is running on: cloud
> slave2 (rank 2, comm 1) of size 3 is running on: cloud
> > n=2^31/4+100
> > x=1:n
> > a=serialize(x,NULL)
> > mpi.bcast.cmd(a<-mpi.bcast(raw(2147484070), 4))
> > invisible(mpi.bcast(a,4))
> MPI_ERR_COUNT: invalid count argument
>
> Clearly mpi thinks 2147484070 (length of a) is over 2^31-1. Notice that I
> have in Rmpi.c
> MPI_Bcast(RAW(sexp_data), XLENGTH(sexp_data), MPI_BYTE, rank, comm[commn]);
>
> I also add the CFLAGS -m64 to compile Rmpi without success.
>
> Anyway I haven't figured out yet. After changing from LENGTH to XLENGTH,
> do you change something else? Notice R uses R_xlen_t for XLENGTH.
>
> Hao
>
> Wei-Chen Chen wrote:
> > Dear Dr. Yu,
> >
> > This probably is not true.
> >
> > MPI-2 did support 64 bits integer as long as the library is compiled with
> > 64 bits compiler. Change to XLENGTH is actually a good idea.
> > Thank you, Simon!
> >
> > This is what I did for pbdMPI and will be in next release.
> >
> > 1. I replace all LENGTH by XLENGTH for all src/*.c
> > 2. I add the next to some common header file
> >
> > #ifndef XLENGTH
> > #define XLENGTH LENGTH
> > #endif
> >
> > So, windows would not be a problem to compile for both 32 and 64 bits
> > system.
> >
> > I test the attached code with OpenMPI 1.4.2 in two processors which have
> > 128GB memory. The followed outputs look good to me except take a while to
> > finish. Let me know if the code or outputs are incorrect.
> >
> > Finally, the only issue to me is do NOT send and wait such a big object
> > between processors, but let processors read their own portion.
> >
> > Everybody is welcomed to the pbdR world.
> >
> > Sincerely,
> > Wei-Chen Chen
> >
> >
> > ### Code
> > library(pbdMPI, quiet = TRUE)
> > init()
> > n <- 6e9
> > a <- rep(comm.rank(), length = n)
> > a <- as.raw(a)
> > comm.print(a[1:5], all.rank = TRUE)
> > b <- length(a)
> > comm.print(b, all.rank = TRUE)
> > a <- spmd.bcast.raw(a)
> > comm.print(a[1:5], all.rank = TRUE)
> > b <- length(a)
> > comm.print(b, all.rank = TRUE)
> > finalize()
> >
> >
> > ### Output
> > Fri Aug  9 13:22:43 EDT 2013
> >
> >     High Performance Computing Resource is available via
> >     ``http://www.nics.tennessee.edu/getting-an-allocation''
> >     with XSEDE at ``http://www.xsede.org/overview''.
> >
> > COMM.RANK = 0
> > [1] 00 00 00 00 00
> > COMM.RANK = 1
> > [1] 01 01 01 01 01
> > COMM.RANK = 0
> > [1] 6e+09
> > COMM.RANK = 1
> > [1] 6e+09
> > COMM.RANK = 0
> > [1] 00 00 00 00 00
> > COMM.RANK = 1
> > [1] 00 00 00 00 00
> > COMM.RANK = 0
> > [1] 6e+09
> > COMM.RANK = 1
> > [1] 6e+09
> > Fri Aug  9 13:23:27 EDT 2013
> >
> >
> >
> >
> > On Thu, Aug 8, 2013 at 9:53 AM, Hao Yu <hyu@stats.uwo.ca> wrote:
> >
> >> To all,
> >>
> >> Look like the max buffer size in any MPI transfer is restricted to
> >> 2^31-1.
> >> Whether XLENGTH is used or not will not change it. This is the
> >> restriction
> >> of MPI-2. MPI-3 will address some of those issues. Until then we either
> >> split the data in R before transferring or define a user-defined type
> >> (e.g. a contiguous type like 10 double as one unit so we can transfer
> >> max
> >> 10*(2^31-1) elements of double).
> >>
> >> Hao
> >>
> >> Simon Urbanek wrote:
> >> > Jim,
> >> >
> >> > On Aug 7, 2013, at 11:59 AM, Jim Gattiker <j.gattiker@gmail.com>
> >> wrote:
> >> >
> >> >> To bcast.Robj, Rmpi uses serialize() to pack the object into "raw",
> >> >> which
> >> >> operates as a vector of bytes. R supports vectors up to 2^31
> >> elements.
> >> >
> >> > that is not true. R supports vectors up to 2^52 elements. That is way
> >> > beyond current RAM sizes and certainly more than 2^31:
> >> >
> >> >> n=6e9
> >> >> x=integer(n)
> >> >> a=serialize(x,NULL)
> >> >> length(a)
> >> > [1] 2.4e+10
> >> >> log2(length(a))
> >> > [1] 34.48232
> >> >
> >> > However Rmpi does not. You have to use XLENGTH and R_xlen_t in the C
> >> code
> >> > if you want to go beyond 2^31.
> >> >
> >> > Cheers,
> >> > Simon
> >> >
> >> >
> >> >> Your
> >> >> object, serialized, is close to that. I'm a little puzzled though,
> >> >> 350x350000 matrix works for me; perhaps there's more to your actual
> >> >> call.
> >> >>
> >> >> Setting up the slave environment explicitly seems to me to be better
> >> >> practice, i.e. using mpi.bcast.Robj2Slave and related calls, then the
> >> >> applyLB() doesn't contain the data. As I'm reading it, I think the
> >> way
> >> >> this
> >> >> is set up currently will send the data to a slave for each
> >> application
> >> >> of
> >> >> the apply.
> >> >>
> >> >> If you're facing larger data sizes, a solution is to explicitly cut
> >> up
> >> >> the
> >> >> object, mpi.bcast.Robj2Slave the pieces in turn, and then use an
> >> >> mpi.bcast.cmd to collect them together. Another approach would be to
> >> >> write
> >> >> to data to a file, and direct the slaves to read it into their
> >> >> environments
> >> >> with an mpi.bcast.cmd.
> >> >>
> >> >> As a comment: I can't tell your application from the code, but if the
> >> >> slaves are to be working each on only part of the data, it's better
> >> to
> >> >> send
> >> >> the slave just the part of the data it needs.
> >> >>
> >> >>    cheers,
> >> >>       jim
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Aug 6, 2013 at 6:37 PM, Ei-ji Nakama <nakama@ki.rim.or.jp>
> >> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> <WARNING>
> >> >>> It is not yet completed...
> >> >>> </WARNING>
> >> >>> http://prs.ism.ac.jp/~nakama/Rhpc/
> >> >>>
> >> >>> 2013/8/7, Xiaochun Sun <xiaoch.sun@gmail.com>:
> >> >>>> The same code worked fine for smaller dataset, such as 350x25000.
> >> Does
> >> >>> that
> >> >>>> mean the current Rmpi don't allow long vectors to be passed to
> >> slaves?
> >> >>> Any
> >> >>>> idea on that or any alternatives to Rmpi?
> >> >>>
> >> >>> Its can treat the considerably big data.
> >> >>>
> >> >>> --
> >> >>> EI-JI Nakama  <nakama (a) ki.rim.or.jp>
> >> >>> "\u4e2d\u9593\u6804\u6cbb"  <nakama (a) ki.rim.or.jp>
> >> >>>
> >> >>> _______________________________________________
> >> >>> R-sig-hpc mailing list
> >> >>> R-sig-hpc@r-project.org
> >> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >> >>>
> >> >>
> >> >>      [[alternative HTML version deleted]]
> >> >>
> >> >> _______________________________________________
> >> >> R-sig-hpc mailing list
> >> >> R-sig-hpc@r-project.org
> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >> >>
> >> >>
> >> >
> >> > _______________________________________________
> >> > R-sig-hpc mailing list
> >> > R-sig-hpc@r-project.org
> >> > https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >> >
> >>
> >>
> >> --
> >> Department of Statistics & Actuarial Sciences
> >> Office Phone#:(519)-661-3622
> >> Fax Phone#:(519)-661-3813
> >> The University of Western Ontario
> >> London, Ontario N6A 5B7
> >> http://www.stats.uwo.ca/yu
> >>
> >> _______________________________________________
> >> R-sig-hpc mailing list
> >> R-sig-hpc@r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
> >>
> >
>
>
> --
> Department of Statistics & Actuarial Sciences
> Office Phone#:(519)-661-3622
> Fax Phone#:(519)-661-3813
> The University of Western Ontario
> London, Ontario N6A 5B7
> http://www.stats.uwo.ca/yu
>

	[[alternative HTML version deleted]]

