[R] multicore and mclapply problem in calculation server

[Juan Antonio Balbuena]

   Hi Patrick,
   Thanks for posting. The wrapper function works as it should. If you do not
   specify return the function returns the last object declared, and this is
   exactly what I intended. Doing some research on my own I found out that the
   problem was here:
   x <- c(1: length(linH))
   stat.matrix <- do.call(rbind, mclapply(x, wrapper, mc.cores= 6))
   Since  linH  =  readLines(conH,  n= N.perm) and N.perm was set to 999,
   apparently  mclapply  sends  _simultaneously_ the 999 jobs to the cpus
   specified in mc.cores. Even if you use more than 6, as in the example, the
   systems gets quickly overflowed. So the solution is to split length(linH)
   into blocks equal to the number of cpus and the message to end-users is "be
   careful with parallel processing functions in R".
   Cheers
   Juan Antonio Balbuena
   El 19/12/2013 22:03, p_connolly escribió:

     On Wed, 18-Dec-2013 at 09:28AM +0100, Juan Antonio Balbuena wrote:
     |>
     |>    Hello
     |>    I am using package multicore for parallel computing in a Altix
     UltraViolet
     |>    1000 server with 64 CPUs and 960 GB of RAM memory. Access is managed
     by
     |>    means of a SGE queue system. This is the first time I am using
     parallel
     |>    computing and my experience with supercomputers is quite limited. So
     any
     |>    help will be much, much appreciated.
     |>    My experiment consists of a number of runs (N.runs) each involving a
     number
     |>    of permutations (N. perms). (An excerpt of the code is included
     below.) The
     |>    permutations are very time consuming and I am using mclapply to
     distribute
     |>    the job among a given number of cpus (usually 12 to 24). The problem
     is that
     |>    the system administrators notice that threads keep increasing as the
     program
     |>    is executed to the point that they compromise the functioning of the
     whole
     |>    system and have to abort the job.
     So, does that mean you get nothing written to 1MH_30.tre?  I'd be
     surprised if you did get anything.  Though there's lots of stuff
     happening I know nothing about, I'm pretty sure there's an issue with
     your wrapper() function.
     [...]
     |>      wrapper <- function (x) {           # THIS FUCNTION IS SUPPOSED TO
     BE
     |>    PARALLELIZED (SEE BELOW)
     |>        treeH <- read.tree(text=linH[x])
     |>        treeP <- read.tree(text=linP[x])
     |>        mrcaL <- MRCALink.simul (treeH, treeP, HP)
     |>        stat.matrix[x,] <- three.stat(mrcaL)
     |>        }
     Nothing is being returned, so your calls to rbind will have nothing to
     put together.  So what they end up trying to do, I've no idea.  That
     function probably needs a final line 'stat.matrix' before the '}'
     One thing I discovered with mclapply is that to use the browser()
     function, it's necessary to make what is x in your example of length 1
     (and probably mc.cores to 1 also) before it's possible to examine
     what's happening at various parts of the function being debugged.
     Judicious use of cat() and Sys.time() to display what's happening at
     various stages is also helpful.
     HTH
     --
     ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
        ___    Patrick Connolly
      {~._.~}                         Great minds discuss ideas
      _( Y )_                        Middle minds discuss events
     (:_~*~_:)                        Small minds discuss people
      (_)-(_)                                   ..... Anon
     ~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.

   --

   Dr. Juan A. Balbuena
   Marine Zoology Unit
   Cavanilles Institute of Biodiversity and Evolutionary Biology
   University of
   Valencia
   [1]http://www.uv.es/~balbuena
   P.O. Box 22085
   [2]http://www.uv.es/cavanilles/zoomarin/index.htm
   46071 Valencia, Spain
   [3]http://cetus.uv.es/mullpardb/index.html
   e-mail: [4]j.a.balbuena at uv.es    tel. +34 963 543 658    fax +34 963 543 733
   ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
   NOTE! For shipments by EXPRESS COURIER use the following street address:
   C/ Catedrático José Beltrán 2, 46980 Paterna (Valencia), Spain.
   ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

References

   1. http://www.uv.es/%7Ebalbuena
   2. http://www.uv.es/cavanilles/zoomarin/index.htm
   3. http://cetus.uv.es/mullpardb/index.html
   4. mailto:j.a.balbuena at uv.es