[R] multicore and mclapply problem in calculation server

[p_connolly]

On Wed, 18-Dec-2013 at 09:28AM +0100, Juan Antonio Balbuena wrote:

|>
|>    Hello
|>    I am using package multicore for parallel computing in a Altix 
UltraViolet
|>    1000 server with 64 CPUs and 960 GB of RAM memory. Access is 
managed by
|>    means of a SGE queue system. This is the first time I am using 
parallel
|>    computing and my experience with supercomputers is quite limited. 
So any
|>    help will be much, much appreciated.
|>    My experiment consists of a number of runs (N.runs) each involving 
a number
|>    of permutations (N. perms). (An excerpt of the code is included 
below.) The
|>    permutations are very time consuming and I am using mclapply to 
distribute
|>    the job among a given number of cpus (usually 12 to 24). The 
problem is that
|>    the system administrators notice that threads keep increasing as 
the program
|>    is executed to the point that they compromise the functioning of 
the whole
|>    system and have to abort the job.

So, does that mean you get nothing written to 1MH_30.tre?  I'd be
surprised if you did get anything.  Though there's lots of stuff
happening I know nothing about, I'm pretty sure there's an issue with
your wrapper() function.

[...]

|>      wrapper <- function (x) {           # THIS FUCNTION IS SUPPOSED 
TO BE
|>    PARALLELIZED (SEE BELOW)
|>        treeH <- read.tree(text=linH[x])
|>        treeP <- read.tree(text=linP[x])
|>        mrcaL <- MRCALink.simul (treeH, treeP, HP)
|>        stat.matrix[x,] <- three.stat(mrcaL)
|>        }

Nothing is being returned, so your calls to rbind will have nothing to
put together.  So what they end up trying to do, I've no idea.  That
function probably needs a final line 'stat.matrix' before the '}'

One thing I discovered with mclapply is that to use the browser()
function, it's necessary to make what is x in your example of length 1
(and probably mc.cores to 1 also) before it's possible to examine
what's happening at various parts of the function being debugged.
Judicious use of cat() and Sys.time() to display what's happening at
various stages is also helpful.

HTH

--
~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.
    ___    Patrick Connolly
  {~._.~}                         Great minds discuss ideas
  _( Y )_                        Middle minds discuss events
(:_~*~_:)                        Small minds discuss people
  (_)-(_)                                   ..... Anon

~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.~.