Looking for advice on parallel techniques for R.
 
I am presently working with parallelizing R code from a package we use locally for analysis.
The multicore library routines parallel(), and mclapply() give mixed results.
For starters, 2 to 8 cores (1 processor) are available only.
 
Setting up an interative for(i=1,N) loop with a function ff() where ff() accepts some matrixes and does some number crunching,
parallel() creates as many R threads as N and leaves them around until collect() reads the pids, gets the returned data values and deposits them.  A cleanup of the threads ensues.
mclapply() creates and deletes R threads dynamically only as many as #cores, returning values which continually build inside the new vector.
With large values of N, parallel() causes problems as there are too many threads to manage and it cant pipeline only a few at a time, so all N are running in parallel, instead of #cores or something in between, - if N = 500, the system slows to a where it has to be rebooted basically.  
With large values of N, mclapply() runs to completion ok, but it drags out the amount of system time (from system.time()) needed since it is continually managing thread create/kill/data exchange, so again its not efficient.  Total elapsed time with mclapply() becomes more not less.
 
Does anyone know how to get around this (without changing a huge amount of R code) ?
I have started looking at snowfall, Rparallel, etc to see if there are better ways of managing the threads, but if anyone on the list has had some experience with this, it would be great to learn how to throttle the thread management or other way of speeding things up.  Thanks greatly.
  



CONFIDENTIALITY NOTICE:  This email and any attachments ...{{dropped:26}}

