[R] Parallel package guidance needed

[Therneau, Terry M., Ph.D.]

I have a process that I need to parallelize, and have a question about two
different ways to proceed.  It is essentially an MCMC exploration where
the likelihood is a sum over subjects (6000 of them), and the per-subject
computation is the slow part.

Here is a rough schematic of the code using one approach:

mymc <- function(formula, data, subset, na.action,  id, etc) {
     # lots of setup, long but computationally quick

     hlog <- function(thisid, param) {
        # compute the loglik for this subject
       ...
       }

    uid <- unique(id)  # multiple data rows for each subject

    for (i in 1:burnin) {
       param <- get_next_proposal()
       loglist <- mclapply(uid, hlog, param=param)
       loglik <- sum(unlist(loglist))
       # process result
       }

   # Now the non-burnin MCMC iterations
  …
}

The second approach is to put cluster formation outside the loop, e.g.,

 ...
 clust <- makeForkCluster()
 for (i in 1:burnin) {
     param <- get_next_proposal()
     loglist <- parLapply(clust, uid, hlog, param=param)
     loglik <- sum(unlist(loglist))
     # process result
     }

   # rest of the code

   stopCluster(clust)

------------------

On the face of it, the second looks like it "could" be more efficient since it
only starts and stops the subprocesses once.  A short trial on one of our
cluster servers seems to say the opposite.  The load average on a quiet machine
never gets much over 5-6  using method 2, and in the 20s for method 1
(detectCores() =80 on the box, we used mc.cores=50).  Wall time for method 2
is looking to be several hours.

Any pointers to documentation/discussion at this level would be much appreciated.  I'm going to be fitting a lot of models.

Terry T.


	[[alternative HTML version deleted]]