[R-pkg-devel] parallel computing slower than sequential computing

[Wang, Zhu]

Hello,

I have a package where parallel computing is slower than sequential computing with cross-validation. For instance, please see the following reproducible example with 10 cores and the relevant lines in the corresponding R function. The R package is also attached. I thought parallel computing should improve the speed. However, it is not obvious to me where it went wrong.

Many thanks in advance.

Zhu Wang

library("mpath")
library("pscl")
data("bioChemists", package = "pscl")
>system.time(cv.zipath(art ~ . | ., data = bioChemists, family = "negbin",     nlambda=100, parallel=FALSE))
    user  system elapsed
  77.430   0.031  79.547
> system.time(cv.zipath(art ~ . | ., data = bioChemists, family = "negbin",     nlambda=100, parallel=TRUE, n.cores=10))
    user  system elapsed
  95.694   0.517 106.072
R> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.2 LTS

The relevant lines in cv.zipath regarding parallel argument are below:

if(parallel){
     registerDoParallel(cores=n.cores)
     i <- 1  ###needed to pass R CMD check with parallel code below
     residmat <- foreach(i=seq(K), .combine=cbind, .packages='mpath') %dopar% {
         omit <- all.folds[[i]]
         fitcv <- do.call("zipath", list(formula, data[-omit,], weights[-omit],  lambda.count=lambda.count, lambda.zero=lambda.zero, nlambda=nlambda, ...))
         logLik(fitcv, newdata=data[omit,, drop=FALSE], Y[omit],                 weights=weights[omit])
     }
     stopImplicitCluster()
     }
     else{
      residmat <- matrix(NA, nlambda, K)
      for(i in seq(K)) {
        if(trace)
          cat("\n CV Fold", i, "\n\n")
        omit <- all.folds[[i]]
        fitcv <- do.call("zipath", list(formula, data[-omit,], weights[-omit],   lambda.count=lambda.count, lambda.zero=lambda.zero, nlambda=nlambda, ...))
        residmat[, i] <- logLik(fitcv, newdata=data[omit,, drop=FALSE],          Y[omit], weights=weights[omit])
      }
     }

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mpath_0.3-13.tar.gz
Type: application/gzip
Size: 83361 bytes
Desc: mpath_0.3-13.tar.gz
URL: <https://stat.ethz.ch/pipermail/r-package-devel/attachments/20190429/c9fc37a1/attachment.gz>