[R-SIG-Mac] parLapply is not using all nodes

[John Magnotti]

Hi Yan,

Sorry it wasn't the simple answer. On my machine, your code creates 3 R
processes (#cores -1) and they are all active.

If you have an even simpler example say

parLapply(cl, 1:8, function_that_takes_a_while)

does that get all the cores going?

John

On Fri, Dec 11, 2015 at 6:58 AM, ALPEROVYCH Yan <ALPEROVYCH at em-lyon.com>
wrote:

> Hi John,
>
> Thank you for the reply. In fact my original list has 122^2 elements in it
> (I simplified the code here), and parLapply behaves in a similar way still.
> Can it be something else?
>
> Yan
>
> On Dec 11, 2015, at 1:51 PM, John Magnotti <john.magnotti at gmail.com
> <mailto:john.magnotti at gmail.com>> wrote:
>
> Hello Yan,
>
> I think parLapply is just assigning a core for every item in the list, G,
> you supplied. Because you have more cores than items in the list, some of
> the cores won't receive any work.
>
>
> John
>
> On Fri, Dec 11, 2015 at 4:00 AM, ALPEROVYCH Yan <ALPEROVYCH at em-lyon.com
> <mailto:ALPEROVYCH at em-lyon.com>> wrote:
> Hello,
>
> I have a piece of code that needs parallelization and it used to work just
> fine before (about 6 months ago). However, I had to rerun it yesterday and
> found out that my code is now behaving in a weird way - not all worker
> processes are charged with the computation. I created a little code that
> allows reproducing the issue. Here is a snapshot of the top command:
>
> PID  COMMAND    %CPU    TIME
> 872  R                          97.7    00:02.45
> 871  R                          0.0     00:00.03
> 870  R                          98.2    00:02.93
> 869  R                          0.0     00:00.03
> 868  R                          97.7    00:02.46
> 867  R                          0.0     00:00.03
> 866  R                          94.4    00:02.36
> 862  R                          1.0             00:04.64
>
> Interestingly, the mclapply command seems to correctly charge all workers
> instantly.
>
> So is this an intended behavior?
>
> Example that reproduces the issue on my machine:
> #
> rm(list = ls())
> library(parallel)
> set.seed(123)
> #
> expr <- expression(pnorm(b0 + b1*x1 + b2*x2 + b3*x3))
> G <- list(
>         D(D(expr, "x2"), "b0"),
>         D(D(expr, "x2"), "b1"),
>         D(D(expr, "x2"), "b2"),
>         D(D(expr, "x2"), "b3"))
> #
> b0 <-  1.2
> b1 <-  0.4
> b2 <-  0.2
> b3 <- -0.6
> x1 <- rnorm(10^7)
> x2 <- rnorm(10^7)
> x3 <- rnorm(10^7)
> #
> nc <- detectCores() - 1
> cl <- makeCluster(nc, type = "FORK")
> grad_g <- parLapply(cl, G, function(Z) lapply(Z, function(x)
> mean(eval(x))))
> stopCluster(cl)
> #
> sessionInfo()
> R version 3.2.3 (2015-12-10)
> Platform: x86_64-apple-darwin13.4.0 (64-bit)
> Running under: OS X 10.11.2 (El Capitan)
>
> locale:
> [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
>
> attached base packages:
> [1] parallel  stats     graphics  grDevices utils     datasets  methods
>  base
>
>
>
>
> ----
> Ce message electronique et tous les fichiers attaches qu'il contient sont
> confidentiels et destines exclusivement à l'usage de la personne à laquelle
> ils sont adresses. Si vous avez reçu ce message par erreur, merci de le
> retourner à son metteur. Les idees et opinions presentees dans ce message
> sont celles de son auteur, et ne representent pas necessairement celles de
> l'institution ou entite affiliee dont l'auteur est l'employe. La
> publication, l'usage, la distribution, l'impression ou la copie non
> autorisee de ce message et des attachements qu'il contient sont strictement
> interdits.
>
> This email and any files transmitted with it are confide...{{dropped:10}}
>
> _______________________________________________
> R-SIG-Mac mailing list
> R-SIG-Mac at r-project.org<mailto:R-SIG-Mac at r-project.org>
> https://stat.ethz.ch/mailman/listinfo/r-sig-mac
>
>
> ----
> Ce message electronique et tous les fichiers attaches qu'il contient sont
> confidentiels et destines exclusivement à l'usage de la personne à laquelle
> ils sont adresses. Si vous avez reçu ce message par erreur, merci de le
> retourner à son metteur. Les idees et opinions presentees dans ce message
> sont celles de son auteur, et ne representent pas necessairement celles de
> l'institution ou entite affiliee dont l'auteur est l'employe. La
> publication, l'usage, la distribution, l'impression ou la copie non
> autorisee de ce message et des attachements qu'il contient sont strictement
> interdits.
>
> This email and any files transmitted with it are confi...{{dropped:15}}