[R] %dopar% parallel processing experiment

[Uwe Ligges]

On 02.07.2011 19:32, ivo welch wrote:
> dear R experts---
>
> I am experimenting with multicore processing, so far with pretty
> disappointing results.  Here is my simple example:
>
> A<- 100000
> randvalues<- abs(rnorm(A))
> minfn<- function( x, i ) { log(abs(x))+x^3+i/A+randvalues[i] }  ## an
> arbitrary function
>
> ARGV<- commandArgs(trailingOnly=TRUE)
>
> if (ARGV[1] == "do-onecore") {
>     library(foreach)
>     discard<- foreach(i = 1:A) %do% uniroot( minfn, c(1e-20,9e20), i ) } else
> if (ARGV[1] == "do-multicore") {
>     library(doMC)
>     registerDoMC()
>     cat("You have", getDoParWorkers(), "cores\n")
>     discard<- foreach(i = 1:A) %dopar% uniroot( minfn, c(1e-20,9e20), i ) } else
> if (ARGV[1] == "plain")
>     for (i in 1:A) discard<- uniroot( minfn, c(1e-20,9e20), i ) else
> cat("sorry, but argument", ARGV[1], "is not plain|do-onecore|do-multicore\n")
>
>
> on my Mac Pro 3,1 (2 quad-cores), R 2.12.0, which reports 8 cores,
>
>    "plain" takes about 68 seconds (real and user, using the unix timing
> function).
>    "do-onecore" takes about 300 seconds.
>    "do-multicore" takes about 210 seconds real, (300 seconds user).
>
> this seems pretty disappointing.  the cores are not used for the most
> part, either.  feedback appreciated.


Feedback is that a single computation within your foreach loop is so 
quick that the overhead of communicating data and results between 
processes costs more time than the actual evaluation, hence you are 
faster with a single process.

What you should do is:

write code that does, e.g., 10000 iterations within 10 other iterations 
and just do a foreach loop around the outer 10. Then you will probably 
be much faster (without testing). But this is essentially the example I 
am using for teaching to show when not to do parallel processing.....

Best,
Uwe Ligges






> /iaw
>
>
> ----
> Ivo Welch (ivo.welch at gmail.com)
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.