[R] Snow parLapply

[Sean Davis]

On 11/11/05 6:48 AM, "Florent Baty" <florent.baty at unibas.ch> wrote:

> Dear R-user,
> 
> I am trying to use the function 'parLapply' from the 'snow' package
> which is supposed to work the same wys as 'lapply' but for a
> parallelized cluster of computers. The function I am trying to call in
> parallel is 'dudi.pca' (from the 'ade4' package) which performs
> principal component analyses. When I call this function on a list of
> dataframes with the regular lapply function it works correctly. If use
> the 'parLapply' there is an error message.
> 
> Example:
>> library("snow")
>> library(ade4)
>> a <- matrix(rnorm(2500),50)
>> b <- matrix(rnorm(10^4),10^2)
>> l1 <- list(a=a,b=b)
>> mycluster <- makeCluster(2,type="MPI")
> Loading required package: Rmpi
> 
>       Rmpi version: 0.4-9
>       Rmpi is an interface (wrapper) to MPI APIs
>       with interactive R slave functionalities.
>       See `library (help=Rmpi)' for details.
>       2 slaves are spawned successfully. 0 failed.
>> lapply(list(a=as.data.frame(a),b=as.data.frame(b)),dudi.pca,scannf=F)
> $a
> Duality diagramm
> class: pca dudi
> $call: FUN(df = X[[1]], scannf = ..1)
> ..
> 
> $b
> Duality diagramm
> class: pca dudi
> $call: FUN(df = X[[2]], scannf = ..1)
> 
> ..
> 
>> 
> parLapply(mycluster,list(a=as.data.frame(a),b=as.data.frame(b)),dudi.pca,scann
> f=F)
> [1] "Error in FUN(X[[1]], ...) : couldn't find function \"as.dudi\"\n"
> [2] "Error in FUN(X[[1]], ...) : couldn't find function \"as.dudi\"\n"
> 
> 
> On the other hand, if I call 'parLapply' with the function 'princomp'
> (which also performs PCA) everything works fine.
> 
> Example:
> 
>> parLapply(mycluster,list(a=as.data.frame(a),b=as.data.frame(b)),princomp)
> $a
> Call:
> princomp(x = X[[1]])
> ..
> 
> $b
> Call:
> princomp(x = X[[1]])
> ..
> 
> Does anybody knows why 'parLapply' does not work correctly with some
> functions?

Florent,

Keep in mind how snow and parLapply work.  They start slaves on each node.
Each slave has its own workspace, and that workspace doesn't contain
dudi.pca unless you tell it to.  Therefore, you will have to do:

clusterEvalQ(mycluster,library(ade4))

To load the ade4 library in all the slaves.  After that, I think your code
will behave as expected.  I highly recommend this website for learning the
basics of snow:

http://www.sfu.ca/~sblay/R/snow.html

Hope that helps.

Sean