[R-sig-hpc] load does not work for me with foreach and %dopar%

[Brian G. Peterson]

Samo,

I think you have a basic misunderstanding of how the parallel execution
works.

In basically any parallel backend for R, you're creating separate
environments (and R processes) in which execution is taking place.

Your example loads x,y,z inside the foreach construct, and then tries to
print them outside the construct from the master  (calling) R process.
The only thing returned from foreach is what you choose to return.
Otherwise, the master process knows nothing of what happened in each of
the workers.


If you put something like

     print(get(objectsNamesVector[i]))

inside your foreach construct, I think you would get the same results
with both %do% and %dopar%

you would, however,  *not* get objects called x,y,z in your
calling/master process with %dopar%. 

Regards,

   - Brian

On Tue, 2011-07-26 at 15:40 +0200, Samo Pahor wrote:
> Hello HPF experts.
> 
> I have a very basic/beginner question regarding using of load, foreach,
> %dopar% and doSMP.
> 
> I encountered some issues with using foreach %dopar% when loading objects
> from disk into memory... Objects are not loaded when I try to load them when
> using foreach %dopar% (it works when I use only %do%) Below is a simple
> example that shows my problem.
> 
> require(doSMP)
> workers <- startWorkers(4)
> registerDoSMP(workers)
> 
> envir = .GlobalEnv
> 
> x <- "X test"
> y <- "Y test"
> z <- "Z test"
> 
> save(x, file="x.RData")
> save(y, file="y.RData")
> save(z, file="z.RData")
> 
> rm(x)
> rm(y)
> rm(z)
> 
> objectsNamesVector <- c("x", "y", "z")
> 
> foreach(i=1:length(objectsNamesVector), .combine=function (...) NULL,
>   .multicombine=TRUE) %do% {
>     print(paste("Loading object ", objectsNamesVector[i]," - ", i, "
> of ",    length(objectsNamesVector), sep=""))
>     load(file=paste(objectsNamesVector[i], ".RData", sep=""), envir=envir)
> }
> 
> print(x)
> print(y)
> print(z)
> 
> rm(x)
> rm(y)
> rm(z)
> 
> foreach(i=1:length(objectsNamesVector), .combine=function (...) NULL,
> .multicombine=TRUE) %dopar% {
>     print(paste("Loading object ", objectsNamesVector[i]," - ", i, "
> of ", length(objectsNamesVector), sep=""))
>     load(file=paste(objectsNamesVector[i], ".RData", sep=""), envir=envir)
> }
> 
> print(x)
> print(y)
> print(z)
> 
> Result of executing this code is(without the ">"):
> 
> envir = .GlobalEnv
> 
> x <- "X test"
> y <- "Y test"
> z <- "Z test"
> 
> save(x, file="x.RData")
> save(y, file="y.RData")
> save(z, file="z.RData")
> 
> rm(x)
> rm(y)
> rm(z)
> 
> objectsNamesVector <- c("x", "y", "z")
> 
> foreach(i=1:length(objectsNamesVector), .combine=function (...) NULL,
>   .multicombine=TRUE) %do% {
> +   print(paste("Loading object ", objectsNamesVector[i]," - ", i, "
> of ", length(objectsNamesVector), sep=""))
> +   load(file=paste(objectsNamesVector[i], ".RData", sep=""), envir=envir)
> + }
> [1] "Loading object x - 1 of 3"
> [1] "Loading object y - 2 of 3"
> [1] "Loading object z - 3 of 3"
> NULL
> 
> print(x)
> [1] "X test"
> print(y)
> [1] "Y test"
> print(z)
> [1] "Z test"
> rm(x)
> rm(y)
> rm(z)
> 
> foreach(i=1:length(objectsNamesVector), .combine=function (...) NULL,
> .multicombine=TRUE) %dopar% {
> +   print(paste("Loading object ", objectsNamesVector[i]," - ", i, "
> of ", length(objectsNamesVector), sep=""))
> +   load(file=paste(objectsNamesVector[i], ".RData", sep=""), envir=envir)
> + }
> NULL
> 
> print(x)
> Error in print(x) : object 'x' not found
> print(y)
> Error in print(y) : object 'y' not found
> print(z)
> Error in print(z) : object 'z' not found
> 
> I understand that I cannot improve IO with foreach since IO is sequential on
> my architecture. I would just like to understand why this load is not
> working... Because I would then like to do more processing after I
> successfully load the objects.
> 
> Thank you for your answer and/or pointer to documetation where I can
> understand this issue.
> 
> Regards, Samo.
> 
> P.S.: I asked this question a few days ago on stackovwerflow (
> http://stackoverflow.com/questions/6804229/load-does-not-work-with-foreach-and-dopar)
> but
> received no answer. Therefore I asked once again here.
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc

-- 
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock