[R-sig-hpc] help with a scoping issue with foreach

[Stephen Weston]

Roman,

When %dopar% is executed with certain parallel backends, it analyzes
the right hand expression to determine if there are any variables that need
to be exported.  If a symbol is referenced in the expression that is defined
in the enclosing environment, it will export that variable to the parallel
workers.  If it is defined in the global environment, or in some package,
it will not automatically export it.

You don't show where "effect.distance" is defined.  If it is defined
in the global environment, you could explicitly export it via the foreach
".export" argument.  See the foreach man page for details.  If it is
defined in a package, you could use the ".packages" argument to
load that package on each of the parallel workers.

Note that this process of automatically exporting variables is performed
in doSNOW and doMPI, but not in doMC.  That is because the multicore
package uses the "fork" system call to create the worker processes.
Processes that are forked get a copy of the parent process's execution
environment, and so variables don't generally need to be exported,
either implicitly or explicitly.  Thus, you could run into problems with
undefined variables with doSNOW, even though your program runs fine
with doMC.

- Steve



On Tue, Aug 24, 2010 at 8:13 AM, Roman Luštrik <roman.lustrik at gmail.com> wrote:
> Dear list,
>
> I once again seek your advice in understanding the following.
> makeDistances is a function I feed to foreach. The computation intensive
> part is done by a second function calcDistance.
>
>    # Run in parallel
>    makeDistances <- function(xy, rst, effect.distance, cnt, num.points) {
>        source("d:/workspace/Texas Ranger/calcDistance.R")
>        for (j in 1:num.points) {
>            file.name <- paste("walker", cnt, "_", j, sep = "")
>            clc <- calcDistance(object = rst, origin.point = xy[j,],
>                      effect.distance = effect.distance)
>            writeRaster(clc, file.name, overwrite = TRUE)
>        }
>    }
>
>    foreach(xy = xy.list, cnt = icount()) %do%
>       makeDistances(xy, rst = rst, effect.distance = effect.distance,
>       cnt, num.points = dim(xy.list[[cnt]])[1])
>
> When I run makeDistances via %do% things work out fine, but I'm unable to
> run via %dopar%. The following error is returned:
>
> Error in makeDistances(xy, rst = rst, effect.distance = effect.distance,  :
>  task 1 failed - "object 'effect.distance' not found"
>
>
> It seems like a scoping issue, however I failed to comprehend how to feed
> the argument to the function to make it work. Any advice will be much
> appreciated.
>
>
> Cheers,
> Roman
>
> --
> In God we trust, all others bring data.
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>