[R-sig-hpc] Problems with Exporting Functions with Foreach/DoSNOW

[Reuben Bellika]

Hello,

I am currently using the "foreach" and "doSNOW" packages to run calculations on our cluster (using SNOW over sockets). Sometimes things get rather complex with multiple functions being called inside the foreach loop. There seems to be a limitation on how I can pass functions to the SNOW cluster workers. If one main function gets evaluated multiple times inside the foreach loop, and this function calls several helper functions, I get an error saying that these helper functions are undefined, even if they are included in the export option.

Here's an example of what I'm talking about:


library(foreach)
library(snow)
library(doSNOW)

# Two helper functions
helper1 <- function(i) { return(i + 1) }
helper2 <- function(i) { return(i + 2) }

# The main function called once each loop
main.fun <- function(i)
{
    # Call two other functions
    return(helper1(i) + helper2(i))
}

# Compute the values (odd numbers from 5 to 23) using a for loop
compute.local <- function()
{
    values <- c()
    for (i in 1:10)
    {
        values <- c(values, main.fun(i))
    }
    
    return(values)
}

# Compute the values (odd numbers from 5 to 23) using a foreach loop
compute.cluster <- function()
{
    values <- foreach(i = 1:10,
                      .export = c("main.fun", "helper1", "helper2"),
                      .combine = "c") %dopar%
    {
        main.fun(i)
    }
    
    return(values)
}

# Start the cluster and register with doSNOW (node names are just examples)
cl <- makeCluster(c("r-node-001", "r-node-002"), type = "SOCK")
registerDoSNOW(cl)

print(compute.local())
print(compute.cluster())

# Stop the cluster
stopCluster(cl)


When I run this example on our cluster, I get the expected output from the compute.local() function:
[1]  5  7  9 11 13 15 17 19 21 23

The compute.cluster() function, however, terminates with an error:
Error in { : task 1 failed - "could not find function "helper1""

Note that there was no problem with exporting main.fun, only helper1 and helper2.

So, what I am wondering, is there something I don't understand about the initialization of the foreach loop, or is there a bug of some type with the doSNOW package? I've tried running this same example using doMC to evaluate the foreach loop instead of doSNOW, and it works perfectly, with no errors. I can work around the error by defining helper1 and helper2 inside the body of main.fun, but this gets awkward in real cases where the helper functions themselves are very complex and I want to reuse them in other places.

The cluster master and worker nodes are running R 2.14.0 on Debian "Lenny" i386.

Any ideas?

Thank you,
Reuben Bellika