[Bioc-devel] BiocParallel Variable Not Found

[Martin Morgan]

Windows uses separate processes that do not share memory (SnowParam()), whereas linux / mac by default use forked processes that share the original memory (MulticoreParam()). So

> y = 1
> param = MulticoreParam()
> res = bplapply(1:2, function(x) y, BPPARAM=param)

works because the function can 'see' y, whereas

> param = SnowParam()
> res = bplapply(1:2, function(x) y, BPPARAM=param)
Error: BiocParallel errors
  element index: 1, 2
  first error: object 'y' not found

fails because the new processes cannot see y. The robust way to implement this makes the FUN in bplapply truly functional, depending only on variables received as arguments

> res = bplapply(1:2, function(x, y) y, y = 1, BPPARAM=param)
>

The situation is more complicated in package code in particular and in general, because R sends not just FUN to the workers, but also the environment (up to but not including the global environment) in which the function applies. So

doit <- function() {
    y = 1
    bplapply(1:2, function(x) y, BPPARAM = SnowParam(2))
}

also works

> res = doit()
>

Note that

fun = function(x) y

doit = function() {
    y = 1
    bplapply(1:2, fun, BPPARAM = SnowParam(2))
}

fails, because y is not defined in the environment in which fun is defined (it's defined in the calling environment, which is different).

The most robust advice is to develop you code with the most conservative assumptions (e.g., register(SnowParam())) and to write functional functions where all variables are passed as arguments.

Martin

On 3/17/20, 6:00 AM, "Bioc-devel on behalf of Dario Strbenac" <bioc-devel-bounces using r-project.org on behalf of dstr7320 using uni.sydney.edu.au> wrote:

    Good day,
    
    I have a loop in a function of my R package which by default uses bpparam() to set the framework used for parallelisation. On Windows, I see the error
    
    Error: BiocParallel errors
      element index: 1, 2, 3, 4, 5, 6, ...
      first error: object 'selParams' not found
    
    This error does not happen on the Linux or MacOS operating systems. It happens using both R 3.6 and the upcoming version 4. The error can be reproduced running the examples of runTests function in ClassifyR.
    
    --------------------------------------
    Dario Strbenac
    University of Sydney
    Camperdown NSW 2050
    Australia
    
    _______________________________________________
    Bioc-devel using r-project.org mailing list
    https://stat.ethz.ch/mailman/listinfo/bioc-devel