[Bioc-devel] BiocParallel: windows vs. mac/linux behavior

[Ludwig Geistlinger]

Great - using `local` solves my problem here.

> You said 'for Windows here serial' but a SerialParam() just runs lapply and would not have problems finding f

Ah, I assumed that the registry under Windows would also have a MulticoreParam as default which would then dispatch to a SerialParam -- but I now see that it actually has a SnowParam as default. Even better!

Thanks a lot,
Ludwig


--
Dr. Ludwig Geistlinger
CUNY School of Public Health

________________________________________
From: Martin Morgan <martin.morgan at roswellpark.org>
Sent: Wednesday, January 31, 2018 6:51 PM
To: Ludwig Geistlinger; bioc-devel at r-project.org
Subject: Re: [Bioc-devel] BiocParallel: windows vs. mac/linux behavior

On 01/31/2018 06:39 PM, Ludwig Geistlinger wrote:
> Hi,
>
>
> I am currently considering the following snippet:
>
>
>> data.ids <- paste0("d", 1:5)
>> f <- function(x) paste("dataset", x, sep=" = ")
>> res <- BiocParallel::bplapply(data.ids, function(d) f(d))
>
>
> Using a recent R-devel on both a Linux machine and a Mac machine, this works fine.
>
>
> However, on a Windows R-devel this throws:
>
>
> Error: BiocParallel errors
>
>        element index: 1, 2, 3, 4, 5 ....
>
>        first error: could not find function "f"
>
>
> Is this a bug or is this related to the different ways in which parallel (for windows here serial) computation is carried out?
>

Windows runs brand-new processes, whereas linux / mac use forked
processes that share the original process's memory. So Windows doesn't
know about the variables defined in the global environment. You can
mimick this on non Windows using SnowParam(), e.g.,

register(bpstart(SnowParam(2))  ## set up a cluster for the duration...

 > res <- BiocParallel::bplapply(data.ids, function(d) f(d))
Error: BiocParallel errors
   element index: 1, 2, 3, 4, 5
   first error: could not find function "f"

The rule is that the environment of the FUN gets serialized to the
worker, up to the global environment. So

local({
     f <- function(x) paste("dataset", x, sep=" = ")
     BiocParallel::bplapply(data.ids, function(d) f(d))
})

works (FUN is defined in the local environment, the local environment
contains f and is serialized to the worker). This also implies that
BiocParallel will find functions that are in the same package as FUN.

A tweak

   BiocParallel::bplapply(data.ids, f)

also works, because f (but not the .GlobalEnv in which it was defined)
is serialized.

You said 'for Windows here serial' but a SerialParam() just runs lapply
and would not have problems finding f.

Martin

>
> Thanks,
>
> Ludwig
>
>
>
> --
> Dr. Ludwig Geistlinger
> CUNY School of Public Health
>
>       [[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>


This email message may contain legally privileged and/or confidential information.  If you are not the intended recipient(s), or the employee or agent responsible for the delivery of this message to the intended recipient(s), you are hereby notified that any disclosure, copying, distribution, or use of this email message is prohibited.  If you have received this message in error, please notify the sender immediately by e-mail and delete this email message from your computer. Thank you.