[Bioc-devel] Redirect workers output to STDERR, how to do so with current BiocParallel::SnowParam()?

[Leonardo Collado Torres]

Hi,

This might be a BioC support website question, but maybe it's a bug.

In previous versions of BiocParallel, you could specify where to
direct the output from the workers by using the 'outfile' argument.
For example, SnowParam(outfile = Sys.getenv('SGE_STDERR_PATH')). I'm
not finding how to do so with the current version (1.3.12).

I understand that SnowParam() has a ... argument that gets passed to
snow::makeCluster(), according to the SnowParam() docs. I also see
that when log = TRUE, a script is used instead of snow. But log =
FALSE by default.

If I use snow, the 'outfile' argument does work as shown below:

## Nothing gets printed by default
> cl <- makeCluster(2, type = 'SOCK')
> y <- clusterApply(cl, 1:2, print)


## Use outfile now, print works
> cl <- makeCluster(2, type = 'SOCK', outfile = NULL)
starting worker for localhost:11671
starting worker for localhost:11671
> y <- clusterApply(cl, 1:2, print)
Type: EXEC
Type: EXEC
[1] 1
[1] 2

> packageVersion('snow')
[1] ‘0.3.13’


However, I can't use 'outfile' with SnowParam:

## SerialParam works
> x <- bplapply(1:2, print, BPPARAM = SerialParam())
[1] 1
[1] 2

## No printing by default with SnowParam, as expected
> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1))

## Can't pass 'outfile' argument
> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, outfile = NULL))
Error in bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, outfile
= NULL)) :
  error in evaluating the argument 'BPPARAM' in selecting a method for
function 'bplapply': Error in envRefSetField(.Object, field, classDef,
selfEnv, elements[[field]]) :
  ‘outfile’ is not a field in class “SnowParam”


Digging at the code, I see that in SnowParam() the ... argument is
saved in .clusterargs

args <- c(list(spec = workers, type = type), list(...))
    .clusterargs <- lapply(args, force)

However, ... is still passed to .SnowParam(), and .SnowParam() fields are:

fields=list(
        cluster="cluster",
        .clusterargs="list",
        .controlled="logical",
        log="logical",
        threshold="ANY",
        logdir="character",
        resultdir="character")


So, I'm wondering if ... should not be passed to .SnowParam().



## Trying to pass outfile to .clusterargs directly doesn't work
## Actually, I'm surprised it didn't crash .SnowParam()
> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, .clusterargs=list(outfile = NULL)))


Full log: https://gist.github.com/9ac957c6cad1c07f4ea4


Thanks,
Leo