[Bioc-devel] Redirect workers output to STDERR, how to do so with current BiocParallel::SnowParam()?

[Valerie Obenchain]

Hi,

Thanks for reporting the bug. Now fixed in 1.13.4 (devel) and 1.2.2 
(release).

 > bplapply(1:2, print, BPPARAM=SnowParam(outfile = NULL))
starting worker for localhost:11031
starting worker for localhost:11031
Type: EXEC
[1] 1
Type: EXEC
[1] 2
Type: DONE
Type: DONE
[[1]]
[1] 1

[[2]]
[1] 2


Some new features have been added to BiocParallel - logging with 
futile.logger, writing out log and result files and control over how a 
job is divided into tasks.

## send log file to 'logdir':
bplapply(1:2, print, BPPARAM=SnowParam(log=TRUE, logdir=tempdir())

## write results to 'resdir':
bplapply(1:2, print, BPPARAM=SnowParam(resdir=tempdir())

## by default jobs are divided evenly over the workers
bplapply(1:100, print, BPPARAM=SnowParam(workers=4))

## force the job to be run in 2 tasks
## (useful when you know runtime or memory requirements)
bplapply(1:100, print, BPPARAM=SnowParam(tasks=2))


It would be great if you had a chance to try any of these out - I'd be 
interested in the feedback. Logging was intended to take the idea of the 
'outfile' from snow further with the ability to add messages that can be 
filtered by threshold.

The package also has a new errors/logging vignette:

http://www.bioconductor.org/packages/3.2/bioc/vignettes/BiocParallel/inst/doc/Errors_Logs_And_Debugging.pdf

Valerie



On 05/21/2015 07:58 AM, Leonardo Collado Torres wrote:
> Hi,
>
> This might be a BioC support website question, but maybe it's a bug.
>
> In previous versions of BiocParallel, you could specify where to
> direct the output from the workers by using the 'outfile' argument.
> For example, SnowParam(outfile = Sys.getenv('SGE_STDERR_PATH')). I'm
> not finding how to do so with the current version (1.3.12).
>
> I understand that SnowParam() has a ... argument that gets passed to
> snow::makeCluster(), according to the SnowParam() docs. I also see
> that when log = TRUE, a script is used instead of snow. But log =
> FALSE by default.
>
> If I use snow, the 'outfile' argument does work as shown below:
>
> ## Nothing gets printed by default
>> cl <- makeCluster(2, type = 'SOCK')
>> y <- clusterApply(cl, 1:2, print)
>
>
> ## Use outfile now, print works
>> cl <- makeCluster(2, type = 'SOCK', outfile = NULL)
> starting worker for localhost:11671
> starting worker for localhost:11671
>> y <- clusterApply(cl, 1:2, print)
> Type: EXEC
> Type: EXEC
> [1] 1
> [1] 2
>
>> packageVersion('snow')
> [1] ‘0.3.13’
>
>
> However, I can't use 'outfile' with SnowParam:
>
> ## SerialParam works
>> x <- bplapply(1:2, print, BPPARAM = SerialParam())
> [1] 1
> [1] 2
>
> ## No printing by default with SnowParam, as expected
>> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1))
>
> ## Can't pass 'outfile' argument
>> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, outfile = NULL))
> Error in bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, outfile
> = NULL)) :
>    error in evaluating the argument 'BPPARAM' in selecting a method for
> function 'bplapply': Error in envRefSetField(.Object, field, classDef,
> selfEnv, elements[[field]]) :
>    ‘outfile’ is not a field in class “SnowParam”
>
>
> Digging at the code, I see that in SnowParam() the ... argument is
> saved in .clusterargs
>
> args <- c(list(spec = workers, type = type), list(...))
>      .clusterargs <- lapply(args, force)
>
> However, ... is still passed to .SnowParam(), and .SnowParam() fields are:
>
> fields=list(
>          cluster="cluster",
>          .clusterargs="list",
>          .controlled="logical",
>          log="logical",
>          threshold="ANY",
>          logdir="character",
>          resultdir="character")
>
>
> So, I'm wondering if ... should not be passed to .SnowParam().
>
>
>
> ## Trying to pass outfile to .clusterargs directly doesn't work
> ## Actually, I'm surprised it didn't crash .SnowParam()
>> x <- bplapply(1:2, print, BPPARAM = SnowParam(workers = 1, .clusterargs=list(outfile = NULL)))
>
>
> Full log: https://gist.github.com/9ac957c6cad1c07f4ea4
>
>
> Thanks,
> Leo
>
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
>


-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, Seattle, WA 98109

Email: vobencha at fredhutch.org
Phone: (206) 667-3158