[Bioc-devel] [Untrusted Server]Re: strange error in Jenkins build forsingleCellWorkflow

[Martin Morgan]

On 09/16/2017 01:53 AM, Aaron Lun wrote:
> Bumping this rather old thread. To re-iterate, I'm updating my simpleSingleCell workflow and I'm running into R's DLL limit. I've added a code block halfway through the workflow that unloads all DLLs and cleans them out, and this works fine during compilation on my local machine.
> 
> 
> However, it seems that the BioC workflow builder uses a pre-processing step whereby it first tries to load all packages contained within library() calls. This hits the DLL limit as it doesn't execute the protective code block, which defeats the purpose of all my fiddling in the first place.
> 
> 
> What options are there? I'm happy to split my workflow into multiple smaller Rmarkdown files that get compiled separately, provided there is appropriate support for this setup from the build system

The workflows have been standardized as packages. The packages put the 
workflow dependencies in the 'Depends:' field, with the idea being that 
the user installing the workflow package 'in the usual way' will get the 
packages used in the vignette installed in their system 'in the usual 
way' without having to execute special variants of biocLite() / 
install.packages() / funky code in the vignette itself to be able to 
build the vignette.

Loading a package loads its Depends: (and Imports:) so triggers the problem.

Writing separate vignettes would not help with this (but might make the 
workflow more palatable; I'm not 100% sure of support for separate work 
flows in a single package, there is no problem with having multiple 
workflow packages on the same general topic).

One could move (some?) packages to Suggests: and use your trick of 
unloading packages part-way through the vignette. But then users will 
find that they need to install packages to complete the vignette.

'We' could add a support for a BBS option that increases R_MAX_NUM_DLLS, 
but that would allow the workflow to build on the build system, but not 
on the users' system.

I think also the R-core approach to this 
(https://stat.ethz.ch/pipermail/r-devel/2016-December/073529.html, 
https://github.com/wch/r-source/commit/757bfa1d7ff373a604d6d34617f9cad78e0c875e) 
is a little insightful, where one could imagine increasing the default 
R_MAX_NUM_DLLS, but apparently on some OS these compete for number of 
open files, and this in turn can be quite low.

I note that users have already struggled with the DLL problem 'in the 
wild' https://stackoverflow.com/a/45552926/547331. This seems 
particularly problematic for workflows, which are appealing to 
relatively novice users.

At the end of the day I think the workflows should make realistic use of 
R resources. I think this means modifying the workflow to use fewer 
DLLs. (this general comment is relevant to other workflows, which for 
instance start by downloading very large data sets -- I know that less 
constrained use of computing resources is supposed to be a selling point 
of the workflows, but in excess this seems counter-productive to their 
primary use as pedagogic tools [rather than, for instance, comprehensive 
exemplars of reproducible research]).

Maybe there is additional discussion about some of the technical aspects 
of workflows that others might contribute.

Martin

> 
> 
> Cheers
> 
> 
> Aaron
> 
> ________________________________
> From: Bioc-devel <bioc-devel-bounces at r-project.org> on behalf of Aaron Lun <alun at wehi.edu.au>
> Sent: Wednesday, 21 June 2017 12:09:13 AM
> To: bioc-devel at r-project.org
> Subject: [Untrusted Server]Re: [Bioc-devel] strange error in Jenkins build forsingleCellWorkflow
> 
> Hi all,
> 
> 
> I'm getting a curious error in the Jenkins log when I try to build the singleCellWorkflow:
> 
> 
> http://docbuilder.bioconductor.org:8080/job/simpleSingleCell/48/label=master/console
> 
> 
> The key part is at the bottom:
> 
> 
> Error: package or namespace load failed for 'GenomicFeatures' in dyn.load(file, DLLpath = DLLpath, ...):
>   unable to load shared object '/var/lib/jenkins/R/x86_64-pc-linux-gnu-library/3.4/Rsamtools/libs/Rsamtools.so':
>    `maximal number of DLLs reached...
> 
> 
> The workflow had previously been running fine on the build system; I'm not quite sure what's going on here, given that it's not even failing at the point where I made the latest changes.
> 
> Cheers,
> 
> Aaron
> 
>          [[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel
> 


This email message may contain legally privileged and/or...{{dropped:2}}