[Bioc-devel] Help for Error "Maximal Number of DLLs reached..."

Wed Mar 21 17:28:41 CET 2018

Thanks guys for your kind help! I would follow your tips and try to fix
these issues.

It's indeed quite hard to maintain it, luckily in past two years I made it.
It's indeed the right time to cut down some unnecessary packages and make
it easier to be maintained.

Thanks for your help! ^_^

Best
Yuan Tian

On Thu, Mar 22, 2018 at 12:14 AM Hervé Pagès <hpages at fredhutch.org> wrote:

> Hi,
>
> On 03/21/2018 07:32 AM, Martin Morgan wrote:
> >
> >
> > On 03/21/2018 10:14 AM, Henrik Bengtsson wrote:
> >> A few quick comments:
> >>
> >> * I don't think the limit was hardened in R 3.4.0; maybe you started to
> >> experience it sound then because more dependent packages started to
> >> rely on
> >> more DLLs?
> >>
> >> * The limit is OS/platform specific so it's not that R core is
> >> unwilling to
> >> fix this - I think they havejust tried to find a safe limit that work
> >> everywhere. Having said this...
> >>
> >> * It'll be less restrictive by default in R 3.5.0: from it's NEWS "The
> >> maximum number of DLLs that can be loaded into R e.g. via dyn.load() has
> >> been increased up to 614 when the OS limit on the number of open files
> >> allows"
> >>
> >> * R.utils::gcDLLs() will attempt to unregister possibly stray DLLs
> >> remaining after unloading packages. So unloading packages and gcDLLs()
> >> may
> >> provide you with a workaround until R 3.5.0 is in place (possibly also
> >> even
> >> after).
> >>
> >> See
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HenrikBengtsson_Wishlist-2Dfor-2DR_issues_29&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=7UNVEDPfEm1j53ksGc7naLgnieltNzOnbv4lPUQh4Jk&e=
> >> for
> >> "random" notes and references to this problem.
> >>
> >>
> >> On Wed, Mar 21, 2018, 04:10 Yuan Tian <tianyuan1991hit at gmail.com>
> wrote:
> >>
> >>> Hello Guys:
> >>>
> >>> I encountered an error like "maximal number of dlls reached...". I am
> >>> maintaining ChAMP package now, which integrated many other packages
> >>> in my
> >>> research field. I have not updated this package in past 2 months but
> >>> suddenly this error happens.
> >>>
> >>> Currently, I think I know the reason is since R 3.4.X, numbers of
> >>> DLLs in
> >>> default R session was set 100. I have tested that using a newly
> >>> started R
> >>> 3.4.4 to install and load ChAMP package, *it works smoothly*, after
> >>> loading, I checked DLL loaded with function getLoadedDLLs(), then I see
> >>> ChAMP used 95 Dlls. I know it's a lot, but some of them are loaded by
> >>> ChAMP's relying packages but itself. *However, ChAMP cannot pass
> >>> Bioconductor check, thus I suspect Bioconductor checking system does
> NOT
> >>> start a new R session for each package right? Which means it's not
> >>> 100 DLLs
> >>> allocated for each package?*
> >
> > ChAMP is passing BiocCheck in release, where R is 3.4.x ?
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_checkResults_release_bioc-2DLATEST_ChAMP_malbec1-2Dchecksrc.html&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=_B73-e5Y2XkRdvs7ssHGb6DAdSnMmG30eE_Ywu1W7iQ&e=
>
> mmmh, this is a stale page. Not sure what it's doing here. Will
> investigate.
>
> Today's "true" results are here:
>
>    http://bioconductor.org/checkResults/release/bioc-LATEST/ChAMP/
>
> Note that you can't get to the stale page by following links from
> the report's main page:
>
>    http://bioconductor.org/checkResults/release/bioc-LATEST/
>
> >
> >
> > Each package is built and tested in a separate process. Vignettes are
> > actually built in a single process (by R, not Bioconductor) so multiple
> > vignettes could load a higher number of DLLs.
>
> Also just to clarify: the build machines don't use any special settings
> to limit the nb of DLLs. They use whatever R uses by default.
>
> >
> >>>
> >>> Currently, I have told my users to modify R environment to allow their
> R
> >>> session with more DLLs. It works but only on users computer, not
> >>> Bioconductor checking system.
>
> Note that, in release (with R 3.4.4), the max DLL error happens at
> the end of an install:
>
>  > biocLite("ChAMP")
> BioC_mirror: https://bioconductor.org
> Using Bioconductor 3.6 (BiocInstaller 1.28.0), R 3.4.3 (2017-11-30).
> Installing package(s) ‘ChAMP’
> trying URL
> '
> https://bioconductor.org/packages/3.6/bioc/src/contrib/ChAMP_2.9.10.tar.gz
> '
> Content type 'application/x-gzip' length 15513863 bytes (14.8 MB)
> ==================================================
> downloaded 14.8 MB
>
> * installing *source* package ‘ChAMP’ ...
> ** R
> ** inst
> ** preparing package for lazy loading
> Warning: replacing previous import ‘igraph::edges’ by ‘graph::edges’
> when loading ‘FEM’
> Warning: replacing previous import ‘igraph::intersection’ by
> ‘graph::intersection’ when loading ‘FEM’
> Warning: replacing previous import ‘igraph::degree’ by ‘graph::degree’
> when loading ‘FEM’
> Warning: replacing previous import ‘igraph::union’ by ‘graph::union’
> when loading ‘FEM’
> Warning: replacing previous import ‘limma::plotMA’ by
> ‘BiocGenerics::plotMA’ when loading ‘FEM’
> Warning: replacing previous import ‘Matrix::colSums’ by
> ‘BiocGenerics::colSums’ when loading ‘FEM’
> Warning: replacing previous import ‘Matrix::colMeans’ by
> ‘BiocGenerics::colMeans’ when loading ‘FEM’
> Warning: replacing previous import ‘Matrix::rowMeans’ by
> ‘BiocGenerics::rowMeans’ when loading ‘FEM’
> Warning: replacing previous import ‘Matrix::rowSums’ by
> ‘BiocGenerics::rowSums’ when loading ‘FEM’
> Warning: replacing previous import ‘Matrix::which’ by
> ‘BiocGenerics::which’ when loading ‘FEM’
> Warning: replacing previous import ‘igraph::normalize’ by
> ‘BiocGenerics::normalize’ when loading ‘FEM’
> Warning: replacing previous import 'plyr::summarise' by
> 'plotly::summarise' when loading 'ChAMP'
> Warning: replacing previous import 'plyr::rename' by 'plotly::rename'
> when loading 'ChAMP'
> Warning: replacing previous import 'plyr::arrange' by 'plotly::arrange'
> when loading 'ChAMP'
> Warning: replacing previous import 'plyr::mutate' by 'plotly::mutate'
> when loading 'ChAMP'
> Error in dyn.load(file, DLLpath = DLLpath, ...) :
>    unable to load shared object
> '/home/hpages/R/R-3.4.3/library/robustbase/libs/robustbase.so':
>    `maximal number of DLLs reached...
> ERROR: lazy loading failed for package 'ChAMP'
> * removing '/home/hpages/R/R-3.4.3/library/ChAMP'
>
> The downloaded source packages are in
>         ‘/tmp/Rtmp87p9oq/downloaded_packages’
> Updating HTML index of packages in '.Library'
> Making 'packages.html' ... done
>
> So even if ChAMP's documentation says something about increasing the
> max number of DLLs, most users are probably going to find this situation
> frustrating and complain about it.
>
> >>> Thus I think I have to reduce dlls used by
> >>> current package right? Like removing some relying packages or
> >>> function. *My
> >>> another question is how many DLLs is allowed by Bioconductor check? I
> >>> think
> >>> it's less than 100. But I don't know I should cut it into 80 or 60 or
> >>> even
> >>> 50 dlls used.*
> >>>
> >>> It's really disappointing that I need to modify quite a lot of code,
> and
> >>> even could hurt some key functionality of the package. Thus here I am
> >>> seeking your help and suggestions here.
> >
> > Frankly, I think a package with so many dependencies cannot be
> > maintained -- a change in any one of those packages could break your
> > package (e.g., by changing their own dependencies to include additional
> > DLLs!), and it must be virtually impossible to get sufficient test
> > coverage to be confident that these problems will be detected before the
> > package is made available to the user. It is time to consider a more
> > modular design focusing on essential features.
>
> The first thing you could try to do is move some deps from Depends or
> Imports to Suggests. Right now, after doing library(ChAMP),
> sessionInfo() reports 36 packages on the search path and another 160
> packages loaded via a namespace! Couldn't some of those packages be
> moved to Suggests without hurting the usability of ChAMP?
>
> One last thing: ChAMP is at version 2.9.10 in release and 2.9.11 in
> devel. This is not good. The y part of x.y.z should always be even
> in release. Generally speaking, you should only bump the z part of the
> version when you make changes to your package. We take care of bumping
> the y part for you at release time. See:
>
>    https://bioconductor.org/developers/how-to/version-numbering/
>
> Too late to fix in release though. Just be aware that when we release
> BioC 3.7, we'll bump ChAMP version to 2.10.0 in the new release and
> to 2.11.0 in the new devel (i.e. BioC 3.8).
>
> Cheers,
> H.
>
> >
> > Martin
> >
> >>>
> >>> Best
> >>> Yuan Tian
> >>>
> >>>          [[alternative HTML version deleted]]
> >>>
> >>> _______________________________________________
> >>> Bioc-devel at r-project.org mailing list
> >>>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
> >>>
> >>>
> >>
> >>     [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> Bioc-devel at r-project.org mailing list
> >>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
> >>
> >>
> >
> >
> > This email message may contain legally privileged and/or...{{dropped:2}}
> >
> > _______________________________________________
> > Bioc-devel at r-project.org mailing list
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=p2zu7G8IjcYJfFGXTvJAlITWCwrBG0zBH72Htgm6go8&s=Rs9gi9iOQtOG4zUXSOO07D7t_smZ_h6OPyISr1DHf8c&e=
> >
>
> --
> Hervé Pagès
>
> Program in Computational Biology
> Division of Public Health Sciences
> Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N, M1-B514
> P.O. Box 19024
> Seattle, WA 98109-1024
>
> E-mail: hpages at fredhutch.org
> Phone:  (206) 667-5791
> Fax:    (206) 667-1319
>

	[[alternative HTML version deleted]]