[Bioc-devel] build machines

Fri Apr 27 00:02:29 CEST 2018

Hi,

The Linux and Windows builders have 32 GB of RAM, the Mac
builders 64 Gb.

We also run concurrent R CMD check's.

Here is a summary:

   platform           RAM   nb of     nb of concurrent
                      (Gb)  cores        R CMD check's
   ---------------------------------------------------
   Linux (malbecs)     32      20                   10
   Windows (tokays)    32      40                   24
   Mac (meridas)       64      24                   18

That's a lot of concurrency. And there is actually more
concurrency than that if you consider the fact that many
packages run things in parallel during 'R CMD check'.
Some packages are good citizens and limit the number of
cores to 1 or 2 only during 'R CMD check' but some packages
try to use all the cores that are available. This will have
a strong impact on the overall progress of the builds. We
don't have an easy way to identify those packages right now.

In average, based on our monitoring of the build machines
things seem to work ok i.e. the concurrent R CMD check's
don't seem to be competing too much to access resources.

But occasionally there could be too much competition. The
crazy big elapsed time compared to the relatively short user
and system times that you observed Kasper are likely to reflect
that. They could be the sign that the machine ran out of memory
and started swapping. Not because it happens to your package
means that your package uses too much memory. The swapping is
the result of the **cumulated** memory usage of all the
R CMD check's running at that moment. It could be worth checking
how much memory R CMD check'ing your package uses though.

The exact set of packages that are being R CMD check'ed at any
given time is in constant fluctuation and will also vary from
one day to the other. This would explain why some days you see
timeouts on some platforms and some days not. We don't have
an easy way to know which packages were competing with yours
during the 40 min window that 'R CMD check' was running on your
package until the build system declared a timeout. It's possible
(by looking at the BBS logs) but is time consuming.

We should probably add some memory at some point to the Windows
builders. 32 Gb is not enough to smoothly run 24 R CMD check's
concurrently.

H.

On 04/26/2018 08:48 AM, Diogo FT Veiga wrote:
> Hi Daniel,
> 
> I have the same issue with my package (new contribution). I just finish
> reviewing the package with the modifications requested.
> 
> I am having a warning because R CMD check is exceeding 5 min, but this is
> happening only in the Windows machine.
> 
> In Linux and OSX the check finishes in <= 4min, while in Windows takes
> ~6min.
> 
> https://urldefense.proofpoint.com/v2/url?u=http-3A__bioconductor.org_spb-5Freports_maser-5Fbuildreport-5F20180425114748.html&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=1aMitB3PnVLoojx1lnj_UT_ZeKlJ_OcJDFT4D6BPXow&e=
> 
> Not sure how to proceed from here.
> 
> Thanks,
> Diogo
> 
> 
> On Thu, Apr 26, 2018 at 9:52 AM, Kasper Daniel Hansen <
> kasperdanielhansen at gmail.com> wrote:
> 
>> We have been working on the minfi package lately, with a move to a
>> DelayedArray backend.
>>
>> Right now there are some weird issues regarding timings in R CMD check.
>> Leaving aside the issue that the tests (now disabled) and examples are too
>> slow, we get some very weird behaviour.
>>
>> An example is the current (soon to be replace) build report of minfi 1.25.2
>> which prints
>>
>> Examples with CPU or elapsed time > 5s
>>                        user system  elapsed
>> preprocessFunnorm   99.388  0.632  148.554
>> combineArrays       64.104  2.120   68.329
>> bumphunter          62.540  1.392   64.107
>> preprocessNoob      43.944  0.016   44.955
>> preprocessQuantile  33.968  0.064   36.547
>> getAnnotation       31.072  0.024   31.126
>> compartments        18.668  0.188   18.871
>> minfiQC             10.124  6.628 1102.929
>> getSex              10.536  0.012   10.561
>> read.metharray       7.504  2.116   82.713
>> read.metharray.exp   9.076  0.032   10.592
>> mapToGenome-methods  4.648  0.548  163.648
>> mdsPlot              0.340  0.204   14.901
>>
>>
>> on Tokay (Linux).  Note minfiQC which has an elapsed time which is crazy
>> high compared to user+system.  Previous build report (which I didn't save)
>> had a timeout on all platforms with a semingly similar behaviour but with
>> the getSex function.  The code did not change in the meantime.  For today's
>> build we only see this on Linux, but yesterday all platforms were affected.
>>
>> This is likely to be very hard to debug.  But I am thinking memory issues:
>> this example requires loading an annotation package and a data package,
>> both of which are "big".  How much RAM does the machines have and are
>> multiple R CMD check's run concurrently?
>>
>> Best,
>> Kasper
>>
>>          [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel at r-project.org mailing list
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=R1DGN1kNpBZ4ZRBCTQzDPQlNYapuBNSYB4JTM6tO60w&e=
>>
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> Bioc-devel at r-project.org mailing list
> https://urldefense.proofpoint.com/v2/url?u=https-3A__stat.ethz.ch_mailman_listinfo_bioc-2Ddevel&d=DwICAg&c=eRAMFD45gAfqt84VtBcfhQ&r=BK7q3XeAvimeWdGbWY_wJYbW0WYiZvSXAJJKaaPhzWA&m=JwiMI-3BEUJlonlihLD_mDkPuEIalQbkrQPSGahzfsg&s=R1DGN1kNpBZ4ZRBCTQzDPQlNYapuBNSYB4JTM6tO60w&e=
> 

-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319