[Bioc-devel] Fw: MungeSumstats Bioconductor

[Hervé Pagès]

Hi Alan,

The specs of the Linux servers have not changed.

However we've recently observed some of these random kills by the Linux 
kernel for a couple of other packages, and we started an application to 
get funding to increase the amount of RAM on these machines.

These random kills are a desperate attempt by the Linux kernel to keep 
the machine alive when it's under extremely high load and running out of 
resources. Unfortunately this is not something that anybody will be able 
to easily reproduce as it only happens under special circumstances e.g. 
during the daily builds when dozens of packages are being checked 
simultaneously.

Regardless of whether we'll manage to increase the memory on the Linux 
servers, it seems to me that the most memory-hungry unit tests in 
MungeSumstats would be a better fit for the long tests. These tests are 
run once a week instead of daily, and use less workers (4 instead of 
28). This means that each package has access to more resources.

See https://contributions.bioconductor.org/long-tests.html for how to 
set up the long tests in your package.

Let me know if you have questions or need help with this.

Best,

H.

On 2/2/24 09:14, alan murphy wrote:
> Hi,
>
> I'm the maintainer of the MungeSumstats package which is currently failing on the devel nebbiolo1 Linux platform because of RAM requirements of the unit tests. These tests use large reference datasets (like BSgenome.Hsapiens.1000genomes.hs37d5::BSgenome.Hsapiens.1000genomes.hs37d5) which cause the issue, e.g. from the output of the devel linux test<https://bioconductor.org/checkResults/3.19/bioc-LATEST/MungeSumstats/nebbiolo1-checksrc.html>:
>
> ```
>
> Validating RSIDs of 92 SNPs using BSgenome::snpsById...
> Killed
>
> ```
> which is from this function:https://github.com/neurogenomics/MungeSumstats/blob/cccf77b2249f52be59fe1749f13a386ffaaae528/R/load_ref_genome_data.R#L49
>
> I would like to keep these units tests as they are very important for me to know if things have gone wrong. I can't scale down the RAM usage, it is only using a few rows of data as it, the issue is that the reference sets are massive rather than the actual set being tested. I currently don't have a lot of these tests set to run on windows/mac platforms so I rely on the Linux machines to run this. I am not sure what has changed but a few releases back, the same tested were not causing this issue and would complete on linux - has the machine spec changed? Is there any chance RAM could be increased for these tests or is there a way to specify not to run the specific unit tests on the Bioconductor server so I can at least keep these for my github actions workflows? See below for the automated message I got about these errors.
>
> Any other suggestions on ways around this would be greatly appreciated!
>
> Thanks,
> Alan.
> ________________________________
> From: CoreTeam Bioconductor<bioconductorcoreteam using gmail.com>
> Sent: Friday 2 February 2024 16:43
> To:alanmurph94 using hotmail.com  <alanmurph94 using hotmail.com>
> Subject: MungeSumstats Bioconductor
>
> Hello Package Maintainer,
>
> We would like to bring to your attention that your package is failing in devel on the linux platform. This is very problematic. Please investigate the issues and fix the package to avoid deprecation.
>
> https://bioconductor.org/checkResults/devel/bioc-LATEST/
> <https://bioconductor.org/checkResults/3.17/workflows-LATEST/TCGAWorkflow>
>
> If you have further questions or concerns please reach out on thebioc-devel using r-project.org<mailto:bioc-devel using r-project.org>
>
> We appreciate your quick attention to this matter
>
> Cheers,
> On behalf of the Bioconductor Core Team
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> Bioc-devel using r-project.org  mailing list
> https://stat.ethz.ch/mailman/listinfo/bioc-devel

-- 
Hervé Pagès

Bioconductor Core Team
hpages.on.github using gmail.com

	[[alternative HTML version deleted]]