[Bioc-devel] Fw: MungeSumstats Bioconductor

[Hervé Pagès]

The longtests builds only run the tests in the longtests/ folder.


Best,

H.


On 2/21/24 09:56, alan murphy wrote:
> Hey Herve,
>
> Thanks for this, I'm going to implement that now. One thing I 
> wondered, is it only the tests in the longtest/ folder that run on 
> these weekly builds or both those and the tests in test/? I'm 
> wondering should I duplicate the tests in both folders that I want to 
> run in both?
>
> Cheers,
> Alan.
> ------------------------------------------------------------------------
> *From:* Hervé Pagès <hpages.on.github using gmail.com>
> *Sent:* Tuesday 6 February 2024 18:58
> *To:* alan murphy <alanmurph94 using hotmail.com>; Bioc-devel using r-project.org 
> <bioc-devel using r-project.org>
> *Subject:* Re: [Bioc-devel] Fw: MungeSumstats Bioconductor
>
> Hi Alan,
>
> The specs of the Linux servers have not changed.
>
> However we've recently observed some of these random kills by the 
> Linux kernel for a couple of other packages, and we started an 
> application to get funding to increase the amount of RAM on these 
> machines.
>
> These random kills are a desperate attempt by the Linux kernel to keep 
> the machine alive when it's under extremely high load and running out 
> of resources. Unfortunately this is not something that anybody will be 
> able to easily reproduce as it only happens under special 
> circumstances e.g. during the daily builds when dozens of packages are 
> being checked simultaneously.
>
> Regardless of whether we'll manage to increase the memory on the Linux 
> servers, it seems to me that the most memory-hungry unit tests in 
> MungeSumstats would be a better fit for the long tests. These tests 
> are run once a week instead of daily, and use less workers (4 instead 
> of 28). This means that each package has access to more resources.
>
> See https://contributions.bioconductor.org/long-tests.html 
> <https://contributions.bioconductor.org/long-tests.html> for how to 
> set up the long tests in your package.
>
> Let me know if you have questions or need help with this.
>
> Best,
>
> H.
>
> On 2/2/24 09:14, alan murphy wrote:
>> Hi,
>>
>> I'm the maintainer of the MungeSumstats package which is currently failing on the devel nebbiolo1 Linux platform because of RAM requirements of the unit tests. These tests use large reference datasets (like BSgenome.Hsapiens.1000genomes.hs37d5::BSgenome.Hsapiens.1000genomes.hs37d5) which cause the issue, e.g. from the output of the devel linux test<https://bioconductor.org/checkResults/3.19/bioc-LATEST/MungeSumstats/nebbiolo1-checksrc.html>  <https://bioconductor.org/checkResults/3.19/bioc-LATEST/MungeSumstats/nebbiolo1-checksrc.html>:
>>
>> ```
>>
>> Validating RSIDs of 92 SNPs using BSgenome::snpsById...
>> Killed
>>
>> ```
>> which is from this function:https://github.com/neurogenomics/MungeSumstats/blob/cccf77b2249f52be59fe1749f13a386ffaaae528/R/load_ref_genome_data.R#L49  <https://github.com/neurogenomics/MungeSumstats/blob/cccf77b2249f52be59fe1749f13a386ffaaae528/R/load_ref_genome_data.R#L49>
>>
>> I would like to keep these units tests as they are very important for me to know if things have gone wrong. I can't scale down the RAM usage, it is only using a few rows of data as it, the issue is that the reference sets are massive rather than the actual set being tested. I currently don't have a lot of these tests set to run on windows/mac platforms so I rely on the Linux machines to run this. I am not sure what has changed but a few releases back, the same tested were not causing this issue and would complete on linux - has the machine spec changed? Is there any chance RAM could be increased for these tests or is there a way to specify not to run the specific unit tests on the Bioconductor server so I can at least keep these for my github actions workflows? See below for the automated message I got about these errors.
>>
>> Any other suggestions on ways around this would be greatly appreciated!
>>
>> Thanks,
>> Alan.
>> ________________________________
>> From: CoreTeam Bioconductor<bioconductorcoreteam using gmail.com>  <mailto:bioconductorcoreteam using gmail.com>
>> Sent: Friday 2 February 2024 16:43
>> To:alanmurph94 using hotmail.com  <mailto:alanmurph94 using hotmail.com>  <alanmurph94 using hotmail.com>  <mailto:alanmurph94 using hotmail.com>
>> Subject: MungeSumstats Bioconductor
>>
>> Hello Package Maintainer,
>>
>> We would like to bring to your attention that your package is failing in devel on the linux platform. This is very problematic. Please investigate the issues and fix the package to avoid deprecation.
>>
>> https://bioconductor.org/checkResults/devel/bioc-LATEST/  <https://bioconductor.org/checkResults/devel/bioc-LATEST/>
>> <https://bioconductor.org/checkResults/3.17/workflows-LATEST/TCGAWorkflow>  <https://bioconductor.org/checkResults/3.17/workflows-LATEST/TCGAWorkflow>
>>
>> If you have further questions or concerns please reach out on thebioc-devel using r-project.org  <mailto:bioc-devel using r-project.org><mailto:bioc-devel using r-project.org>  <mailto:bioc-devel using r-project.org>
>>
>> We appreciate your quick attention to this matter
>>
>> Cheers,
>> On behalf of the Bioconductor Core Team
>>
>> 	[[alternative HTML version deleted]]
>>
>> _______________________________________________
>> Bioc-devel using r-project.org  <mailto:Bioc-devel using r-project.org>  mailing list
>> https://stat.ethz.ch/mailman/listinfo/bioc-devel  <https://stat.ethz.ch/mailman/listinfo/bioc-devel>
> -- 
> Hervé Pagès
>
> Bioconductor Core Team
> hpages.on.github using gmail.com  <mailto:hpages.on.github using gmail.com>

-- 
Hervé Pagès

Bioconductor Core Team
hpages.on.github using gmail.com

	[[alternative HTML version deleted]]