[R-pkg-devel] Too many cores used in examples (not caused by data.table)

[Dirk Eddelbuettel]

On 19 October 2023 at 05:57, Helske, Jouni wrote:
| I am having difficulties in getting the latest version of the bssm (https://github.com/helske/bssm) package to CRAN, as the pretest issues a NOTE that the package uses too many cores in some of the examples ("Examples with CPU time > 2.5 times elapsed time"). I've seen plenty of discussion about this issue in relation to the data.table package, but bssm does not use it. Also, while bssm uses OpenMP in some functions, these are not called in the example in question (?exchange), and by default the number of threads in the parallelisable functions is set to 1.
| 
| But I just realised that bssm uses Armadillo via RcppArmadillo, which uses OpenMP by default for some elementwise operations. So, I wonder if that could be the culprit? However, I would think that in such case there would be many other packages with RcppArmadillo encountering the same CRAN issues. Has anyone experienced this with their packages which use RcppArmadillo but not data.table, or can say whether my guess is correct? I haven't been able to reproduce the issue myself on r-hub or my own linux, so I can't really test whether setting #define ARMA_DONT_USE_OPENMP helps.

You have some options to control OpenMP.

There is an environment variable (OMP_THREAD_LIMIT), and there is an CRAN
add-on package (RhpcBLASctl) which, if memory serves, also sets this. Looking
at the Armadillo documentation we see another variable (ARMA_OPENMP_THREADS).

I really think CRAN made a mistake here pushing this down on all package
maintainers.  It is too much work, some will get frustrated, some will get it
wrong and I fear in aggregate we end up with less performant software (as
some will 'cave in' and hard-wire single threaded computes). 

Dirk

-- 
dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org