[R-pkg-devel] [Tagged] Re: multithreading in packages

[Viechtbauer, Wolfgang (SP)]

One thing I did not see mentioned in this thread (pun intended) so far:

For what kind of computations is multithreading supposed to be used within the package being developed? If the computations involve a lot of linear/matrix algebra, then one could just use R with other linear algebra routines (e.g., OpenBLAS, Atlas, MKL, BLIS) and get the performance benefits of multicore processing of those computations without having to change a single line of code in the package (although in my experience, most of the performance benefits come from switching to something like OpenBLAS and using it single-threaded).

This aside, I am personally more in favor of explicitly parallelizing those things that are known to be embarrassingly parallelizable using packages like parallel, future, etc. since a package author should know best when these situations arise and can take the necessary steps to parallelize those computations -- but making the use of parallel processing in these cases an option, not a default. I have seen way too many cases in HPC environments where jobs are being parallelized, the package is doing parallel processing, and multicore linear algebra routines are being used all simultaneously, which is just a disaster.

Finally, I don't think the HPC task view has been mentioned so far:

https://cran.r-project.org/web/views/HighPerformanceComputing.html

(not even by Dirk just now, who maintains it!)

Best,
Wolfgang

>-----Original Message-----
>From: R-package-devel [mailto:r-package-devel-bounces using r-project.org] On Behalf Of
>Dirk Eddelbuettel
>Sent: Saturday, 09 October, 2021 18:33
>To: Ben Bolker
>Cc: r-package-devel using r-project.org
>Subject: Re: [R-pkg-devel] [Tagged] Re: multithreading in packages
>
>
>On 9 October 2021 at 12:08, Ben Bolker wrote:
>|    FWIW there is some machinery in the glmmTMB package for querying,
>| setting, etc. the number of OpenMP threads.
>|
>| https://github.com/glmmTMB/glmmTMB/search?q=omp
>
>https://cloud.r-project.org/package=RhpcBLASctl
>
>Dirk
>
>--
>https://dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org
>
>______________________________________________
>R-package-devel using r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/r-package-devel