[R-pkg-devel] Setting OpenMP threads (globally) for an R package

[Evan Biederstedt]

Hi Simon

I really appreciate the help, thanks for the message.
I think uncontrolled forking could be the issue, though I don't see all
cores used via `htop`; I just see the memory quickly surge.


*> There are many things that are not allowed inside mclapply so that's
where I would look. *
Could you detail this a bit more? This could be what's happening....


*>Threads typically don't cause memory explosion, because OpenMP threads
don't allocate new memory, but uncontrolled forking does*
Do you have insight on how to explicitly limit forking? It looks like
Henrik had been thinking about this earlier:
https://github.com/HenrikBengtsson/Wishlist-for-R/issues/94

Moreover, could you explain how setting the OpenMP global variables e.g.
`OMP_NUM_THREADS=1` would stop forking? I don't quite follow this.


*> It may be better to look at the root cause first, but for that we would
need more details on what you are doing.*
Functions with mclapply do indeed show this "memory surging" behavior, e.g.

https://github.com/kharchenkolab/numbat/blob/main/R/main.R#L940-L963


Thanks, Evan

On Thu, Mar 17, 2022 at 7:23 PM Simon Urbanek <simon.urbanek using r-project.org>
wrote:

> Evan,
>
> honestly, I think your request may be a red herring. Threads typically
> don't cause memory explosion, because OpenMP threads don't allocate new
> memory, but uncontrolled forking does. There are many things that are not
> allowed inside mclapply so that's where I would look. It may be better to
> look at the root cause first, but for that we would need more details on
> what you are doing.
>
> Cheers,
> Simon
>
>
> > On Mar 18, 2022, at 2:51 AM, Evan Biederstedt <
> evan.biederstedt using gmail.com> wrote:
> >
> > Hi R-package-devel
> >
> > I'm developing an R package which uses `parallel::mclapply` and several
> > other library dependencies which possibly rely upon OpenMP.
> Unfortunately,
> > some functions explode the amount of memory used.
> >
> > I've noticed that if I set `export OMP_NUM_THREADS=1` before starting R,
> > the memory is far more manageable.
> >
> > My question is, if there a way for me to achieve this behavior within
> the R
> > package itself?
> >
> > My initial try was to use `R/zzz.R` and an `.onLoad()` function to load
> > these global variables upon loading the library.
> >
> > ```
> > .onLoad <- function(libname, pkgname){
> >  Sys.setenv(OMP_NUM_THREADS=1)
> > }
> > ```
> >
> > But this doesn't work. The memory still explodes. In fact, I'm worried
> that
> > this cannot be done within an R package itself, as R has already started,
> > e.g.  https://stackoverflow.com/a/27320691/5269850
> >
> > Is there a recommended approach for this problem when writing R packages?
> >
> > Package here: https://github.com/kharchenkolab/numbat
> >
> > Related question on SO:
> >
> https://stackoverflow.com/questions/71507979/set-openmp-threads-for-all-dependencies-in-r-package
> >
> > Any help appreciated. Thanks, Evan
> >
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-package-devel using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-package-devel
> >
>
>

	[[alternative HTML version deleted]]