[R-pkg-devel] Setting OpenMP threads (globally) for an R package

[Vladimir Dergachev]

On Fri, 18 Mar 2022, Simon Urbanek wrote:

> Evan,
>
> honestly, I think your request may be a red herring. Threads typically don't cause memory explosion, because OpenMP threads don't allocate new memory, but uncontrolled forking does. There are many things that are not allowed inside mclapply so that's where I would look. It may be better to look at the root cause first, but for that we would need more details on what you are doing.

Well, actually this is a real issue as glibc has a concept of "memory 
arenas".

When using old-style heap allocation a multi-threaded program would 
have to obtain a lock to the memory allocator - which is a problem in 
object-oriented code that constantly allocates and deallocates small 
pieces of memory.

The solution was to have multiple "arenas" from which memory can be 
allocated and to prevent lock contention. Which works ok when each thread 
uses the same set of objects, but breaks when doing openmp() style 
computation where tasks are assigned to compute threads at random.

The memory footprint of the program can easily grow to the number of 
memory arenas times whatever the footprint was in single-threaded case.

You can force glibc to use heap only with this environment setting:

MALLOC_ARENA_MAX=1

If this happens to make your program slower, this is an indication that 
the program calls memory allocation function too frequently and needs to 
be optimized. This tends to improve both multi-threaded and 
single-threaded performance as memory allocation calls are rather slow 
even without a lock.

best

Vladimir Dergachev

>
> Cheers,
> Simon
>
>