[R-pkg-devel] RFC: an interface to manage use of parallelism in packages

[Ivan Krylov]

В Wed, 25 Oct 2023 13:54:53 -0700
"Reed A. Cartwright" <racartwright using gmail.com> пишет:

> For a comparison, I'd recommend looking at how GNU make does parallel
> processing. It uses the concept of job server and job slots. What I
> like about it is that it is implemented at the OS level because make
> needs to support interacting with non-make processes. On Windows it
> uses a named semaphore, and on Unix-like system it uses named pipes
> or simple pipes to pass tokens around.

Thank you for pointing me towards the job server in GNU make! This is
exactly the kind of suggestion I was looking for. I also appreciate you
signing up for an account on Codeberg to give me a more detailed
writeup.

I agree that named semaphores seem to be a better fit for the job than
my first design. There are some corner cases to handle (what if the
parent process starts up and the semaphore already exists? what if the
user wants to reduce the allowance but the semaphore count is lower than
expected? what are the stongest restrictions on a POSIX semaphore name?
I've seen implementations placing them under /dev/shm%s and /tmp/%s.sem
so far), but it should be possible for me to implement the core concept
in a few days.

Additional care might be needed regarding the number of BLAS threads. R
might have to become its own FlexiBLAS and pass an additional
environment variable to children to ensure the limit being taken into
account.

-- 
Best regards,
Ivan