[R-pkg-devel] Problem with using parallel runtime system with R

[Sameh M. Abdulah]

I will try to talk to my team to fork it for more investigation from your side.

However, this is the story.  We have a C package which uses a STARPU runtime system (http://starpu.gforge.inria.fr/doc/starpu.pdf) to facility the parallelization through a set of task-based parallel algorithms. Once you execute
Your program StarPU created several threads to parallelize your tasks based on a specific scheduling mechanism. 

In the case of R, R is used only one process and if I used the default behavior without any modification, all the StarPU threads are using the same physical core which is used by the default R process. We added
A simple modification which helps to well spawn the threads to the other available physical cores by disabling the KMP_AFFINITY (Sys.setenv(KMP_AFFINITY= "disabled")). However,
Even the performance is improved, the R package is about 5X slow up compared to the C version. The difference we found that In case of R different threads is controlled by R which means that we can see from top 
Command that R is using n threads on n cores. In the case of C package, StarPU is using these threads.  One more difference from htop command that the virtual memory usage of the R package is doubled which we cannot 
Understand or even know if it is related to the performance issue or not 



--Sameh



On 5/10/19, 6:12 PM, "Iñaki Ucar" <iucar using fedoraproject.org> wrote:

    On Fri, 10 May 2019 at 17:06, Sameh M. Abdulah
    <sameh.abdulah using kaust.edu.sa> wrote:
    >
    > Actually not yet. It is on Github but in a private repo. If it helps I can fork it for public access for now.
    
    Providing the code always improves your chances of getting help.
    Otherwise, the information is probably too limited.
    
    Iñaki