[R-pkg-devel] Problem with using parallel runtime system with R
Sameh M. Abdulah
@@meh@@bdu|@h @end|ng |rom k@u@t@edu@@@
Fri May 10 17:37:57 CEST 2019
I will try to talk to my team to fork it for more investigation from your side.
However, this is the story. We have a C package which uses a STARPU runtime system (http://starpu.gforge.inria.fr/doc/starpu.pdf) to facility the parallelization through a set of task-based parallel algorithms. Once you execute
Your program StarPU created several threads to parallelize your tasks based on a specific scheduling mechanism.
In the case of R, R is used only one process and if I used the default behavior without any modification, all the StarPU threads are using the same physical core which is used by the default R process. We added
A simple modification which helps to well spawn the threads to the other available physical cores by disabling the KMP_AFFINITY (Sys.setenv(KMP_AFFINITY= "disabled")). However,
Even the performance is improved, the R package is about 5X slow up compared to the C version. The difference we found that In case of R different threads is controlled by R which means that we can see from top
Command that R is using n threads on n cores. In the case of C package, StarPU is using these threads. One more difference from htop command that the virtual memory usage of the R package is doubled which we cannot
Understand or even know if it is related to the performance issue or not
On 5/10/19, 6:12 PM, "Iñaki Ucar" <iucar using fedoraproject.org> wrote:
On Fri, 10 May 2019 at 17:06, Sameh M. Abdulah
<sameh.abdulah using kaust.edu.sa> wrote:
> Actually not yet. It is on Github but in a private repo. If it helps I can fork it for public access for now.
Providing the code always improves your chances of getting help.
Otherwise, the information is probably too limited.
More information about the R-package-devel