[R-sig-hpc] parallel and openblas
claudia.beleites at ipht-jena.de
Wed Apr 25 18:54:03 CEST 2012
I agree with Simon.
- Not restricting the children doesn't lead to problems with both
implicit and explicit parallelization for me:
E.g. if I decide to run explicitly 3 parallel threads and have 12 cores,
I use GOTO_NUM_THREADS 4, and that works nicely.
- if users do not have access to taskset, then restriction to 1 core is
certainly more problematic than no restriction.
Am 25.04.2012 17:21, schrieb Simon Urbanek:
> On Apr 25, 2012, at 11:10 AM, Dirk Eddelbuettel wrote:
>> Sorry to follow-up on my own post from minutes ago, but maybe it is better
>> the way it is as it allows to explicitly spread tasks each of which would
>> limit itself to one core. Isn't this the best way to avoid "clogging" when
>> we use, say, snow to get to eight cores, and each of the eight cores wants to
>> do BLAS work on eight cores... That was after all the initial issue in
>> Martin's email.
>> And when I use the affinity trick (now breaking the single line for display)
>> edd at max:~$ r -e 'library(parallel); \
>> system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())); \
>> cores <- detectCores(); \
>> print(cores); \
>> mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)'
>> pid 24436's current affinity mask: 1
>> pid 24436's new affinity mask: ff
>>  8
>> all is well -- eight cores humming along just fine per htop under OpenBLAS.
>> Maybe this is actually better as it effectively gives us a run-time toggle?
> I'd argue that the problem is affinity of 1 by default - that is certainly not what you want or expect and bad in most cases. It would make much more sense to restrict the *children* to one CPU each rather than the parent process so that forked processes stay on their cores - that is IMHO the only place where setting affinity makes any sense at all. Also the system(..) trick is ugly and highly system-specific (you don't even know if you can access taskset). I would consider OpenBLAS' affinity setting as a pretty bad bug (certainly form user's point of view).
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
Institute of Photonic Technology
email: claudia.beleites at ipht-jena.de
phone: +49 3641 206-133
fax: +49 2641 206-399
More information about the R-sig-hpc