[R-sig-hpc] parallel and openblas

[Claudia Beleites]

I agree with Simon.

- Not restricting the children doesn't lead to problems with both
implicit and explicit parallelization for me:
E.g. if I decide to run explicitly 3 parallel threads and have 12 cores,
I use GOTO_NUM_THREADS 4, and that works nicely.

- if users do not have access to taskset, then restriction to 1 core is
certainly more problematic than no restriction.

Claudia



Am 25.04.2012 17:21, schrieb Simon Urbanek:
> 
> On Apr 25, 2012, at 11:10 AM, Dirk Eddelbuettel wrote:
> 
>>
>> Sorry to follow-up on my own post from minutes ago, but maybe it is better
>> the way it is as it allows to explicitly spread tasks each of which would
>> limit itself to one core.  Isn't this the best way to avoid "clogging" when
>> we use, say, snow to get to eight cores, and each of the eight cores wants to
>> do BLAS work on eight cores...  That was after all the initial issue in
>> Martin's email.
>>
>> And when I use the affinity trick (now breaking the single line for display)
>>
>>  edd at max:~$ r -e 'library(parallel); \
>>                   system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())); \
>>                   cores <- detectCores(); \
>>                   print(cores); \
>>                   mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)'
>>  pid 24436's current affinity mask: 1
>>  pid 24436's new affinity mask: ff
>>  [1] 8
>>
>> all is well -- eight cores humming along just fine per htop under OpenBLAS.  
>>
>> Maybe this is actually better as it effectively gives us a run-time toggle?
>>
> 
> I'd argue that the problem is affinity of 1 by default - that is certainly not what you want or expect and bad in most cases. It would make much more sense to restrict the *children* to one CPU each rather than the parent process so that forked processes stay on their cores - that is IMHO the only place where setting affinity makes any sense at all. Also the system(..) trick is ugly and highly system-specific (you don't even know if you can access taskset). I would consider OpenBLAS' affinity setting as a pretty bad bug (certainly form user's point of view).
> 
> Cheers,
> Simon
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc


-- 
Claudia Beleites
Spectroscopy/Imaging
Institute of Photonic Technology
Albert-Einstein-Str. 9
07745 Jena
Germany

email: claudia.beleites at ipht-jena.de
phone: +49 3641 206-133
fax:   +49 2641 206-399