[R-sig-hpc] parallel and openblas

[Simon Urbanek]

On Apr 25, 2012, at 11:10 AM, Dirk Eddelbuettel wrote:

> 
> Sorry to follow-up on my own post from minutes ago, but maybe it is better
> the way it is as it allows to explicitly spread tasks each of which would
> limit itself to one core.  Isn't this the best way to avoid "clogging" when
> we use, say, snow to get to eight cores, and each of the eight cores wants to
> do BLAS work on eight cores...  That was after all the initial issue in
> Martin's email.
> 
> And when I use the affinity trick (now breaking the single line for display)
> 
>  edd at max:~$ r -e 'library(parallel); \
>                   system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())); \
>                   cores <- detectCores(); \
>                   print(cores); \
>                   mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)'
>  pid 24436's current affinity mask: 1
>  pid 24436's new affinity mask: ff
>  [1] 8
> 
> all is well -- eight cores humming along just fine per htop under OpenBLAS.  
> 
> Maybe this is actually better as it effectively gives us a run-time toggle?
> 

I'd argue that the problem is affinity of 1 by default - that is certainly not what you want or expect and bad in most cases. It would make much more sense to restrict the *children* to one CPU each rather than the parent process so that forked processes stay on their cores - that is IMHO the only place where setting affinity makes any sense at all. Also the system(..) trick is ugly and highly system-specific (you don't even know if you can access taskset). I would consider OpenBLAS' affinity setting as a pretty bad bug (certainly form user's point of view).

Cheers,
Simon