[R-sig-hpc] parallel and openblas
simon.urbanek at r-project.org
Wed Apr 25 17:21:27 CEST 2012
On Apr 25, 2012, at 11:10 AM, Dirk Eddelbuettel wrote:
> Sorry to follow-up on my own post from minutes ago, but maybe it is better
> the way it is as it allows to explicitly spread tasks each of which would
> limit itself to one core. Isn't this the best way to avoid "clogging" when
> we use, say, snow to get to eight cores, and each of the eight cores wants to
> do BLAS work on eight cores... That was after all the initial issue in
> Martin's email.
> And when I use the affinity trick (now breaking the single line for display)
> edd at max:~$ r -e 'library(parallel); \
> system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())); \
> cores <- detectCores(); \
> print(cores); \
> mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)'
> pid 24436's current affinity mask: 1
> pid 24436's new affinity mask: ff
>  8
> all is well -- eight cores humming along just fine per htop under OpenBLAS.
> Maybe this is actually better as it effectively gives us a run-time toggle?
I'd argue that the problem is affinity of 1 by default - that is certainly not what you want or expect and bad in most cases. It would make much more sense to restrict the *children* to one CPU each rather than the parent process so that forked processes stay on their cores - that is IMHO the only place where setting affinity makes any sense at all. Also the system(..) trick is ugly and highly system-specific (you don't even know if you can access taskset). I would consider OpenBLAS' affinity setting as a pretty bad bug (certainly form user's point of view).
More information about the R-sig-hpc