[R-sig-hpc] parallel and openblas

[Stephen Weston]

I was able to confirm that when I built R using OpenBLAS on my
Linux machine, my CPU affinity was modified right at the
beginning of the R session:

  $ grep Cpus_allowed /proc/self/status
  Cpus_allowed: ffffffff,ffffffff
  Cpus_allowed_list:    0-63
  $ bin/R
  > readLines('/proc/self/status')[32]
  [1] "Cpus_allowed:\t00000000,00000001"

I then confirmed that this causes problems for parallel packages
such as "parallel" by trying to use all six cores of my machine
using the "mclapply" function:

  > library(parallel)
  > cores <- detectCores()
  > mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)

When I executed "top" from another window and pressed "1", it
showed that only one core was being used, and there were six R
sessions, each getting 17% of the CPU.

I also confirmed that "Cpus_allowed" was being set to the same
value for each of the workers:

  > mclapply(1:cores, function(i) readLines('/proc/self/status')[32],
mc.cores=cores)
  [[1]]
  [1] "Cpus_allowed:\t00000000,00000001"

  [[2]]
  [1] "Cpus_allowed:\t00000000,00000001"

  [[3]]
  [1] "Cpus_allowed:\t00000000,00000001"

  [[4]]
  [1] "Cpus_allowed:\t00000000,00000001"

  [[5]]
  [1] "Cpus_allowed:\t00000000,00000001"

  [[6]]
  [1] "Cpus_allowed:\t00000000,00000001"

That is definitely not what you want to see, and explains why
"mclapply" is only able to use one core.

When I rebuilt and reinstalled OpenBLAS after editing
Makefile.rule so that it contained the line:

  NO_AFFINITY = 1

and then restarted R, the problem went away:

  $ bin/R
  > readLines('/proc/self/status')[32]
  [1] "Cpus_allowed:\tffffffff,ffffffff"

This time when I ran "mclapply", "top" confirmed that I was
using all six cores at about 100%.

I didn't try this experiment with the older GotoBLAS2, but I
believe the results would be the same.

- Steve


On Tue, Apr 24, 2012 at 8:37 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
>
> On 24 April 2012 at 15:45, Stephen Weston wrote:
> | There's an interesting discussion entitled "all processes run on
> | one CPU core" at:
> |
> |     https://github.com/ipython/ipython/issues/840
> |
> | Someone was experiencing a very similar problem to the one that
> | Claudia described using GotoBLAS2 with IPython and NumPy.
> | Apparently it was fixed by recompiling GotoBLAS2 with the
> | "NO_AFFINITY" parameter set to "1" in Makefile.rule, and then
> | rebuilding "NumPy".
> |
> | It seems pretty strange, but GotoBLAS2/OpenBLAS may be modifying
> | the affinity of the R process by calling sched_setaffinity() when
> | it is initialized, and that is causing the problems that Claudia
> | and Martin have seen.
> |
> | So perhaps the solution is to recompile GotoBLAS2/OpenBLAS with
> | NO_AFFINITY=1, and then rebuild R with it.
>
> Good discussion, but one important nit: never a need to rebuild a R (provided
> you have external / dynamically linked BLAS).
>
> Just restart R.
>
> Dirk
>
> --
> R/Finance 2012 Conference on May 11 and 12, 2012 at UIC in Chicago, IL
> See agenda, registration details and more at http://www.RinFinance.com