[R-sig-hpc] parallel and openblas

[Dirk Eddelbuettel]

[ Sylvestre: I am tossing you in the middle of a thread here. We may have a
buglet in OpenBLAS where NO_AFFINITY=1 might be a good config value to add. ]

Steve,

Nice work, and mostly confirming against OpenBLAS, Atlas and a local (old)
GotoBLAS2. 

On 25 April 2012 at 10:24, Stephen Weston wrote:
| I was able to confirm that when I built R using OpenBLAS on my
| Linux machine, my CPU affinity was modified right at the
| beginning of the R session:
| 
|   $ grep Cpus_allowed /proc/self/status
|   Cpus_allowed: ffffffff,ffffffff
|   Cpus_allowed_list:    0-63
|   $ bin/R
|   > readLines('/proc/self/status')[32]
|   [1] "Cpus_allowed:\t00000000,00000001"

What kernel is that?  On 3.0.0-17 (Ubuntu 11.10, infrequently rebooted) I get 

  edd at max:~$ grep Cpus_allowed /proc/self/status
  Cpus_allowed:   ff
  Cpus_allowed_list:      0-7
  edd at max:~$

| I then confirmed that this causes problems for parallel packages
| such as "parallel" by trying to use all six cores of my machine
| using the "mclapply" function:
| 
|   > library(parallel)
|   > cores <- detectCores()
|   > mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)
| 
| When I executed "top" from another window and pressed "1", it
| showed that only one core was being used, and there were six R
| sessions, each getting 17% of the CPU.

When I run these three commands as a single line for r (from the littler package)

  edd at max:~$ r -e 'library(parallel);  cores <- detectCores(); print(cores); mclapply(1:cores, function(i) repeat sqrt(3.14159), mc.cores=cores)'
  [1] 8
  ^C
  edd at max:~$ 

I also get just one core covered. That is with 

  edd at max:~$ COLUMNS=94 dpkg -l|grep "blas\|atlas" | cut -c-78
  ii  gotoblas2-helper  0.1-12.local.1    GotoBLAS2 helper
  ii  libblas-dev       1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st
  ii  libblas-test      1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te
  ii  libblas3gf        1.2.20110419-2ubu Basic Linear Algebra Reference impleme
  ii  libopenblas-base  0.1alpha2.2-3     Optimized BLAS (linear algebra) librar
  ii  libopenblas-dev   0.1alpha2.2-3     Optimized BLAS (linear algebra) librar
  edd at max:~$ 

where OpenBLAS provides BLAS as default.

That was after I had removed Atlas which is still my default. So if I
reiinstall Atlas (which "ranks higher" in the defaults and hence replaces
OpenBLAS) everything is fine -- eight cores used.

  edd at max:~$ COLUMNS=94 dpkg -l|grep "blas\|atlas" | cut -c-78
  ii  gotoblas2-helper  0.1-12.local.1    GotoBLAS2 helper
  ii  libatlas-base-dev 3.8.4-3build1     Automatically Tuned Linear Algebra Sof
  ii  libatlas-dev      3.8.4-3build1     Automatically Tuned Linear Algebra Sof
  ii  libatlas3gf-base  3.8.4-3build1     Automatically Tuned Linear Algebra Sof
  ii  libblas-dev       1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st
  ii  libblas-test      1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te
  ii  libblas3gf        1.2.20110419-2ubu Basic Linear Algebra Reference impleme
  ii  libopenblas-base  0.1alpha2.2-3     Optimized BLAS (linear algebra) librar
  ii  libopenblas-dev   0.1alpha2.2-3     Optimized BLAS (linear algebra) librar
  edd at max:~$ 
 
 
| I also confirmed that "Cpus_allowed" was being set to the same
| value for each of the workers:
| 
|   > mclapply(1:cores, function(i) readLines('/proc/self/status')[32],
| mc.cores=cores)
|   [[1]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
|   [[2]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
|   [[3]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
|   [[4]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
|   [[5]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
|   [[6]]
|   [1] "Cpus_allowed:\t00000000,00000001"
| 
| That is definitely not what you want to see, and explains why
| "mclapply" is only able to use one core.
| 
| When I rebuilt and reinstalled OpenBLAS after editing
| Makefile.rule so that it contained the line:
| 
|   NO_AFFINITY = 1
| 
| and then restarted R, the problem went away:
| 
|   $ bin/R
|   > readLines('/proc/self/status')[32]
|   [1] "Cpus_allowed:\tffffffff,ffffffff"
| 
| This time when I ran "mclapply", "top" confirmed that I was
| using all six cores at about 100%.
| 
| I didn't try this experiment with the older GotoBLAS2, but I
| believe the results would be the same.

I can confirm this. Using the packages 

edd at max:~$ COLUMNS=94 dpkg -l|grep "blas\|atlas" | cut -c-78
ii  gotoblas2         1.13-1            GotoBLAS2
ii  gotoblas2-helper  0.1-12.local.1    GotoBLAS2 helper
ii  libblas-dev       1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, st
ii  libblas-test      1.2.20110419-2ubu Basic Linear Algebra Subroutines 3, te
ii  libblas3gf        1.2.20110419-2ubu Basic Linear Algebra Reference impleme
edd at max:~$

where the GotoBLAS2 (locally built, using the gotoblas2-helper package) now
provide BLAS, everything sticks to one core when running the mclapply.  

I guess I'd need to fix gotoblas2-helper and rebuild the gotoblas2.  Or stick
with / hope for a corrected OpenBLAS build.

Dirk
 
| - Steve
| 
| 
| On Tue, Apr 24, 2012 at 8:37 PM, Dirk Eddelbuettel <edd at debian.org> wrote:
| >
| > On 24 April 2012 at 15:45, Stephen Weston wrote:
| > | There's an interesting discussion entitled "all processes run on
| > | one CPU core" at:
| > |
| > |     https://github.com/ipython/ipython/issues/840
| > |
| > | Someone was experiencing a very similar problem to the one that
| > | Claudia described using GotoBLAS2 with IPython and NumPy.
| > | Apparently it was fixed by recompiling GotoBLAS2 with the
| > | "NO_AFFINITY" parameter set to "1" in Makefile.rule, and then
| > | rebuilding "NumPy".
| > |
| > | It seems pretty strange, but GotoBLAS2/OpenBLAS may be modifying
| > | the affinity of the R process by calling sched_setaffinity() when
| > | it is initialized, and that is causing the problems that Claudia
| > | and Martin have seen.
| > |
| > | So perhaps the solution is to recompile GotoBLAS2/OpenBLAS with
| > | NO_AFFINITY=1, and then rebuild R with it.
| >
| > Good discussion, but one important nit: never a need to rebuild a R (provided
| > you have external / dynamically linked BLAS).
| >
| > Just restart R.
| >
| > Dirk
| >
| > --
| > R/Finance 2012 Conference on May 11 and 12, 2012 at UIC in Chicago, IL
| > See agenda, registration details and more at http://www.RinFinance.com

-- 
R/Finance 2012 Conference on May 11 and 12, 2012 at UIC in Chicago, IL
See agenda, registration details and more at http://www.RinFinance.com