[R-sig-Debian] Foreach %dopar% operator incorrectly load balancing

[Sean Southard]

Dear R-SIG-Debian,

I am using R version 3.1.2 with rstudio-server 0.98.113 on debian build
3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u1 x86_64 GNU/Linux.

I often use the %dopar% operator in from the foreach package to run code in
parallel. However, the only other use on the box seemingly installed a few
items and suddenly %dopar% will use far more than the number of cores I am
specifying and seems to load balance between all of them. The issue with
this is that it doesn't actually seem to perform non-trivial tasks at all
anymore.

This is an example of testing code I've been using for testing the %dopar%
 loop.

library(iterators)
library(foreach)
library(doParallel)
library(Parallel)
nCores <- 4

cl <- makeCluster(nCores)
registerDoParallel(cl)
trials = 100000

x <- iris[which(iris[,5] != "setosa"),c(1,5)]
    t2 <- system.time({
    r2 <- foreach(icount(trials), .combine=cbind) %dopar% {
    ind <- sample(100,100,replace= TRUE)
    results1 <- glm(x[ind,2]~x[ind,1],family=binomial(logit))
    coefficients(results1)
   }})[3]

stopCluster(cl)

Another interesting behavior is that I can now stop R code that should be
running on the slave workers where as previously I had to use a kill command
to handle the workers.

I have checked the system logs and can't seem to find what's been changed
so it's unclear to me exactly how to proceed. It is almost reminiscent of a
change in the BLAS library but this behavior persists after recompiling R
to not use the shared BLAS lib.

here is the output from lsof -p 23618 | grep 'blas\|lapack'

R       39781 smsouthard  mem    REG               8,17  3576576
1055038 /usr/local/lib/R/lib/libRlapack.so
R       39781 smsouthard  mem    REG               8,17   655336
1572936 /usr/lib/libblas/libblas.so.3.0

Output from R CMD ldd /usr/local/lib/R/bin/exec/R

linux-vdso.so.1 (0x00007ffca59fd000)
libR.so => /usr/local/lib/R/lib/libR.so (0x00007f15fdc30000)
libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f15fd9f7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f15fd7da000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15fd431000)
libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f15fd190000)
libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3
(0x00007f15fce72000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15fcb71000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
(0x00007f15fc933000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f15fc6e9000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f15fc4c6000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f15fc2bd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f15fc0b9000)
libicuuc.so.48 => /usr/lib/x86_64-linux-gnu/libicuuc.so.48 (0x00007f15fbd4a000)
libicui18n.so.48 => /usr/lib/x86_64-linux-gnu/libicui18n.so.48
(0x00007f15fb97e000)/lib64/ld-linux-x86-64.so.2 (0x00007f15fe1cf000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f15fb768000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f15fb53d000)
libicudata.so.48 => /usr/lib/x86_64-linux-gnu/libicudata.so.48
(0x00007f15fa1cd000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15f9ec2000)

Is it possible to rebuild R without any shared library's? I am open to any
suggestions at this point.

Thanks for your help guys!

	[[alternative HTML version deleted]]