[R-sig-hpc] foreach + doMC not fully parallel?

[Brian D Peyser PhD]

On Thu, 2010-08-26 at 18:35 -0400, Sean Davis wrote: 
> Assuming there is nothing else running on the system? 
> 
> 
> In any case, it appears that you are reading text files and writing to
> disk.  Only one core can use the IO at a time, so that may explain
> this.  Try running something entirely computational and see if you can
> get close to 100% on all four cores.  
> 
> 
> Sean
>  

I hadn't thought about IO, but the fact that the processes always add up
to almost 100% of 1/2/3 cores makes me think that something is
preventing them from computing on all the cores rather than just waiting
for I/O. The I/O is actually very minor, but I took out all of it and
still had the same result.

So next I decided to stop my other resource hogs, such as Firefox and
Evolution. that had no impact. I also ran from the terminal instead of
from Emacs, with the same result.

What's strange is the usage will vary over time. With preschedule=FALSE,
each iteration will spawn a new process, and that seems to be when
changes in resource utilization tend to occur. I am thinking that there
may be some trouble with my kernel, since %dopar% is spawning 4
additional R processes as it should--my kernel just isn't handling it
properly. In fact, I did:
> registerDoMC(cores=3)
and got only 2 cores maxed, never 3. For some reason 2 (or more) of the
processes are always on the same core. Although when I registered 3
cores and only had 2 iterations left, both remaining processes used
100%. And of course the last remaining process used 100% until it
finished as well. Just weird. To be thorough, I tried:
> registerDoMC(cores=2)
and never saw 2 cores used.

Last, and interestingly, when I did:
> registerDoMC(cores=5)
(on my 4-core machine) I sometimes maxed all 4 cores. I also saw only 1,
2, or 3 cores used during the computation. Running 5 processes on 1 core
certainly doesn't help, though! Frustrating....

Anyone else using Ubuntu 10.04? Maybe it's my kernel screwing this up.

$ uname -srvmo
Linux 2.6.32-24-generic #38 SMP Mon Jul 26 15:08:51 EDT 2010 x86_64
GNU/Linux

-Brian