[R-sig-hpc] foreach + doMC not fully parallel?
Brian D Peyser PhD
bpeyser at jhmi.edu
Fri Aug 27 17:37:46 CEST 2010
On Thu, 2010-08-26 at 18:35 -0400, Sean Davis wrote:
> Assuming there is nothing else running on the system?
> In any case, it appears that you are reading text files and writing to
> disk. Only one core can use the IO at a time, so that may explain
> this. Try running something entirely computational and see if you can
> get close to 100% on all four cores.
I hadn't thought about IO, but the fact that the processes always add up
to almost 100% of 1/2/3 cores makes me think that something is
preventing them from computing on all the cores rather than just waiting
for I/O. The I/O is actually very minor, but I took out all of it and
still had the same result.
So next I decided to stop my other resource hogs, such as Firefox and
Evolution. that had no impact. I also ran from the terminal instead of
from Emacs, with the same result.
What's strange is the usage will vary over time. With preschedule=FALSE,
each iteration will spawn a new process, and that seems to be when
changes in resource utilization tend to occur. I am thinking that there
may be some trouble with my kernel, since %dopar% is spawning 4
additional R processes as it should--my kernel just isn't handling it
properly. In fact, I did:
and got only 2 cores maxed, never 3. For some reason 2 (or more) of the
processes are always on the same core. Although when I registered 3
cores and only had 2 iterations left, both remaining processes used
100%. And of course the last remaining process used 100% until it
finished as well. Just weird. To be thorough, I tried:
and never saw 2 cores used.
Last, and interestingly, when I did:
(on my 4-core machine) I sometimes maxed all 4 cores. I also saw only 1,
2, or 3 cores used during the computation. Running 5 processes on 1 core
certainly doesn't help, though! Frustrating....
Anyone else using Ubuntu 10.04? Maybe it's my kernel screwing this up.
$ uname -srvmo
Linux 2.6.32-24-generic #38 SMP Mon Jul 26 15:08:51 EDT 2010 x86_64
More information about the R-sig-hpc