[R-sig-hpc] foreach + doMC not fully parallel?

[Norm Matloff]

If your machine uses DMA for disk access, then your disk activity should
not be reflected in CPU utilization.  Some systems allow you to
enable/disable DMA, and you may want to look into enabling it.

But even if the CPU is being used for disk actions, I don't see why it
should always be one particular core that is handling that action.  It
should be pretty much random across the cores.  Maybe I'm missing
something?

Norm Matloff

On Thu, Aug 26, 2010 at 06:35:01PM -0400, Sean Davis wrote:
> On Thu, Aug 26, 2010 at 6:14 PM, Brian D Peyser PhD <bpeyser at jhmi.edu>wrote:
> 
> > Hi everyone,
> >
> > I am trying to run code in parallel across 4 cores with foreach and
> > doMC. The code runs fine, and it will spawn 4 R processes, but these
> > processes aren't all on separate cores. Often, I will see one core maxed
> > with each R process taking about 24% of the CPU. I also see something
> > like 96%, 98%, 48%, 48% (3 of 4 cores being used) or 32%, 32%, 96%, 32%.
> > (2 cores). This may change during the run, and I almost never see 4
> > cores getting maxed.
> >
> > I am running like this (pseudo-code):
> >
> > > library("doMC")
> > > registerDoMC(cores=4) # I have a 4-core processor
> > >       # Read in data....
> > > foreach(obj=gsub(".txt", "",
> > files), .inorder=FALSE, .errorhandling="pass", .options.multicore =
> > list(preschedule = FALSE)) %dopar% {
> > >       # Do lots of computation with get(obj)
> > >       # Output a PNG plot and some text
> > > }
> >
> > I have also tried with preschedule=TRUE, which has the same problem,
> > though then a process or two will end early since it has been going on
> > its own core instead of sharing. This is on Linux (Ubuntu 10.04 64-bit
> > desktop edition), and I am running R from Emacs with ESS (maybe I should
> > try from the terminal instead--hadn't thought to do that). Any thoughts
> > that could help?
> >
> >
> Assuming there is nothing else running on the system?
> 
> In any case, it appears that you are reading text files and writing to disk.
>  Only one core can use the IO at a time, so that may explain this.  Try
> running something entirely computational and see if you can get close to
> 100% on all four cores.
> 
> Sean
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc