[R] MC-Simulation with foreach: Some cores finish early

[Ben Bolker]

peter_petersen <henning.bumann <at> gmail.com> writes:

> I run a MC-Simulation using the the packages "foreach" and "doMC" on a
> PowerMac with 24 cores. There are roughly a hundred parametersets and I
> parallelized the program in a way, that each core computes one of these
> parametersets completely.
> 
> The problem ist, that some parametersets take a lot longer to compute than
> others. After a while there are only a quarter of the cores still computing
> (their first parameterset), while others are already finished. But some
> parametersets are still untouched.
> 
> I have thought about changing my parameterfile in a way, that every
> combination takes roughly the same time (longer computations are offset with
> less repetitions), but maybe there is a more elegant solution.

  It sounds to me like this would require writing an entire
batch scheduling system within R -- i.e., the system would have
to maintain a queue and track which cores were finished. I'd
love to know if someone's written it, but I sort of doubt it ...