[R-sig-hpc] foreach on HPC

[Brian G. Peterson]

On 10/15/2014 04:11 AM, Stefan Lüdtke wrote:
> Dear friends,
>
> we run an application on a HPC using the foreach package linked with
> doMPI. Each iteration is quite RAM hungry, so we have been touching the
> limit of 128GB. At the moment, the admin is figuring out whether an "out
> of memory" killed the job or not, so apparently it did not finish
> successfully.
>
> Now coming to my question: A couple of iterations run fine and I was
> checking the load of the host every now and then using "htop". I was
> observing that htop listed R processes that did not use any cpu at all
> but have been occupying up to 4% of RAM. So, what are these processes?
> Do I still see finished iterations that eat RAM from the system, and if
> so, how do I get rid of them??

If you're on a single machine, you will have much more efficient RAM 
usage if you use doMC or doParallel with a fork cluster than using doMPI.

The workers in an MPI cluster will not be torn down and released until 
everything is done in an MPI cluster.  The fork cluster is more efficient.

If you're on multiple machines, then doMPI or doRedis (or zmq or 
similar) are necessary, but on one machine fork will have far less overhead.

Regards,

Brian

-- 
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock