[R-sig-hpc] foreach on HPC
Brian G. Peterson
brian at braverock.com
Wed Oct 15 11:46:54 CEST 2014
On 10/15/2014 04:11 AM, Stefan Lüdtke wrote:
> Dear friends,
>
> we run an application on a HPC using the foreach package linked with
> doMPI. Each iteration is quite RAM hungry, so we have been touching the
> limit of 128GB. At the moment, the admin is figuring out whether an "out
> of memory" killed the job or not, so apparently it did not finish
> successfully.
>
> Now coming to my question: A couple of iterations run fine and I was
> checking the load of the host every now and then using "htop". I was
> observing that htop listed R processes that did not use any cpu at all
> but have been occupying up to 4% of RAM. So, what are these processes?
> Do I still see finished iterations that eat RAM from the system, and if
> so, how do I get rid of them??
If you're on a single machine, you will have much more efficient RAM
usage if you use doMC or doParallel with a fork cluster than using doMPI.
The workers in an MPI cluster will not be torn down and released until
everything is done in an MPI cluster. The fork cluster is more efficient.
If you're on multiple machines, then doMPI or doRedis (or zmq or
similar) are necessary, but on one machine fork will have far less overhead.
Regards,
Brian
--
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock
More information about the R-sig-hpc
mailing list