[R-sig-hpc] communicating memory requirments

[Claudia Beleites]

Hi Max,

> I have some scripts for a book that I will be publishing and they use
> parallel processing (via foreach). Some of the analyses use more
> memory than some users will have on hand and, as the number of
> workers increases, so do the memory demands.
> 
> I'd like to report the memory requirements in a way that most people
> will understand (including me).
> 
> On OS X, I've run the scripts and did a roiling append of 'top' to
> capture the memory used by the master process and the workers over
> time.
> 
> Can anyone suggest which parameters I should report (e.g. VSIZE)? Is
> the situation appreciably different on Windows?

I guess I'd report (graph) mem used + swap used in human-readable
units, and maybe indicate total RAM and total RAM + SWAP in the graph.

> I admit to being fairly ignorant on this (complicated) subject so any
> approach to informing the users would be very welcome.

The remainder is no direct answer to your question, but here's what I
usually do:

I use a custom function to list object sizes: 
https://r-forge.r-project.org/scm/viewvc.php/pkg/cbmisc/R/lscb.R?view=markup&root=hyperspec

According to ? object.size, this is not 100% correct, but it gives a
good overview what is going on, and it yields numbers that people
understand. 

I tell my colleagues to multiply this with the number of explicitly
parallel threads, and that the rule of thumb is that they should expect
to need something between 3 times that much and 3 times the size of the
largest single object available. 

This has the advantage that people can estimate whether their number of
parallel threads has a chance, before they run into heavy swapping
(which of course means huge response times of the server).

HTH,

Claudia

> 
> Thanks,
> 
> Max
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc



-- 
Claudia Beleites
Spectroscopy/Imaging
Institute of Photonic Technology 
Albert-Einstein-Str. 9
07745 Jena
Germany

email: claudia.beleites at ipht-jena.de
phone: +49 3641 206-133
fax:   +49 2641 206-399