[R-sig-hpc] communicating memory requirments

[Martin Morgan]

On 02/19/2013 05:16 AM, Max Kuhn wrote:
> I have some scripts for a book that I will be publishing and they use
> parallel processing (via foreach). Some of the analyses use more memory
> than some users will have on hand and, as the number of workers increases,
> so do the memory demands.
>
> I'd like to report the memory requirements in a way that most people will
> understand (including me).

not really answering your question, but it seems like parallel evaluation in 
shared memory computers comes with an implicit need to manage memory, and that 
one would rather strive to implement algorithms that do their job in a memory 
efficient way. Probably this means iterating through data and aggregating 
results, which is the approach of biglm. The user shouldn't really be exposed to 
the need to choose a computer (or package) based on memory consumption of 
algorithms. I'm not throwing stones, having seldom managed this myself.

Martin

>
> On OS X, I've run the scripts and did a roiling append of 'top' to capture
> the memory used by the master process and the workers over time.
>
> Can anyone suggest which parameters I should report (e.g. VSIZE)? Is the
> situation appreciably different on Windows?
>
> I admit to being fairly ignorant on this (complicated) subject so any
> approach to informing the users would be very welcome.
>
> Thanks,
>
> Max
>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-hpc mailing list
> R-sig-hpc at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-hpc
>


-- 
Computational Biology / Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N.
PO Box 19024 Seattle, WA 98109

Location: Arnold Building M1 B861
Phone: (206) 667-2793