Thanks for the replies.

Being more specific, the analyses are mostly related to resampling of
various sorts (esp. the embarrassingly parallel type) that take a data
object into a worker, run a model and send summary statistics back to the
master

>From the basic memory profiling I've done (outside of R while the scripts
are running), I might see that the master process consumes 2500M (VSIZE)
and each worker eats up a separate chunk (say 1800M). I have a medium
memory size machine that I've used and figured out how many workers I can
afford before I exceed physical memory (usually by first exceeding it).

My experience is that people will just naively run the scripts and send an
email when it doesn't work within the limits on their machine. In the hopes
that some people might read the comments before running and I'd like to let
them know what they're in for. My current verbiage is:

> ### WARNING: Be aware of how much memory is needed to parallel
> ### process. It can very quickly overwhelm the available hardware. We
> ### estimate the memory usage (VSIZE = total memory size) to be
> ### 2566M/core.

That last number is determined over the workers that were created while the
script was running. I'm curious to know if the use of VSIZE is appropriate
(especially outside of *nix) and if this helps the user at all.

The scripts are at the chapter level, so a number of objects are created in
the same session. I dont think I need to profile at the object level but at
the session level so Rprofmem doesn't appear to be the best tool.

Thanks,

Max


On Tue, Feb 19, 2013 at 1:11 PM, Martin Morgan <mtmorgan@fhcrc.org> wrote:

> On 02/19/2013 05:16 AM, Max Kuhn wrote:
>
>> I have some scripts for a book that I will be publishing and they use
>> parallel processing (via foreach). Some of the analyses use more memory
>> than some users will have on hand and, as the number of workers increases,
>> so do the memory demands.
>>
>> I'd like to report the memory requirements in a way that most people will
>> understand (including me).
>>
>
> not really answering your question, but it seems like parallel evaluation
> in shared memory computers comes with an implicit need to manage memory,
> and that one would rather strive to implement algorithms that do their job
> in a memory efficient way. Probably this means iterating through data and
> aggregating results, which is the approach of biglm. The user shouldn't
> really be exposed to the need to choose a computer (or package) based on
> memory consumption of algorithms. I'm not throwing stones, having seldom
> managed this myself.
>
> Martin
>
>
>> On OS X, I've run the scripts and did a roiling append of 'top' to capture
>> the memory used by the master process and the workers over time.
>>
>> Can anyone suggest which parameters I should report (e.g. VSIZE)? Is the
>> situation appreciably different on Windows?
>>
>> I admit to being fairly ignorant on this (complicated) subject so any
>> approach to informing the users would be very welcome.
>>
>> Thanks,
>>
>> Max
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________**_________________
>> R-sig-hpc mailing list
>> R-sig-hpc@r-project.org
>> https://stat.ethz.ch/mailman/**listinfo/r-sig-hpc<https://stat.ethz.ch/mailman/listinfo/r-sig-hpc>
>>
>>
>
> --
> Computational Biology / Fred Hutchinson Cancer Research Center
> 1100 Fairview Ave. N.
> PO Box 19024 Seattle, WA 98109
>
> Location: Arnold Building M1 B861
> Phone: (206) 667-2793
>



-- 

Max

	[[alternative HTML version deleted]]

