[R] Determining the maximum memory usage of a function

[Jonathan Greenberg]

Jim:

Thanks, but I'm looking for something that can be used somewhat
automatically -- the function in question would be user-provided and
passed to my "chunking" algorithm, so in this case it would be the
end-user (not me) who would have to embed these -- would
Rprof(memory.profiling=TRUE)
# my function
Rprof(NULL)
... and then taking the max of the tseries output be a reasonable
approach?  If so, which of the three outputs (vsize.small vsize.large
  nodes) would be best compared against the available memory?

Cheers!

--j

On Thu, Jun 20, 2013 at 10:07 AM, jim holtman <jholtman at gmail.com> wrote:
> What I would do is to use "memory.size()" to get the amount of memory being
> used.  Do a call at the beginning of the function to determine the base, and
> then at other points in the code to see what the difference from the base is
> and keep track of the maximum difference.  I am not sure if just getting the
> memory usage at the end would be sufficient since there may be some garbage
> collection in between, or you might be creating some large objects and then
> deleting/reusing them.  So keep track after large chunks of code to see what
> is happening.
>
>
> On Thu, Jun 20, 2013 at 10:45 AM, Jonathan Greenberg <jgrn at illinois.edu>
> wrote:
>>
>> Folks:
>>
>> I apologize for the cross-posting between r-help and r-sig-hpc, but I
>> figured this question was relevant to both lists.  I'm writing a
>> function to be applied to an input dataset that will be broken up into
>> chunks for memory management reasons and for parallel execution.  I am
>> trying to determine, for a given function, what the *maximum* memory
>> usage during its execution is (which may not be the beginning or the
>> end of the function, but somewhere in the middle), so I can "plan" for
>> the chunk size (e.g. have a table of chunk size vs. max memory usage).
>>
>> Is there a trick for determining this?
>>
>> --j
>>
>> --
>> Jonathan A. Greenberg, PhD
>> Assistant Professor
>> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
>> Department of Geography and Geographic Information Science
>> University of Illinois at Urbana-Champaign
>> 607 South Mathews Avenue, MC 150
>> Urbana, IL 61801
>> Phone: 217-300-1924
>> http://www.geog.illinois.edu/~jgrn/
>> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>
>
>
>
> --
> Jim Holtman
> Data Munger Guru
>
> What is the problem that you are trying to solve?
> Tell me what you want to do, not how you want to do it.



-- 
Jonathan A. Greenberg, PhD
Assistant Professor
Global Environmental Analysis and Remote Sensing (GEARS) Laboratory
Department of Geography and Geographic Information Science
University of Illinois at Urbana-Champaign
607 South Mathews Avenue, MC 150
Urbana, IL 61801
Phone: 217-300-1924
http://www.geog.illinois.edu/~jgrn/
AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007