[R] How to free memory used by R.

[Lariani,Sofiane,LAUSANNE,NRC/BAS]

>>>>> I follow the memory used by R at each run. This memory increase
considerably at each run. >>>>> After three times  the message is no more
memory (or some thing like this). Even by >>>>> >>>>> deleting all objects
and using gc(), the memory dont decrease!!! the only solution is to >>>>>
kill R.

>>>>> Windows and R 1.7.1

>>>>  thank you
>>>>> ps: I dont think that R developper are aware about the importance of
this memory problem. 


-----Original Message-----
From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk]
Sent: lundi, 1. septembre 2003 19:50
To: Thomas W Blackwell
Cc: Lariani,Sofiane,LAUSANNE,NRC/BAS; 'r-help at stat.math.ethz.ch'
Subject: Re: [R] How to free memory used by R. 


On Mon, 1 Sep 2003, Thomas W Blackwell wrote:

> On Mon, 1 Sep 2003, Lariani,Sofiane,LAUSANNE,NRC/BAS wrote:
> 
> > I want to free memory used by R. The usage of rm and gc give no result.
I'm
> > running  an algorithm consuming a huge memory and I need to recover the
> > memory used by R between 2 call of my algorithm.

What was the exact error message behind this?

>>>>> I follow the memory used by R at each run. This memory increase
considerably at each run. >>>>> After three times  the message is no more
memory (or some thing like this). Even by >>>>> >>>>> deleting all objects
and using gc(), the memory dont decrease!!! the only solution is to >>>>>
kill R.

> > Thank you in advance for your help.
> >
> > e-mail: sofiane.lariani at rdls.nestle.com
> >
> > Sofiane Lariani
> 
> What operating system and what version of R, please ?

Yes, please?

>>>>> Windows and R 1.7.1

> Other people will have a much better understanding of the
> details here than I have, but here goes.
> 
> The fact that you ask this question suggests that you are
> running the unnamed, memory intensive algorithm inside a loop.
> It's my understanding that  gc()  is called automatically
> every time control comes back to the command line prompt.
> So, gc() is only an issue inside a loop.  You might TRY
> writing a wrapper function:
> 
> my.algorithm <- function(...) {
>    algorithm(...)
>    gc()           }
> 
> and call the wrapper rather than  algorithm()  directly,
> inside the loop.  Chances are this won't help, because
> the loop will try to keep intermediate results.
> (At least it used to do this in old Splus prior to 1990.)

Not in R (and S-PLUS only existed went on sale in about 1988).
Unless there is a bug or a memory leak, gc() is called whenever a naive 
attempt to get memory fails.  There are various levels of gc(), and they 
are all tried.  So when R reports it is short of memory, it is unable to 
free up enough of the memory it has for the needed memory, and all objects 
not currently in use have been deleted.

If you want a lot of memory for a single object then it has to be 
contiguous.  R no longer compacts objects, so it can become increasingly 
hard to get large chunks of memory as the loop runs.

> If nothing else works, do as we all used to do prior to 1990.
> Write a batch script which does one iteration of the algorithm
> and saves just the intermediate results you want.  Then write
> a shell script (I assume you're using a version of unix) that
> simply runs  R BATCH scriptname logfile  again and again, one
> call for each iteration of the loop.  This guarantees that
> nothing is saved except what you specify.  (It's clunky, yes,
> but this works when nothing else will.)  

That's true of S, but not I think of R (or at least the memory reduction 
is very small).  We say For loops are not implemented in R because they 
are not needed, and For loops automate this.


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595