[R] memory storage explosion

[Peter Dalgaard BSA]

Saket Joshi <joshi at engr.orst.edu> writes:

> Hi,
> 
> I tried using 'for' loops and also tried explicit calls to gc() (infact
> that is how I found out when and how much memory is being allocated and that it
> is not getting freed).
> 
> Could this be a bug in R? If not so, the there must be a way to free this
> memory used. Could you give me some idea as to how I can do that?

I suspect R is just doing what you ask of it. You seem to be
collecting a large number of rpart objects (returned from abc()) and
saving them up in the return value from lapply(). Each rpart object
has a reference to the data it was calculated on (and these references
cannot point to the same storage since you modify the data frame
inside abc()), so you end up storing 126 objects the size of your
original 25MB dataset.


> On Sun, 1 Sep 2002, Roger Peng wrote:
> 
> > You could make an explicit call to 'gc' to force a garbage collection
> > (although I don't think this would really make a difference, just
> > guessing).  Also, you might try using a 'for' loop instead of 'lapply' in
> > your 'fgh' function.
> >
> > -roger
> > _______________________________
> > UCLA Department of Statistics
> > rpeng at stat.ucla.edu
> > http://www.stat.ucla.edu/~rpeng
> >
> > On Sun, 1 Sep 2002, Saket Joshi wrote:
> >
> > > Hi all,
> > >
> > > I am using R1.5.0 under unix.
> > >
> > > I have 2 data.frames x and yx. both have the same number of rows. I wrote
> > > a function abc to do the following:
> > >
> > > abc <- function(x, zyx)
> > > {
> > > x[1] <- zyx   # zyx is a column of yx
> > > rpart(x)
> > > }
> > >
> > > Then I wrote another function fgh:
> > >
> > > fgh <- function()
> > > {
> > > yx <- ............... # yx created
> > > x <- ................  # x created
> > > lapply(yx, abc, x)
> > > }
> > >
> > > when I ran function fgh at the prompt, the program started running out of
> > > memory and the process running R gets killed. The reason for
> > > the memory running out is that with every call to the abc function in the
> > > implicit loop lapply, the memory equivalent to that of x is allocated in the
> > > variable storage (Vcells). The trouble is that the memory allocated is not
> > > freed when the function abc is run by lapply for the next element of yx
> > > i.e. in the next loop.
> > >
> > > Since x takes up 25 MB of space and since yx has 126 columns, a lot of
> > > memory space keeps getting allocated without being freed even when it is no
> > > longer required.
> > >
> > > I wonder if there are any options that I should set so that the memory
> > > allocated within a function is freed when the function is exited. But I am
> > > not sure how to do that or else how to free the memory by any other way.
> > >
> > > Could someone tell me how I can avoid this memory explosion?
> > >
> > > Thanks in advance,
> > > -Saket.
> > >
> > >
> > >
> > > -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
> > > r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
> > > Send "info", "help", or "[un]subscribe"
> > > (in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
> > > _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
> > >
> >
> 
> -.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
> r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
> Send "info", "help", or "[un]subscribe"
> (in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
> _._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._
> 

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._