[R] Windows Memory Issues

Tue Dec 9 18:08:58 CET 2003

I would also like some clarification about R memory management.  Like Doug,
I didn't find anything about consecutive calls to gc() to free more memory.
We run into memory limit problems every now and then and a better
understanding of R's memory management would go a long way.  I am interested
in learning more and was wondering if there is any specific R documentation
that explains R's memory usage?  Or maybe some good links about memory and
garbage collection.  Thanks.

Benjamin Stabler
Transportation Planning Analysis Unit
Oregon Department of Transportation
555 13th Street NE, Suite 2
Salem, OR 97301  Ph: 503-986-4104

-------------------------------------------

Message: 21
Date: Mon, 8 Dec 2003 09:51:12 -0800 (PST)
From: Douglas Grove <dgrove at fhcrc.org>
Subject: Re: [R] Windows Memory Issues
To: Prof Brian Ripley <ripley at stats.ox.ac.uk>
Cc: r-help at stat.math.ethz.ch
Message-ID:
	<Pine.LNX.4.44.0312080921260.27288-100000 at echidna.fhcrc.org>
Content-Type: TEXT/PLAIN; charset=US-ASCII

On Sat, 6 Dec 2003, Prof Brian Ripley wrote:

> I think you misunderstand how R uses memory.  gc() does not free up all 
> the memory used for the objects it frees, and repeated calls will free 
> more.  Don't speculate about how memory management works: do your 
> homework!

Are you saying that consecutive calls to gc() will free more memory than
a single call, or am I misunderstanding?   Reading ?gc and ?Memory I don't
see anything about this mentioned.  Where should I be looking to find 
more comprehensive info on R's memory management??  I'm not writing any
packages, just would like to have a better handle on efficiently using
memory as it is usually the limiting factor with R.  FYI, I'm running
R1.8.1 and RedHat9 on a P4 with 2GB of RAM in case there is any platform
specific info that may be applicable.

Thanks,

Doug Grove
Statistical Research Associate
Fred Hutchinson Cancer Research Center

> In any case, you are using an outdated version of R, and your first
> course of action should be to compile up R-devel and try that, as there 
> has been improvements to memory management under Windows.  You could also 
> try compiling using the native malloc (and that *is* described in the 
> INSTALL file) as that has different compromises.
> 
> 
> On Sat, 6 Dec 2003, Richard Pugh wrote:
> 
> > Hi all,
> >  
> > I am currently building an application based on R 1.7.1 (+ compiled
> > C/C++ code + MySql + VB).  I am building this application to work on 2
> > different platforms (Windows XP Professional (500mb memory) and Windows
> > NT 4.0 with service pack 6 (1gb memory)).  This is a very memory
> > intensive application performing sophisticated operations on "large"
> > matrices (typically 5000x1500 matrices).
> >  
> > I have run into some issues regarding the way R handles its memory,
> > especially on NT.  In particular, R does not seem able to recollect some
> > of the memory used following the creation and manipulation of large data
> > objects.  For example, I have a function which receives a (large)
> > numeric matrix, matches against more data (maybe imported from MySql)
> > and returns a large list structure for further analysis.  A typical call
> > may look like this .
> >  
> > > myInputData <- matrix(sample(1:100, 7500000, T), nrow=5000)
> > > myPortfolio <- createPortfolio(myInputData)
> >  
> > It seems I can only repeat this code process 2/3 times before I have to
> > restart R (to get the memory back).  I use the same object names
> > (myInputData and myPortfolio) each time, so I am not create more large
> > objects ..
> >  
> > I think the problems I have are illustrated with the following example
> > from a small R session .
> >  
> > > # Memory usage for Rui process = 19,800
> > > testData <- matrix(rnorm(10000000), 1000) # Create big matrix
> > > # Memory usage for Rgui process = 254,550k
> > > rm(testData)
> > > # Memory usage for Rgui process = 254,550k
> > > gc()
> >          used (Mb) gc trigger  (Mb)
> > Ncells 369277  9.9     667722  17.9
> > Vcells  87650  0.7   24286664 185.3
> > > # Memory usage for Rgui process = 20,200k
> >  
> > In the above code, R cannot recollect all memory used, so the memory
> > usage increases from 19.8k to 20.2.  However, the following example is
> > more typical of the environments I use .
> >  
> > > # Memory 128,100k
> > > myTestData <- matrix(rnorm(10000000), 1000)
> > > # Memory 357,272k
> > > rm(myTestData)
> > > # Memory 357,272k
> > > gc()
> >           used (Mb) gc trigger  (Mb)
> > Ncells  478197 12.8     818163  21.9
> > Vcells 9309525 71.1   31670210 241.7
> > > # Memory 279,152k
> >  
> > Here, the memory usage increases from 128.1k to 279.1k
> >  
> > Could anyone point out what I could do to rectify this (if anything), or
> > generally what strategy I could take to improve this?
> >  
> > Many thanks,
> > Rich.
> >  
> > Mango Solutions
> > Tel : (01628) 418134
> > Mob : (07967) 808091
> >  
> > 
> > 	[[alternative HTML version deleted]]
> > 
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> > 
> > 
> 
> -- 
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>