[R] Windows Memory Issues

[Douglas Grove]

On Sat, 6 Dec 2003, Prof Brian Ripley wrote:

> I think you misunderstand how R uses memory.  gc() does not free up all 
> the memory used for the objects it frees, and repeated calls will free 
> more.  Don't speculate about how memory management works: do your 
> homework!

Are you saying that consecutive calls to gc() will free more memory than
a single call, or am I misunderstanding?   Reading ?gc and ?Memory I don't
see anything about this mentioned.  Where should I be looking to find 
more comprehensive info on R's memory management??  I'm not writing any
packages, just would like to have a better handle on efficiently using
memory as it is usually the limiting factor with R.  FYI, I'm running
R1.8.1 and RedHat9 on a P4 with 2GB of RAM in case there is any platform
specific info that may be applicable.

Thanks,

Doug Grove
Statistical Research Associate
Fred Hutchinson Cancer Research Center




 
> In any case, you are using an outdated version of R, and your first
> course of action should be to compile up R-devel and try that, as there 
> has been improvements to memory management under Windows.  You could also 
> try compiling using the native malloc (and that *is* described in the 
> INSTALL file) as that has different compromises.
> 
> 
> On Sat, 6 Dec 2003, Richard Pugh wrote:
> 
> > Hi all,
> >  
> > I am currently building an application based on R 1.7.1 (+ compiled
> > C/C++ code + MySql + VB).  I am building this application to work on 2
> > different platforms (Windows XP Professional (500mb memory) and Windows
> > NT 4.0 with service pack 6 (1gb memory)).  This is a very memory
> > intensive application performing sophisticated operations on "large"
> > matrices (typically 5000x1500 matrices).
> >  
> > I have run into some issues regarding the way R handles its memory,
> > especially on NT.  In particular, R does not seem able to recollect some
> > of the memory used following the creation and manipulation of large data
> > objects.  For example, I have a function which receives a (large)
> > numeric matrix, matches against more data (maybe imported from MySql)
> > and returns a large list structure for further analysis.  A typical call
> > may look like this .
> >  
> > > myInputData <- matrix(sample(1:100, 7500000, T), nrow=5000)
> > > myPortfolio <- createPortfolio(myInputData)
> >  
> > It seems I can only repeat this code process 2/3 times before I have to
> > restart R (to get the memory back).  I use the same object names
> > (myInputData and myPortfolio) each time, so I am not create more large
> > objects ..
> >  
> > I think the problems I have are illustrated with the following example
> > from a small R session .
> >  
> > > # Memory usage for Rui process = 19,800
> > > testData <- matrix(rnorm(10000000), 1000) # Create big matrix
> > > # Memory usage for Rgui process = 254,550k
> > > rm(testData)
> > > # Memory usage for Rgui process = 254,550k
> > > gc()
> >          used (Mb) gc trigger  (Mb)
> > Ncells 369277  9.9     667722  17.9
> > Vcells  87650  0.7   24286664 185.3
> > > # Memory usage for Rgui process = 20,200k
> >  
> > In the above code, R cannot recollect all memory used, so the memory
> > usage increases from 19.8k to 20.2.  However, the following example is
> > more typical of the environments I use .
> >  
> > > # Memory 128,100k
> > > myTestData <- matrix(rnorm(10000000), 1000)
> > > # Memory 357,272k
> > > rm(myTestData)
> > > # Memory 357,272k
> > > gc()
> >           used (Mb) gc trigger  (Mb)
> > Ncells  478197 12.8     818163  21.9
> > Vcells 9309525 71.1   31670210 241.7
> > > # Memory 279,152k
> >  
> > Here, the memory usage increases from 128.1k to 279.1k
> >  
> > Could anyone point out what I could do to rectify this (if anything), or
> > generally what strategy I could take to improve this?
> >  
> > Many thanks,
> > Rich.
> >  
> > Mango Solutions
> > Tel : (01628) 418134
> > Mob : (07967) 808091
> >  
> > 
> > 	[[alternative HTML version deleted]]
> > 
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> > 
> > 
> 
> -- 
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>