[R] Windows Memory Issues

[Prof Brian Ripley]

I think you misunderstand how R uses memory.  gc() does not free up all 
the memory used for the objects it frees, and repeated calls will free 
more.  Don't speculate about how memory management works: do your 
homework!

In any case, you are using an outdated version of R, and your first
course of action should be to compile up R-devel and try that, as there 
has been improvements to memory management under Windows.  You could also 
try compiling using the native malloc (and that *is* described in the 
INSTALL file) as that has different compromises.


On Sat, 6 Dec 2003, Richard Pugh wrote:

> Hi all,
>  
> I am currently building an application based on R 1.7.1 (+ compiled
> C/C++ code + MySql + VB).  I am building this application to work on 2
> different platforms (Windows XP Professional (500mb memory) and Windows
> NT 4.0 with service pack 6 (1gb memory)).  This is a very memory
> intensive application performing sophisticated operations on "large"
> matrices (typically 5000x1500 matrices).
>  
> I have run into some issues regarding the way R handles its memory,
> especially on NT.  In particular, R does not seem able to recollect some
> of the memory used following the creation and manipulation of large data
> objects.  For example, I have a function which receives a (large)
> numeric matrix, matches against more data (maybe imported from MySql)
> and returns a large list structure for further analysis.  A typical call
> may look like this .
>  
> > myInputData <- matrix(sample(1:100, 7500000, T), nrow=5000)
> > myPortfolio <- createPortfolio(myInputData)
>  
> It seems I can only repeat this code process 2/3 times before I have to
> restart R (to get the memory back).  I use the same object names
> (myInputData and myPortfolio) each time, so I am not create more large
> objects ..
>  
> I think the problems I have are illustrated with the following example
> from a small R session .
>  
> > # Memory usage for Rui process = 19,800
> > testData <- matrix(rnorm(10000000), 1000) # Create big matrix
> > # Memory usage for Rgui process = 254,550k
> > rm(testData)
> > # Memory usage for Rgui process = 254,550k
> > gc()
>          used (Mb) gc trigger  (Mb)
> Ncells 369277  9.9     667722  17.9
> Vcells  87650  0.7   24286664 185.3
> > # Memory usage for Rgui process = 20,200k
>  
> In the above code, R cannot recollect all memory used, so the memory
> usage increases from 19.8k to 20.2.  However, the following example is
> more typical of the environments I use .
>  
> > # Memory 128,100k
> > myTestData <- matrix(rnorm(10000000), 1000)
> > # Memory 357,272k
> > rm(myTestData)
> > # Memory 357,272k
> > gc()
>           used (Mb) gc trigger  (Mb)
> Ncells  478197 12.8     818163  21.9
> Vcells 9309525 71.1   31670210 241.7
> > # Memory 279,152k
>  
> Here, the memory usage increases from 128.1k to 279.1k
>  
> Could anyone point out what I could do to rectify this (if anything), or
> generally what strategy I could take to improve this?
>  
> Many thanks,
> Rich.
>  
> Mango Solutions
> Tel : (01628) 418134
> Mob : (07967) 808091
>  
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> 
> 

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595