[R] Memory issues..

[Prof Brian Ripley]

On Wed, 12 Nov 2003, JFRI (Jesper Frickman) wrote:

> How much processing takes place before you get to the lme call? Maybe R
> has just used up the memory on something else. I think there is a fair
> amount of memory leak, as I get similar problems with my program. I use

Windows, right?  I don't think this is memory leak, but rather
fragmentation.  Hopefully the memory management in R-devel will ease this, 
and you might like to compile that up and try it.

On R 1.8.0 on Windows you have to be able to find a block of contiguous 
memory of the needed size, so fragmentation can kill you.  Try increasing 
--max-memory-size unless you are near 2Gb.

> R 1.8.0. My program goes as follows.
> 
> 1. Use RODBC to get a data.frame containing assays to analyze (17 assays
> are found).
> 2. Define an AnalyzeAssay(assay, suffix) function to do the following:
> 	a) Use RODBC to get data.
> 	b) Store dataset "limsdata" in workspace using the <<- operator
> to avoid the following error in qqnorm.lme: Error in eval(expr, envir,
> enclos) : Object "limsdata" not found, when I call it with a grouping
> formula like: ~ resid(.) | ORDCURV.
> 	c) Call lme to analyze data.
> 	d) Produce some diagnostic plots. Record them by setting
> record=TRUE on the trellis.device
> 	e) Save the plots on win.metafile using replayPlot(...)
> 	f) Save text to a file using sink(...)
> 
> 3. Call the function for each assay using the code:
> 
> # Analyze each assay
> for(i in 1:length(assays[,1]))
> {
> 	writeLines(paste("Analyzing ", assays$DILUTION[i], " ",
> assays$PROFNO[i], "...", sep=""))
> 	flush.console()
> 	AnalyzeAssay(assays$DILUTION[i], assays$PROFNO[i])
> 
> 	# Clean up memory
> 	rm(limsdata)
> 	gc()
> }
> 
> As you can see, I try to remove the dataset stored in workspace and then
> call gc() to clean up my memory as I go.
> 
> Nevertheless, when I come to assay 11 out of 17, it stops with a memory
> allocation error. I have to quit R, and start again with assay 11, then
> it stops again with assay 15 and finally 17. The last assays have much
> more data than the first ones, but all assays can be completed as long
> as I keep restarting...
> 
> Maybe restarting the job can help you getting it done?

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595