[R] Memory issues..
Prof Brian Ripley
ripley at stats.ox.ac.uk
Wed Nov 12 16:54:58 CET 2003
On Wed, 12 Nov 2003, JFRI (Jesper Frickman) wrote:
> How much processing takes place before you get to the lme call? Maybe R
> has just used up the memory on something else. I think there is a fair
> amount of memory leak, as I get similar problems with my program. I use
Windows, right? I don't think this is memory leak, but rather
fragmentation. Hopefully the memory management in R-devel will ease this,
and you might like to compile that up and try it.
On R 1.8.0 on Windows you have to be able to find a block of contiguous
memory of the needed size, so fragmentation can kill you. Try increasing
--max-memory-size unless you are near 2Gb.
> R 1.8.0. My program goes as follows.
> 1. Use RODBC to get a data.frame containing assays to analyze (17 assays
> are found).
> 2. Define an AnalyzeAssay(assay, suffix) function to do the following:
> a) Use RODBC to get data.
> b) Store dataset "limsdata" in workspace using the <<- operator
> to avoid the following error in qqnorm.lme: Error in eval(expr, envir,
> enclos) : Object "limsdata" not found, when I call it with a grouping
> formula like: ~ resid(.) | ORDCURV.
> c) Call lme to analyze data.
> d) Produce some diagnostic plots. Record them by setting
> record=TRUE on the trellis.device
> e) Save the plots on win.metafile using replayPlot(...)
> f) Save text to a file using sink(...)
> 3. Call the function for each assay using the code:
> # Analyze each assay
> for(i in 1:length(assays[,1]))
> writeLines(paste("Analyzing ", assays$DILUTION[i], " ",
> assays$PROFNO[i], "...", sep=""))
> AnalyzeAssay(assays$DILUTION[i], assays$PROFNO[i])
> # Clean up memory
> As you can see, I try to remove the dataset stored in workspace and then
> call gc() to clean up my memory as I go.
> Nevertheless, when I come to assay 11 out of 17, it stops with a memory
> allocation error. I have to quit R, and start again with assay 11, then
> it stops again with assay 15 and finally 17. The last assays have much
> more data than the first ones, but all assays can be completed as long
> as I keep restarting...
> Maybe restarting the job can help you getting it done?
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
More information about the R-help