[R] Windows Memory Issues
Prof Brian Ripley
ripley at stats.ox.ac.uk
Tue Dec 9 18:29:14 CET 2003
On Tue, 9 Dec 2003 Benjamin.STABLER at odot.state.or.us wrote:
> I would also like some clarification about R memory management. Like Doug,
> I didn't find anything about consecutive calls to gc() to free more memory.
It was a statement about Windows, and about freeing memory *to Windows*.
Douglas Grove apparently had misread both the subject line and the
sentence.
> We run into memory limit problems every now and then and a better
> understanding of R's memory management would go a long way. I am interested
> in learning more and was wondering if there is any specific R documentation
> that explains R's memory usage? Or maybe some good links about memory and
> garbage collection. Thanks.
There are lots of comments in the source files. And as I already said
(but has been excised below), this is not relevant to the next version of
R anyway.
BTW, the message below has been selectively edited, so please consult the
original.
> Message: 21
> Date: Mon, 8 Dec 2003 09:51:12 -0800 (PST)
> From: Douglas Grove <dgrove at fhcrc.org>
> Subject: Re: [R] Windows Memory Issues
> To: Prof Brian Ripley <ripley at stats.ox.ac.uk>
> Cc: r-help at stat.math.ethz.ch
> Message-ID:
> <Pine.LNX.4.44.0312080921260.27288-100000 at echidna.fhcrc.org>
> Content-Type: TEXT/PLAIN; charset=US-ASCII
>
> On Sat, 6 Dec 2003, Prof Brian Ripley wrote:
>
> > I think you misunderstand how R uses memory. gc() does not free up all
> > the memory used for the objects it frees, and repeated calls will free
> > more. Don't speculate about how memory management works: do your
> > homework!
>
> Are you saying that consecutive calls to gc() will free more memory than
> a single call, or am I misunderstanding? Reading ?gc and ?Memory I don't
> see anything about this mentioned. Where should I be looking to find
> more comprehensive info on R's memory management?? I'm not writing any
> packages, just would like to have a better handle on efficiently using
> memory as it is usually the limiting factor with R. FYI, I'm running
> R1.8.1 and RedHat9 on a P4 with 2GB of RAM in case there is any platform
> specific info that may be applicable.
>
> Thanks,
>
> Doug Grove
> Statistical Research Associate
> Fred Hutchinson Cancer Research Center
>
>
> > In any case, you are using an outdated version of R, and your first
> > course of action should be to compile up R-devel and try that, as there
> > has been improvements to memory management under Windows. You could also
> > try compiling using the native malloc (and that *is* described in the
> > INSTALL file) as that has different compromises.
> >
> >
> > On Sat, 6 Dec 2003, Richard Pugh wrote:
> >
> > > Hi all,
> > >
> > > I am currently building an application based on R 1.7.1 (+ compiled
> > > C/C++ code + MySql + VB). I am building this application to work on 2
> > > different platforms (Windows XP Professional (500mb memory) and Windows
> > > NT 4.0 with service pack 6 (1gb memory)). This is a very memory
> > > intensive application performing sophisticated operations on "large"
> > > matrices (typically 5000x1500 matrices).
> > >
> > > I have run into some issues regarding the way R handles its memory,
> > > especially on NT. In particular, R does not seem able to recollect some
> > > of the memory used following the creation and manipulation of large data
> > > objects. For example, I have a function which receives a (large)
> > > numeric matrix, matches against more data (maybe imported from MySql)
> > > and returns a large list structure for further analysis. A typical call
> > > may look like this .
> > >
> > > > myInputData <- matrix(sample(1:100, 7500000, T), nrow=5000)
> > > > myPortfolio <- createPortfolio(myInputData)
> > >
> > > It seems I can only repeat this code process 2/3 times before I have to
> > > restart R (to get the memory back). I use the same object names
> > > (myInputData and myPortfolio) each time, so I am not create more large
> > > objects ..
> > >
> > > I think the problems I have are illustrated with the following example
> > > from a small R session .
> > >
> > > > # Memory usage for Rui process = 19,800
> > > > testData <- matrix(rnorm(10000000), 1000) # Create big matrix
> > > > # Memory usage for Rgui process = 254,550k
> > > > rm(testData)
> > > > # Memory usage for Rgui process = 254,550k
> > > > gc()
> > > used (Mb) gc trigger (Mb)
> > > Ncells 369277 9.9 667722 17.9
> > > Vcells 87650 0.7 24286664 185.3
> > > > # Memory usage for Rgui process = 20,200k
> > >
> > > In the above code, R cannot recollect all memory used, so the memory
> > > usage increases from 19.8k to 20.2. However, the following example is
> > > more typical of the environments I use .
> > >
> > > > # Memory 128,100k
> > > > myTestData <- matrix(rnorm(10000000), 1000)
> > > > # Memory 357,272k
> > > > rm(myTestData)
> > > > # Memory 357,272k
> > > > gc()
> > > used (Mb) gc trigger (Mb)
> > > Ncells 478197 12.8 818163 21.9
> > > Vcells 9309525 71.1 31670210 241.7
> > > > # Memory 279,152k
> > >
> > > Here, the memory usage increases from 128.1k to 279.1k
> > >
> > > Could anyone point out what I could do to rectify this (if anything), or
> > > generally what strategy I could take to improve this?
> > >
> > > Many thanks,
> > > Rich.
> > >
> > > Mango Solutions
> > > Tel : (01628) 418134
> > > Mob : (07967) 808091
> > >
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > ______________________________________________
> > > R-help at stat.math.ethz.ch mailing list
> > > https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> > >
> > >
> >
> > --
> > Brian D. Ripley, ripley at stats.ox.ac.uk
> > Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
> > University of Oxford, Tel: +44 1865 272861 (self)
> > 1 South Parks Road, +44 1865 272866 (PA)
> > Oxford OX1 3TG, UK Fax: +44 1865 272595
> >
> > ______________________________________________
> > R-help at stat.math.ethz.ch mailing list
> > https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> >
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>
>
--
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
More information about the R-help
mailing list