[R] Cleaning up the memory
Prof Brian Ripley
ripley at stats.ox.ac.uk
Fri Aug 10 20:42:30 CEST 2007
On Fri, 10 Aug 2007, Monica Pisica wrote:
> Thanks! I will look into ...
> I have 4 GB RAM, and i was monitoring the memory with Windows task
> manager so i was looking how R "gets" more and more memory allocation
> from less than 100Mb to .... 1500Mb .....
Then you are almost certainly fragmenting the address space.
We still don't know your OS and whether you have enabled the /3GB switch
(if relevant to that version of Windows). Most versions of Windows have
a 2Gb address space, but some can be as high as 4Gb (Vista 64 which I use
is one: the details are in the rw-FAQ for the latest versions of R, e.g.
R-patched and R-devel). That factor of 2 can make a big difference.
> My initial tables are between 30 to 80 Mb and the resulting tables that
> incorporate the initial tables plus PCA and kmeans results are inbetween
> 50 to 200MB or thereabouts!
> And yes, i don't really care about memory allocation in detail - what i
> want is to free that memory after every cycle ;-)
> Although, after i didn't do anything in R and it was idle for more than
> 30 min. the memory allocation according to Task manager dropped to 15 Mb
> ..... which is good - but i cannot wait inbetween cycles half an hour
> though .....
Calling gc() will reduce the memory allocation, but that is not the point.
You can have 15Mb allocated and still not a 50Mb hole in the address
space (although that would be extremely unlucky, not having several 200Mb
holes is quite likely).
> Again thanks,
> Monica> Date: Fri, 10 Aug 2007 18:28:07 +0100> From:
> ripley at stats.ox.ac.uk> To: pisicandru at hotmail.com> CC:
> r-help at stat.math.ethz.ch> Subject: Re: [R] Cleaning up the memory> > On
> Fri, 10 Aug 2007, Monica Pisica wrote:> > >> > Hi,> >> > I have 4 huge
> tables on which i want to do a PCA analysis and a kmean > > clustering.
> If i run each table individually i have no problems, but if > > i want
> to run it in a for loop i exceed the memory alocation after the > >
> second table, even if i save the results as a csv table and i clean up >
> > all the big objects with rm command. To me it seems that even if i
> don't > > have the objects anymore, the memory these objects used to
> occupy is not > > cleared. Is there any way to clear up the memory as
> well? I don't want > > to close R and start it up again. Also i am
> running R under Windows.> > See ?gc, which does the clearing.> >
> However, unless you study the memory allocation in detail (which you >
> cannot do from R code), you don't actually know that this is the
> problem. > More likely is that you have fragmentation of your 32-bit
> address space: > see ?"Memory-limits".> > Without any idea what memory
> you have and what 'huge' means, we can only > make wild guesses. It
> might be worth raising the memory limit (the > --max-mem-size flag).> >
> >> > thanks,> >> > Monica> >
> _________________________________________________________________> >
> [[trailing spam removed]]> >> > [[alternative HTML version deleted]]> >>
> > ______________________________________________> >
> R-help at stat.math.ethz.ch mailing list> >
> https://stat.ethz.ch/mailman/listinfo/r-help> > PLEASE do read the
> posting guide http://www.R-project.org/posting-guide.html> > and provide
> commented, minimal, self-contained, reproducible code.> >> > -- > Brian
> D. Ripley, ripley at stats.ox.ac.uk> Professor of Applied Statistics,
> http://www.stats.ox.ac.uk/~ripley/> University of Oxford, Tel: +44 1865
> 272861 (self)> 1 South Parks Road, +44 1865 272866 (PA)> Oxford OX1 3TG,
> UK Fax: +44 1865 272595
> Messenger Café ? open for fun 24/7. Hot games, cool activities served
> daily. Visit now. http://cafemessenger.com?ocid=TXT_TAGLM_AugWLtagline
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272866 (PA)
Oxford OX1 3TG, UK Fax: +44 1865 272595
More information about the R-help