[R] Cleaning up the memory

[Prof Brian Ripley]

On Fri, 10 Aug 2007, Monica Pisica wrote:

>
> Thanks! I will look into ...
>
> I have 4 GB RAM, and i was monitoring the memory with Windows task 
> manager so i was looking how R "gets" more and more memory allocation 
> from less than 100Mb to .... 1500Mb .....

Then you are almost certainly fragmenting the address space.

We still don't know your OS and whether you have enabled the /3GB switch 
(if relevant to that version of Windows).   Most versions of Windows have 
a 2Gb address space, but some can be as high as 4Gb (Vista 64 which I use 
is one: the details are in the rw-FAQ for the latest versions of R, e.g. 
R-patched and R-devel).  That factor of 2 can make a big difference.

> My initial tables are between 30 to 80 Mb and the resulting tables that 
> incorporate the initial tables plus PCA and kmeans results are inbetween 
> 50 to 200MB or thereabouts!
>
> And yes, i don't really care about memory allocation in detail - what i 
> want is to free that memory after every cycle ;-)
>
> Although, after i didn't do anything in R and it was idle for more than 
> 30 min. the memory allocation according to Task manager dropped to 15 Mb 
> ..... which is good - but i cannot wait inbetween cycles half an hour 
> though .....

Calling gc() will reduce the memory allocation, but that is not the point.
You can have 15Mb allocated and still not a 50Mb hole in the address 
space (although that would be extremely unlucky, not having several 200Mb 
holes is quite likely).

>
> Again thanks,
>
> Monica> Date: Fri, 10 Aug 2007 18:28:07 +0100> From: 
> ripley at stats.ox.ac.uk> To: pisicandru at hotmail.com> CC: 
> r-help at stat.math.ethz.ch> Subject: Re: [R] Cleaning up the memory> > On 
> Fri, 10 Aug 2007, Monica Pisica wrote:> > >> > Hi,> >> > I have 4 huge 
> tables on which i want to do a PCA analysis and a kmean > > clustering. 
> If i run each table individually i have no problems, but if > > i want 
> to run it in a for loop i exceed the memory alocation after the > > 
> second table, even if i save the results as a csv table and i clean up > 
> > all the big objects with rm command. To me it seems that even if i 
> don't > > have the objects anymore, the memory these objects used to 
> occupy is not > > cleared. Is there any way to clear up the memory as 
> well? I don't want > > to close R and start it up again. Also i am 
> running R under Windows.> > See ?gc, which does the clearing.> > 
> However, unless you study the memory allocation in detail (which you > 
> cannot do from R code), you don't actually know that this is the 
> problem. > More likely is that you have fragmentation of your 32-bit 
> address space: > see ?"Memory-limits".> > Without any idea what memory 
> you have and what 'huge' means, we can only > make wild guesses. It 
> might be worth raising the memory limit (the > --max-mem-size flag).> > 
> >> > thanks,> >> > Monica> > 
> _________________________________________________________________> > 
> [[trailing spam removed]]> >> > [[alternative HTML version deleted]]> >> 
> > ______________________________________________> > 
> R-help at stat.math.ethz.ch mailing list> > 
> https://stat.ethz.ch/mailman/listinfo/r-help> > PLEASE do read the 
> posting guide http://www.R-project.org/posting-guide.html> > and provide 
> commented, minimal, self-contained, reproducible code.> >> > -- > Brian 
> D. Ripley, ripley at stats.ox.ac.uk> Professor of Applied Statistics, 
> http://www.stats.ox.ac.uk/~ripley/> University of Oxford, Tel: +44 1865 
> 272861 (self)> 1 South Parks Road, +44 1865 272866 (PA)> Oxford OX1 3TG, 
> UK Fax: +44 1865 272595 
> _________________________________________________________________ 
> Messenger Café ? open for fun 24/7. Hot games, cool activities served 
> daily. Visit now. http://cafemessenger.com?ocid=TXT_TAGLM_AugWLtagline

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595