[R-SIG-Finance] R Memory Usage

[Brian G. Peterson]

On 04/10/2011 10:49 AM, Elliot Joel Bernstein wrote:
> This is not specifically a finance question, but I'm working with financial
> data (daily stock returns), and I suspect many people using R for financial
> analysis face similar issues. The basic problem I'm having is that with a
> moderately large data set (800 stocks x 11 years), performing a few
> operations such as data transformations, fitting regressions, etc., results
> in R using an enormous amount of memory -- sometimes upwards of 5GB -- even
> after using gc() to try and free some memory up. I've read several posts to
> various R mailing lists over the years indicating that R does not release
> memory back to the system on certain OSs (64 bit Linux in my case), so I
> understand that this is "normal" behavior for R. How do people typically
> work around this to do exploratory analysis on large data sets without
> having to constantly restart R to free up memory?

Your data set isn't very large.  We use R on months of tick data at a time.

You are correct that R keeps the memory allocated for its use, but it 
*will* release it to the OS in large chunks if the OS asks for more 
memory back.  So, once you've used a lot of RAM, R will keep it 
available, expecting you to use it again, but release it to the OS if 
the kernel asks.  This shouldn't negatively affect performance on 64 bit 
linux (it's the environment we use too).

You *do* want to be careful how many copies you make and keep around. 
Things like overwriting the original var with your transformed data, 
using environments, etc can vastly improve your memory profile.

Regards,

   - Brian

-- 
Brian G. Peterson
http://braverock.com/brian/
Ph: 773-459-4973
IM: bgpbraverock