[R] practical memory limits

[Greg Snow]

An old rule of thumb was that you should have 6 times as much memory as
your dataset will take.  But I think pretty much everything has been
improved since then, so you should be able to get by with less (others
may be able to give a better rule of thumb these days).

You might want to look at the biglm package, it allows you to do
regression models with only a portion of your data loaded at a time,
allowing for pretty much any size of data set in a limited memory
situation.

Hope this helps,

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at intermountainmail.org
(801) 408-8111
 
 

> -----Original Message-----
> From: r-help-bounces at stat.math.ethz.ch 
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of ivo welch
> Sent: Saturday, February 10, 2007 11:03 AM
> To: r-help at stat.math.ethz.ch
> Subject: [R] practical memory limits
> 
> Dear R experts:  I want to learn what the practically useful 
> memory limits are for good work with R.
> 
> (My specific problem is that I want work with daily stock returns.
> In ASCII, the data set is about 72 million returns, that 
> would have to go into a sparse matrix (not all stocks exist 
> for the whole series).
> As a guess, this will consume about 700MB.  My main use will 
> be linear operations---regressions, means, etc.)
> 
> I am on linux, so I can create swap space, but I am concerned 
> that the thrashing will be so bad that the computer will 
> become worthless.  In fact, the last time I used it was over 
> 3 years ago.  Since then, I have just turned it off.
> 
> I have 2GB of RAM right now, and could upgrade this to 4GB.
> 
> Are there some general guidelines as to what the relationship 
> between data sets and memory should be under R?  I know this 
> will vary with the task involved, but some guidance would be 
> better than none.
> 
> regards,
> 
> /iaw
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>