[R] heap size trouble

[Prof Brian D Ripley]

On Fri, 2 Jun 2000, Strumila, John wrote:

> Doesn't using an external DB just delay the virtual memory issue?
> 
> Eventually when the data is manipulated in R, the memory requirement is
> still the same - or am I missing something?

The idea (which was not clear from Doug's message I grant) is to
get the database to do all the selection for you.  That can include
passing blocks of data to R at a time, especially passing just a few
columns at time.  That way R only needs space for a minimal piece
of the dataset.

On the original issue, this is an FAQ, with an example in the FAQ, and it
might be a good convention just to point people to the FAQ.


> -----Original Message-----
> From: Douglas Bates [mailto:bates at stat.wisc.edu]
> Sent: Wednesday, 31 May 2000 23:36
> To: Jim Lemon
> Cc: r-help at stat.math.ethz.ch
> Subject: Re: [R] heap size trouble
> 
> 
> Jim Lemon <bitwrit at ozemail.com.au> writes:
> 
> > karamian wrote:
> > 
> > ...I want to load a file that contains 93 thousand raws and 22 colums of
> > data (essentially float)...
> > 
> > I just had to process over 199000 records with four numeric values.  If
> > I remember correctly, I used:
> > 
> > --vsize 30M  --nsize 500000
> > 
> > which pretty much ate all the RAM (64M) I had.  Don't forget to "rm" big
> > data sets before you exit, or R will bomb when you next try to load
> > without the increased memory.  Just reread from the data file when you
> > need them again (and it helps to exit other apps before starting R to
> > avoid disk thrashing).
> 
> Another approach is to use a relational database to store such a large
> table and load the table into R from the database.  There are several
> interfaces into R from relational databases.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._