[R] heap size trouble
Prof Brian D Ripley
ripley at stats.ox.ac.uk
Fri Jun 2 08:26:23 CEST 2000
On Fri, 2 Jun 2000, Strumila, John wrote:
> Doesn't using an external DB just delay the virtual memory issue?
> Eventually when the data is manipulated in R, the memory requirement is
> still the same - or am I missing something?
The idea (which was not clear from Doug's message I grant) is to
get the database to do all the selection for you. That can include
passing blocks of data to R at a time, especially passing just a few
columns at time. That way R only needs space for a minimal piece
of the dataset.
On the original issue, this is an FAQ, with an example in the FAQ, and it
might be a good convention just to point people to the FAQ.
> -----Original Message-----
> From: Douglas Bates [mailto:bates at stat.wisc.edu]
> Sent: Wednesday, 31 May 2000 23:36
> To: Jim Lemon
> Cc: r-help at stat.math.ethz.ch
> Subject: Re: [R] heap size trouble
> Jim Lemon <bitwrit at ozemail.com.au> writes:
> > karamian wrote:
> > ...I want to load a file that contains 93 thousand raws and 22 colums of
> > data (essentially float)...
> > I just had to process over 199000 records with four numeric values. If
> > I remember correctly, I used:
> > --vsize 30M --nsize 500000
> > which pretty much ate all the RAM (64M) I had. Don't forget to "rm" big
> > data sets before you exit, or R will bomb when you next try to load
> > without the increased memory. Just reread from the data file when you
> > need them again (and it helps to exit other apps before starting R to
> > avoid disk thrashing).
> Another approach is to use a relational database to store such a large
> table and load the table into R from the database. There are several
> interfaces into R from relational databases.
Brian D. Ripley, ripley at stats.ox.ac.uk
Professor of Applied Statistics, http://www.stats.ox.ac.uk/~ripley/
University of Oxford, Tel: +44 1865 272861 (self)
1 South Parks Road, +44 1865 272860 (secr)
Oxford OX1 3TG, UK Fax: +44 1865 272595
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
More information about the R-help