[R] memory allocation error

[Peter Dalgaard BSA]

"M. Edward (Ed) Borasky" <znmeb at aracnet.com> writes:

> I was *just* going to post this same thing myself! My environment is a 400
> MHz P III running Windows 2000 Professional, and I get the error at 127 MB on
> my 128 MB machine :-). And I get the same thing on my 192 MB machine at home.
> I am doing a "read.csv" on a 15 MB "CSV" file. What's even more interesting is
> that this consumes so much memory that "q()" stops working!! I have to kill it
> with the Task Manager. I haven't checked to see how big my pagefile is yet,
> but it's obviously checking how much RAM is installed and hitting some kind of
> wall there. I have some bigger Linux boxes available that I can try this
> one on, so I may be able to get it to completion without waiting for the
> latest R or trimming my data set. I can't do much about the rows, but it's
> easy for me to drop unneeded columns; the raw data is coming from Microsoft
> Access.

(See Brian's post on the memory options)

R does have a rather bad habit of expanding data when they are loaded
into memory by storing into 8 byte doubles etc. Some even worse
inflation takes place during read.table (et al.) processing. In your
case, you seem to be able to read one file but not the second so you
might hope that both would actually fit when converted and try
something like

x<-read.csv(...)
save(x,file="filex")
rm(x)
y<-read.csv(...)
save(y,file="filey")
rm(y)
...

load("filex")
load("filey")

A more interesting option is to set up an ODBC connection to the
Access database and use the RODBC package, which should allow you to
select on columns, etc. You can even do that to your CSV file
(slightly tricky, I have only tried it once and it took a while to
figure out that you need to set up the ODBC connection to be to the
*directory* containing the text file(s)).

[Anyone for implementing a columns= option for scan() and read.xxx() ??]

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._