[R] Huge memory comsumption with foreign and RPgSQL
Timothy H. Keitt
Timothy.Keitt at sunysb.edu
Wed Jan 17 16:23:51 CET 2001
I usually import directly into postgresql first and then read the data
using rpgsql. In psql, create a table, e.g.,
create table my_table (col1 int, col2 float, ...)
then format your data as a tab-separated ascii file, one column per
variable. In psql,
\copy my_table from 'filename'
\g copy my_table from 'filename' using delimiters 'delim' with null
as 'null string'
Once the data are in postgresql, fire up R and read the tables with rpgsql.
Peter Dalgaard BSA wrote:
> Perttu Muurimäki <Perttu.Muurimaki at Helsinki.Fi> writes:
>> I know this is something R isn't meant to do well but I tried it anyway :)
>> I have this SPSS-datafile (size 31 MB). When I converted it to a R object
>> with read.spss("datafile.sav") I ended up with a .RData-file which was 229
>> MB big. Is this considered normal?
> Doesn't sound completely unreasonable: If all your fields fit in a
> single byte to begin with and get converted to double in the process,
> you'll have an inflation by a factor of 8.
>> Then I tried to dump that object into a database with RPgSQL-package
>> function db.write.table(object) (Memory ran out first time I tried to
>> convert SPSS-file into a R-object so I was quite prepared for the
>> database manouvre ; I increased the size of swap (working with linux) to
>> 2500 MB) The process kept going and going and getting bigger and bigger.
>> After 6 hours and 30 minutes I aborted it. At that time the process had
>> grown into 1400 MB:s. Again, is this considered normal? And further more,
>> am I likely to succeed if I'm patient enough?
> This, however, sounds a bit excessive, although I wouldn't know
> exactly what goes on inside RPgSQL... If it is converting every field
> in the entire data frame to string form before sending it to the
> database, then I might understand. Might it be possible to send it in
> smaller blocks?
Timothy H. Keitt
Department of Ecology and Evolution
State University of New York at Stony Brook
Phone: 631-632-1101, FAX: 631-632-7626
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !) To: r-help-request at stat.math.ethz.ch
More information about the R-help