[R] write.table and large datasets

[Ott Toomet]

Hi,

I'll continue the discussion about the write.table() and problems with large
datasets.

The databases I have to work with are quite huge, 7500 obs x 1200 vars were
on of the smallest of them.  I usually write a perl script to preprocess
them line-by-line and extract only the variables which I need later.  This
results into quite a manageable size but I have to have the dataset in ASCII
form.

As it appears, R could easily read in a dataset of that size (it was in
STATA format) but the conversion to ascii was quite complicated.  I
remember, I have done it with a larger dataset, using SPSS and a rougly
equal computer (128M memory).  It did not took much time, perhaps a minute
or so.  That's why I was surprised that apparently similar function in R was
unable to save the dataset at all.

I still think that it would be a good idea to be able to transform big
datasets to ascii, it is perhaps the most simple and open standard which I
think everybody else can use too.  Perhaps it is more meaningful to make a
less-general function (in package external) which can save at least numeric
and character variables?  I have made some attempts myself but these are
still in very early stage.

So, what is the general opinion -- is such kind of a conversion utility
needed in R?

Regards,

Ott Toomet


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._