[R] R tools for large files

[Andrew C. Ward]

Dear Murray,

One way that works very well for many people (including me)
is to store the data in an external database, such as MySQL,
and read in just the bits you want using the excellent
package RODBC. Getting a database to do all the selecting
is very fast and efficient, leaving R to concentrate on the
analysis and visualisation. This is all described in the
R Import/Export Manual.


Regards,

Andrew C. Ward

CAPE Centre
Department of Chemical Engineering
The University of Queensland
Brisbane Qld 4072 Australia
andreww at cheque.uq.edu.au


Quoting Murray Jorgensen <maj at stats.waikato.ac.nz>:

> I'm wondering if anyone has written some functions or
> code for handling 
> very large files in R. I am working with a data file that
> is 41 
> variables times who knows how many observations making up
> 27MB altogether.
> 
> The sort of thing that I am thinking of having R do is
> 
> - count the number of lines in a file
> 
> - form a data frame by selecting all cases whose line
> numbers are in a 
> supplied vector (which could be used to extract random
> subfiles of 
> particular sizes)
> 
> Does anyone know of a package that might be useful for
> this?
> 
> Murray
> 
> -- 
> Dr Murray Jorgensen     
> http://www.stats.waikato.ac.nz/Staff/maj.html
> Department of Statistics, University of Waikato,
> Hamilton, New Zealand
> Email: maj at waikato.ac.nz                               
> Fax 7 838 4155
> Phone  +64 7 838 4773 wk    +64 7 849 6486 home    Mobile
> 021 1395 862
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>