[R] Dealing With Extremely Large Files

[Gabor Grothendieck]

Not sure if it applies to your file or not but if it does then the
sqldf package facilitates reading a large file into an SQLite database.
Its a front end to RSQLite which is a front end to SQLite and it
reads the data straight into the database without going through
R so R does not limit it in any way -- its only actuated from R.
The code to do this is basically just two lines of code.  You don;t
have to install database software (its included with RSQLite package)
and you don't have to set up a database at all -- it does that for you
automatically.

See example 6e on the home page which creates a database
transparently, reads in the data and extracts random rows from
the database into R:
http://sqldf.googlecode.com

On Fri, Sep 26, 2008 at 3:55 PM, zerfetzen <zerfetzen at yahoo.com> wrote:
>
> Hi,
> I'm sure that a large fixed width file, such as 300 million rows and 1,000
> columns, is too large for R to handle on a PC, but are there ways to deal
> with it?
>
> For example, is there a way to combine some sampling method with read.fwf so
> that you can read in a sample of 100,000 records, for example?
>
> Something like this may make analysis possible.
>
> Once analyzed, is there a way to, say, read in only x rows at a time, save
> and score each subset separately, and finally append them back together?
>
> I haven't seen any information on this, if it is possible.  Thank you for
> reading, and sorry if the information was easily available and I simply
> didn't find it.
> --
> View this message in context: http://www.nabble.com/Dealing-With-Extremely-Large-Files-tp19695311p19695311.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>