[R] How to separate huge dataset into chunks
filteau at unc.edu
Tue Mar 24 09:41:04 CET 2009
Im trying to take a huge dataset (1.5 GB) and separate it into smaller
chunks with R.
So far I had nothing but problems.
I cannot load the whole dataset in R due to memory problems. So, I
instead try to load a few (100000) lines at a time (with read.table).
However, R kept crashing (with no error message) at about the 6800000
line. This is extremely frustrating.
To try to fix this, I used connections with read.table. However, I now
get a cryptic error telling me no lines available in input.
Is there any way to make this work?
More information about the R-help