[R] Working with large datafiles
tlumley at u.washington.edu
Mon Oct 4 17:42:38 CEST 2004
On Mon, 4 Oct 2004, Greg Butler wrote:
> I have been enjoying r for some time now, but was wondering about working
> with larger data files. When I try to load in big files with more than
> 20,000 records, the programs seems unbable to store all the records. Is
> there some way that I can increase the size of records that I work with?
> Ideally I would like to work with census data which can hold a million
You should be able to handle 20,000 records on a reasonable computer (my
laptop, with 256Mb memory can, very slowly, do survey analyses on a file
with 26,000 records and about 100 variables).
A million records is likely to be infeasible. A 32bit computer can't
even address enough memory to store that much data. You would need to put
the data either in a database or in a file format such as netCDF or hdf5
that allows smaller chunks to be read and processed.
More information about the R-help