[R] Reading big files in chunks-ff package

Mav mastorvarela at gmail.com
Sat Mar 24 21:29:42 CET 2012


Hello!
A question about reading large CSV files

I need to analyse several files with  sizes larger than 3 GB. Those files
have more than 10million rows (and up to 25 million) and 9 columns. Since I
don´t have a large RAM memory,  I think that the ff package can really help
me. I am trying to use read.csv.ffdf but I have some questions:

How can I read the files in several chunks…with an automatic way of
calculating the number of rows to include in each chunk? (my problem is that
the files have different number of rows)

For instance…. I have used
read.csv.ffdf(NULL, “file.csv”, sep="|", dec=".",header = T,row.names =
NULL,colClasses = c(rep("integer", 3), rep("integer", 10), rep("integer",
6)))
 But with this way I am reading the whole file....I would prefer to read it
in chunks....but I don´t know  how to read it in chunks

I have read the ff documentation but I am not good with R!
 
Thanks in advance!

--
View this message in context: http://r.789695.n4.nabble.com/Reading-big-files-in-chunks-ff-package-tp4502070p4502070.html
Sent from the R help mailing list archive at Nabble.com.



More information about the R-help mailing list