I have very large csv files (up to 1GB each of ASCII text). I'd like to be able to read them directly in to R. The problem I am having is with the variable length of the data in each record.
   
  Here's a (simplified) example:
   
  $ cat foo.csv
Name,Start Month,Data
Foo,10,-0.5615,2.3065,0.1589,-0.3649,1.5955
Bar,21,0.0880,0.5733,0.0081,2.0253,-0.7602,0.7765,0.2810,1.8546,0.2696,0.3316,0.1565,-0.4847,-0.1325,0.0454,-1.2114
   
  The records consist of rows with some set comma-separated fields (e.g. the "Name" & "Start Month" fields in the above) and then the data follow as a variable-length list of comma-separated values until a new line is encountered.
   
  Now I can use e.g.
   
  fileName="foo.csv"  
ta<-read.csv(fileName, header=F, skip=1, sep=",", dec=".", fill=T)  
   
  which does the job nicely:
   
     V1 V2      V3     V4     V5      V6      V7     V8    V9    V10    V11    V12    V13     V14     V15    V16     V17
1 Foo 10 -0.5615 2.3065 0.1589 -0.3649  1.5955     NA    NA     NA     NA     NA     NA      NA      NA     NA      NA
2 Bar 21  0.0880 0.5733 0.0081  2.0253 -0.7602 0.7765 0.281 1.8546 0.2696 0.3316 0.1565 -0.4847 -0.1325 0.0454 -1.2114

   
  but the problem is with files on the order of 1GB this either crunches for ever or runs out of memory trying ... plus having all those NAs isn't too pretty to look at. 
   
  (I have a MATLAB version that can read this stuff into an array of cells in about 3 minutes).
   
  I really want a fast way to read the data part into a list; that way I can access data in the array of lists containing the records by doing something ta[[i]]$data.
   
  Ideas?
   
  Thanks,
   
  Jack.

			
---------------------------------


	[[alternative HTML version deleted]]


