I am using readLines to read a fairly large ASCII file.  readLines reads
a fixed number of lines, then other R code processes the data, then
readLines reads the same number of lines again, then other R code
processes the data, then ....

 

Sort of like:

 

conn<-file('filename','r')

for (chunk in 1:100000) {

   Lines<-readLines(conn,n=25)

  # process "Lines"

}

 

The code is working, but I notice that it slows down greatly as time
progresses.  It took 2 seconds to read my first chunk of data, 4 seconds
to read the next chunk, 10 after that.  The quasi-exponential trend has
slowed, thank goodness, but after about a hundred reads, the read time
for the next chunk is over a minute.  Let me stress that the number of
lines read in each chunk of data is absolutely fixed.

 

The only processing I am doing at the point is to parse the new data,
and rbind the results to an existing data frame.  Processing of new data
in no way depends on earlier data.

 

So, my question is why is the reading taking longer as time goes on?  Is
there a way to fix this?  Is there a better method than readLines?

 

Thanks.


	[[alternative HTML version deleted]]

