[R] R tools for large files

[Duncan Murdoch]

Murray:

You said that you wanted more control over the selection of lines than
scan() was giving you.  This may have been suggested by someone else
(there have been a lot of  messages in this thread!), but I think what
you want to do is to "roll your own" reader, using textConnection.

For example, if you want to read lines 1000 through 1100, you'd do it
like this:

 lines <- readLines("foo.txt", 1100)[1000:1100]
 result <- scan(textConnection(lines), list( .... ))

This will be inefficient with very large files if you want to read
every nth line or a block at the end of the file, because the first
line reads a contiguous block and then subsets.  In that case you
probably want to write your own loop that builds up the lines variable
first.

Duncan Murdoch