[R] Process File Line By Line Without Slurping into Object

[Duncan Murdoch]

Gundala Viswanath wrote:
> Dear all,
>
> In general practice one would slurp the whole file
> using this method before processing the data:
>
> dat <- read.table(filename)
>
> or variations of it.
>
> Is there a way we can access the file line by line
> without slurping/storing them into object?
>
> I am thinking something like this in Perl:
>
> __BEGIN__
> open INFILE, '<' , 'filename.txt' or die $!;
> while (<INFILE>) {
>     my $line = $_;
>     # then process line by line
> }
> __END__
>
> the reason I want to do that is because the data
> I am processing are large (~ 5GB), my PC may not
> be able to handle that.
>   
Use a connection.  For example,

con <- file("filename.txt", "rt")
repeat {
    line <- readLines(con, 1)
    if (!length(line)) break
    # process the line, or change the "1" to a bigger number, and 
process the block of lines
}
close(con)

Duncan Murdoch