[R] Incremental ReadLines

William Dunlap wdunlap at tibco.com
Thu Apr 14 22:58:33 CEST 2011

[see below]

From: Frederik Lang [mailto:frederiklang at gmail.com] 
Sent: Thursday, April 14, 2011 12:56 PM
To: William Dunlap
Cc: r-help at r-project.org
Subject: Re: [R] Incremental ReadLines

	Hi Bill,
	Thank you so much for your suggestions. I will try and alter my
	Regarding the even shorter solution outside the loop it looks
good but my problem is that not all observations have the same variables
so that three different observations might look like this:
	Id: 1
	Var1: false
	Var2: 6
	Var3: 8
	Id: 2
	Id: 3
	Var1: true
	3 4 5
	Var2: 7
	Var3: 3
	Doing it without looping through I thought my data had to quite
systematic, which it is not. I might be wrong though.

Doing the simple preallocation that I describe should speed it up
a lot with very little effort.  It is more work to manipulate the
columns one at a time instead of using data.frame subscripting and
it may not be worth it if you have lots of columns.

If you have a lot of this sort of file and feel that it will be worth
the programming time to do something fancier, here is some code that
reads lines of the form

> cat(lines, sep="\n")
Id: First
  Var1: false
  Var2: 6
  Var3: 8

Id: Second
Id: Last
  Var1: true
  Var3: 8

and produces a matrix with the Id's along the rows and the Var's
along the columns:

> f(lines)
       Var1    Var2 Var3
First  "false" "6"  "8"
Second NA      NA   NA
Last   "true"  NA   "8"

The function f is:

f <- function (lines)
    # keep only lines with colons
    lines <- grep(value = TRUE, "^.+:", lines)
    lines <- gsub("^[[:space:]]+|[[:space:]]+$", "", lines)
    isIdLine <- grepl("^Id:", lines)
    group <- cumsum(isIdLine)
    rownames <- sub("^Id:[[:space:]]*", "", lines[isIdLine])
    lines <- lines[!isIdLine]
    group <- group[!isIdLine]
    varname <- sub("[[:space:]]*:.*$", "", lines)
    value <- sub(".*:[[:space:]]*", "", lines)
    colnames <- unique(varname)
    col <- match(varname, colnames)
    retval <- array(NA_character_, c(length(rownames),
        dimnames = list(rownames, colnames))
    retval[cbind(group, col)] <- value

The main trick is the matrix subscript given to retval on the
penultimate line.

	Thanks again,
	On Thu, Apr 14, 2011 at 12:56 PM, William Dunlap
<wdunlap at tibco.com> wrote:

		I have two suggestions to speed up your code, if you
		must use a loop.
		First, don't grow your output dataset at each iteration.
		Instead of
		    cases <- 0
		    output <- numeric(cases)
		    while(length(line <- readLines(input, n=1))==1) {
		       cases <- cases + 1
		       output[cases] <- as.numeric(line)
		preallocate the output vector to be about the size of
		its eventual length (slightly bigger is better),
		    output <- numeric(0)
		with the likes of
		    output <- numeric(500000)
		and when you are done with the loop trim down the length
		if it is too big
		    if (cases < length(output)) length(output) <- cases
		Growing your dataset in a loop can cause quadratic or
		growth in time with problem size and the above sort of
		code should make the time grow linearly with problem
		Second, don't do data.frame subscripting inside your
		Instead of
		    data <- data.frame(Id=numeric(cases))
		    while(...) {
		        data[cases, 1] <- newValue
		    Id <- numeric(cases)
		    while(...) {
		        Id[cases] <- newValue
		    data <- data.frame(Id = Id)
		This is just the general principal that you don't want
		repeat the same operation over and over in a loop.
		dataFrame[i,j] first extracts column j then extracts
		i from that column.  Since the column is the same every
		you may as well extract the column outside of the loop.
		Avoiding the loop altogether is the fastest.  E.g., the
		you showed does the same thing as
		  idLines <- grep(value=TRUE, "Id:", readLines(file))
		  data.frame(Id = as.numeric(sub("^.*Id:[[:space:]]*",
"", idLines)))
		You can also use an external process (perl or grep) to
		out the lines that are not of interest.
		Bill Dunlap
		Spotfire, TIBCO Software
		wdunlap tibco.com

		> -----Original Message-----
		> From: r-help-bounces at r-project.org
		> [mailto:r-help-bounces at r-project.org] On Behalf Of
		> Sent: Wednesday, April 13, 2011 10:58 AM
		> To: r-help at r-project.org
		> Subject: Re: [R] Incremental ReadLines
		> Hi there,
		> I am having a similar problem with reading in a large
		> file with around
		> 550.000 observations with each 10 to 100 lines of
		> description. I am trying
		> to parse it in R but I have troubles with the size of
		> file. It seems
		> like it is slowing down dramatically at some point. I
		> be happy for any
		> suggestions. Here is my code, which works fine when I
		> doing a subsample
		> of my dataset.
		> #Defining datasource
		> file <- "filename.txt"
		> #Creating placeholder for data and assigning column
		> data <- data.frame(Id=NA)
		> #Starting by case = 0
		> case <- 0
		> #Opening a connection to data
		> input <- file(file, "rt")
		> #Going through cases
		> repeat {
		>   line <- readLines(input, n=1)
		>   if (length(line)==0) break
		>   if (length(grep("Id:",line)) != 0) {
		>     case <- case + 1 ; data[case,] <-NA
		>     split_line <- strsplit(line,"Id:")
		>     data[case,1] <- as.numeric(split_line[[1]][2])
		>     }
		> }
		> #Closing connection
		> close(input)
		> #Saving dataframe
		> write.csv(data,'data.csv')
		> Kind regards,
		> Frederik
		> --
		> View this message in context:
		> Sent from the R help mailing list archive at
		> ______________________________________________
		> R-help at r-project.org mailing list
		> https://stat.ethz.ch/mailman/listinfo/r-help
		> PLEASE do read the posting guide
		> http://www.R-project.org/posting-guide.html
		> and provide commented, minimal, self-contained,
reproducible code.

More information about the R-help mailing list