environment[[names[j]]][i]<-I does seem to take longer than
environment$name[i]<-I when looping over the length of the
vector involved.  Using a list instead of an environment seems
to work better.

Below are 4 functions that populate a data.frame row by row.
   f0 : the naīve approach
   f1 : use 3 vectors in an environment, use env$name[i] <- newValue
   f1a : use n vectors in an environment, use env[[names[j]]][i] <- newValue
   f2 : like f1a, but using a list, use env[[j]][i] <- newValue

I compared times with
  > nRows <- c(10000, 20000, 40000)
  > fs <- list(f0, f1, f1a, f2)
  > times <- matrix(NA, nrow=length(nRows), ncol=length(fs), dimnames=list(nRow=as.character(nRows), func=c("f0","f1","f1a","f2")))
  > for(i in seq_along(nRows)) for(j in seq_along(fs)) times[i,j] <- system.time(fs[[j]](nRows[i]))["elapsed"]
  > times
        func
  nRow       f0   f1   f1a   f2
   10000  8.61 0.19  1.79 0.24
   20000 22.52 0.36  5.53 0.45
   40000 72.11 0.70 17.36 0.88

f1 and f2 scale roughly linearly up 2 million rows and produce
identical results:
  > system.time(r1 <- f1(2e6))
    user  system elapsed
    48.38    2.25   52.72
  > system.time(r2 <- f2(2e6))
    user  system elapsed
    46.34    1.47   50.39
  > identical(r1, r2)
  [1] TRUE

(All timing done on a 5 year old 32-bit Windows laptop with 2 Gb of RAM.)

f0 <- function (nRow)
{
    incrSize <- 10000
    curSize <- 10000
    data <- data.frame(x = numeric(curSize), y = numeric(curSize),
        z = numeric(curSize))
    for (i in seq_len(nRow)) {
        if (i > curSize) {
            data <- rbind(data, data.frame(x = numeric(incrSize),
                y = numeric(incrSize), z = numeric(incrSize)))
            curSize <- nrow(data)
        }
        data[i, ] <- c(i + 0.1, i + 0.2, i + 0.3)
    }
    data[seq_len(nRow), , drop = FALSE]
}


f1 <- function (nRow)
{
    incrSize <- 10000
    curSize <- min(10000, nRow)
    data <- as.environment(list(x = numeric(curSize), y = numeric(curSize),
        z = numeric(curSize)))
    for (i in seq_len(nRow)) {
        if (i > curSize) {
            curSize <- min(curSize + incrSize, nRow)
            for (name in objects(data)) {
                length(data[[name]]) <- curSize
            }
        }
        data$x[i] <- i + 0.1
        data$y[i] <- i + 0.2
        data$z[i] <- i + 0.3
    }
    data.frame(as.list(data)[c("x","y","z")])
}

f1a <- function (nRow, colNames=c("x","y","z"))
{
    incrSize <- 10000
    curSize <- min(10000, nRow)
    data <- as.environment(list(x = numeric(curSize), y = numeric(curSize),
        z = numeric(curSize)))
    colNums <- seq_along(colNames)
    for (i in seq_len(nRow)) {
        if (i > curSize) {
            curSize <- min(curSize + incrSize, nRow)
            for (name in objects(data)) {
                length(data[[name]]) <- curSize
            }
        }
        for(j in colNums) {
            data[[colNames[j]]][i] <- i + j/10
        }
    }
    data.frame(as.list(data)[colNames])
}

f2 <- function (nRow, colNames=c("x","y","z"))
{
    incrSize <- 10000
    curSize <- min(10000, nRow)
    colNums <- seq_along(colNames)
    data <- lapply(colNames, function(nm) numeric(curSize))
    for (i in seq_len(nRow)) {
        if (i > curSize) {
            curSize <- min(curSize + incrSize, nRow)
            for (j in colNums) {
                length(data[[j]]) <- curSize
            }
        }
        for(j in colNums) {
            data[[j]][i] <- i + j/10
        }
    }
    names(data) <- colNames
    data.frame(data)
}

Bill Dunlap
Spotfire, TIBCO Software
wdunlap tibco.com
From: Peter Meilstrup [mailto:peter.meilstrup@gmail.com]
Sent: Wednesday, February 15, 2012 1:39 AM
To: William Dunlap
Cc: r-help@r-project.org
Subject: Re: [R] Filling out a data frame row by row.... slow!

On Tue, Feb 14, 2012 at 2:31 PM, William Dunlap <wdunlap@tibco.com<mailto:wdunlap@tibco.com>> wrote:
If you must repeatedly append rows to a data.frame,
try making the dataset you are filling in a bunch
of independent vectors, perhaps in a new environment
to keep things organized.

One complication is I don't know the names of the columns I'm assigning to before I read them off the file. And crazily, if I change this:
       data$x[i] <- i + 0.1

where data is an environment and x a primitive vector, to use a computed name instead:

 data[[colname]][i] <- i + 0.1

Then I get back to way-superlinear performance. Eventually I found I could work around it like:

eval(substitute(var[ix] <- data,
                          list(var=as.name<http://as.name>(colname), ix=i, data = i+0.1)),
               envir = data)

but... as workarounds go that seems to be on the crazy nuts end of the scale. Why does [[]] impose a performance penalty?

Peter

	[[alternative HTML version deleted]]

