I'm reading a file and using the file to populate a data frame. The way the
file is laid out, I need to fill in the data frame one row at a time.

When I start reading my file, I don't know how many rows I will need. It's
on the order of a million.

Being mindful of the time expense of reallocation, I decided on a strategy
of doubling the data frame size every time I needed to expand it ...
therefore memory is never more than 50% wasted, and it should still finish
in O(N) time.

But it still, somehow has an O(N^2) performance characteristic. It seems
like just setting a single element is slow in a larger data frame as
compared to a smaller one. Here is a toy function to illustrate,
reallocating and filling in single rows in a data frame, and shows the
slowdown:

populate.data.frame.test <- function(n=1000000, chunk=1000) {
  i = 0;
  df <- data.frame(a=numeric(0), b=numeric(0), c=numeric(0));
  t <- proc.time()[2]
  for (i in 1:n) {
    if (i %% chunk == 0) {
      elapsed <- -(t - (t <- proc.time()[2]))
      cat(sprintf("%d rows: %g rows per sec, nrows = %d\n", i,
chunk/elapsed, nrow(df)))

    }

    ##double data frame size if necessary
    while (nrow(df)<i) {
      df[max(i, 2*nrow(df)),] <- NA
      cat(sprintf("Doubled to %d rows\n", nrow(df)));
    }

    ##fill in one row
    df[i, c('a', 'b', 'c')] <- list(runif(1), i, runif(1))
  }
}

Is there a way to do this that avoids the slowdown? The data cannot be
represented as a matrix (different columns have different types.)

Peter

	[[alternative HTML version deleted]]

