Cole,
Bioconductor's high throughput sequencing infrastructure package IRanges 
contains code that may be useful for speeding up base::rbind.data.frame. 
I've extracted the salient bits from that rbind method, but left the 
corner case handling code out. IRanges's rbind method took the approach 
of treating a data set as a list of equal length columns, and so it 
contains a number of lapplys and vector concatenation calls. Given that 
base::rbind.data.frame sits at the core of many operations, I'm not sure 
if patches would be accepted for it, but I could take a crack at it.

biocRBind <- function(..., deparse.level=1)
{
   ## Simplified version of IRanges's rbind method for DataFrame
   ## Removed all data checks, ignored row names
   args <- list(...)
   df <- args[[1L]]
   cn <- colnames(df)
   cl <- unlist(lapply(as.list(df, use.names = FALSE), class))
   factors <- unlist(lapply(as.list(df, use.names = FALSE), is.factor))
   cols <- lapply(seq_len(length(df)), function(i) {
     cols <- lapply(args, `[[`, cn[i])
     if (factors[i]) { # combine factor levels, coerce to character
       levs <- unique(unlist(lapply(cols, levels), use.names=FALSE))
       cols <- lapply(cols, as.character)
     }
     combined <- do.call(c, unname(cols))
     if (factors[i])
       combined <- factor(combined, levs)
     as(combined, cl[i])
   })
   names(cols) <- colnames(df)
   do.call(data.frame, cols)
}

# create list of data.frames
set.seed(123)
dat <- vector("list", 20000)
for(i in seq_along(dat)) {
   size <- sample(1:30, 1)
   dat[[i]] <- data.frame(id=rep(i, size), value=rnorm(size), 
letter=sample(LETTERS, size, replace=TRUE), ind=sample(c(TRUE,FALSE), 
size, replace=TRUE))
}

# sample runs
 > system.time(do.call(biocRBind, dat))
    user  system elapsed
   2.120   0.000   2.125
 > system.time(do.call(biocRBind, dat))
    user  system elapsed
   2.092   0.000   2.091
 > system.time(do.call(biocRBind, dat))
    user  system elapsed
   2.080   0.000   2.077
 > sessionInfo()
R Under development (unstable) (2012-04-19 r59111)
Platform: x86_64-unknown-linux-gnu (64-bit)

locale:
  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
  [7] LC_PAPER=C                 LC_NAME=C
  [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] tools_2.16.0


Cheers,
Patrick


On 4/19/2012 3:34 PM, Cole Beck wrote:
> It's normal for me to create a list of data.frames and then use 
> do.call('rbind', list(...)) to create a single data.frame.  However, 
> I've noticed as the size of the list grows large, it is perhaps better 
> to do this in chunks.  As an example here's a list of 20,000 similar 
> data.frames.
>
> # create list of data.frames
> dat <- vector("list", 20000)
> for(i in seq_along(dat)) {
>   size <- sample(1:30, 1)
>   dat[[i]] <- data.frame(id=rep(i, size), value=rnorm(size), 
> letter=sample(LETTERS, size, replace=TRUE), ind=sample(c(TRUE,FALSE), 
> size, replace=TRUE))
> }
> # combine into one data.frame, normal usage
> # system.time(do.call('rbind', dat)) # takes 2-3 minutes
> combine <- function(x, steps=NA, verbose=FALSE) {
>   nr <- length(x)
>   if(is.na(steps)) steps <- nr
>   while(nr %% steps != 0) steps <- steps+1
>   if(verbose) cat(sprintf("step size: %s\r\n", steps))
>   dl <- vector("list", steps)
>   for(i in seq(steps)) {
>     ix <- seq(from=(i-1)*nr/steps+1, length.out=nr/steps)
>     dl[[i]] <- do.call("rbind", x[ix])
>   }
>   do.call("rbind", dl)
> }
> # combine into one data.frame
> system.time(combine(dat, 100)) # takes 5-10 seconds
>
> I'm very surprised by this result.  Does this improvement seem 
> reasonable?  I would think "do.call" could utilize something similar 
> by default when the length of "args" is too high.  Is using "do.call" 
> not recommended in this scenario?
>
> Regards,
> Cole Beck
>
> ______________________________________________
> R-devel@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel 

	[[alternative HTML version deleted]]

