[R] poor rbind performance

[jim holtman]

Read the data into a list and then:

do.call('rbind', myList)

at the end so you do it only once.  You are having to reallocate
memory each iteration, so no wonder it is slow.

On 7/17/07, Aydemir, Zava (FID) <Zava.Aydemir at morganstanley.com> wrote:
> Hi
>
> I rbind data frames in a loop in a cumulative way and the performance
> detriorates very quickly.
>
> My code looks like this:
>
> for( k in 1:N)
> {
>    filename <- paste("/tmp/myData_",as.character(k),".txt",sep="")
>    myDataTmp <- read.table(filename,header=TRUE,sep=",")
>    if( k == 1) {
>        myData <- myDataTmp
>    }
>    else{
>        myData <- rbind(myData,myDataTmp)
>    }
> }
>
> Some more details:
> - the size of the stored text files is about 100,000 rows and 50 columns
> each
> - for k=1: rbind takes 0.0004 seconds
> - for k=2: rbind takes 13 seconds
> - for k=3: rbind takes 30 seconds
> - for k=4: rbind takes 36 seconds
> etc
>
> Any suggestions to improve speed?
>
> Thanks
>
> Zava
> --------------------------------------------------------
>
> This is not an offer (or solicitation of an offer) to buy/se...{{dropped}}
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Jim Holtman
Cincinnati, OH
+1 513 646 9390

What is the problem you are trying to solve?