[R] Why are big data.frames slow? What can I do to get it faster?

[Uwe Ligges]

Marcus Jellinghaus wrote:
> 
> Hello,
> 
> I´m quite new to this list.
> I have a high frequency-dataset with more than 500.000 records.
> I want to edit a data.frame "Test". My small programm runs fine with a small
> part of the dataset (just 100 records), but it is very slow with a huge
> dataset. Of course it get´s slower with more records, but when I change just
> the size of the frame and keep the number of edited records fixed, I see
> that it is also getting slower.
> 
> Here is my program:
> 
> print(dim(test)[1])
> Sys.time()
> for(i in 1:100) {
>   test[i,6] = paste(test[i,2],"-",test[i,3], sep = "")
> }
> Sys.time()
> 
> I connect 2 currency symbols to a currency pair.
> I always calculate only for the first 100 lines.
> WHen I load just 100 lines in the data.frame "test", it takes 1 second.
> When I load 1000 lines, editing 100 lines takes 2 seconds,
> 10,000 lines loaded and 100 lines editing takes 5 seconds,
> 100,000 lines loaded and editing 100 lines takes 31 seconds,
> 500,000 lines loaded and editing 100 lines takes 11 minutes(!!!).
> 
> My computer has 1 GB Ram, so that shouldn´t be the reason.
> 
> Of course, I could work with many small data.frames instead of one big, but
> the program above is just the very first step and so I don´t want to split.
> 
> Is there a way to edit big data.frames without waiting for a long time?

Well, the point is, I guess, to address elements in a large data.frame,
which reasonably takes much more time than in a small one.

Maybe it's an idea to use vectorized operations instead of the loop,
which is preferable, if your computation is easy vectorizable without a
big penalty of memory consumption:

 test[1:100, 6] <- paste(test[1:100, 2], "-", test[1:100, 3], sep = "")
or 
 test[ , 6] <- paste(test[ , 2], "-", test[ , 3], sep = "")
for the whole data.frame.

Uwe Ligges
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._