hi uwe---thanks for the clarification.  of course, my example should always
be done in vectorized form.  I only used it to show how iterative access
compares in the simplest possible fashion.  <100 accesses per seconds is
REALLY slow, though.

I don't know R internals and the learning curve would be steep.  moreover,
there is no guarantee that changes I would make would be accepted.  so, I
cannot do this.

however, for an R expert, this should not be too difficult.  conceptually,
if data frame element access primitives are create/write/read/destroy in the
code, then it's truly trivial.  just add a matrix (dim the same as the data
frame) of byte pointers to point at the storage upon creation/change time.
 this would be quick-and-dirty.  for curiosity, do you know which source
file has the data frame internals?  maybe I will get tempted anyway if it is
simple enough.

(a more efficient but more involved way to do this would be to store a data
frame internally always as a matrix of data pointers, but this would
probably require more surgery.)

It is also not as important for me, as it is for others...to give a good
impression to those that are not aware of the tradeoffs---which is most
people considering to adopt R.

/iaw


----
Ivo Welch (ivo.welch@gmail.com)




2011/7/2 Uwe Ligges <ligges@statistik.tu-dortmund.de>

> Some comments:
>
> the comparison matrix rows vs. matrix columns is incorrect: Note that R has
> lazy evaluation, hence you construct your matrix in the timing for the rows
> and it is already constructed in the timing for the columns, hence you want
> to use:
>
>  M <- matrix( rnorm(C*R), nrow=R )
>  D <- as.data.frame(matrix( rnorm(C*R), nrow=R ) )
>  example(M)
>  example(D)
>
> Further on, you are correct with you statement that data.frame indexing is
> much slower, but if you can store your data in matrix form, just go on as it
> is.
>
> I doubt anybody is really going to make the index operation you cited
> within a loop. Then, with a data.frame, I can live with many vectorized
> replacements again:
>
> > system.time(D[,20] <- sqrt(abs(D[,20])) + rnorm(1000))
>   user  system elapsed
>   0.01    0.00    0.01
>
> > system.time(D[20,] <- sqrt(abs(D[20,])) + rnorm(1000))
>   user  system elapsed
>   0.51    0.00    0.52
>
> OK, it would be nice to do that faster, but this is not easy. I think R
> Core is happy to see contributions to make it faster without breaking
> existing features.
>
>
>
> Best wishes,
> Uwe
>
>
>
>
> On 02.07.2011 20:35, ivo welch wrote:
>
>> This email is intended for R users that are not that familiar with R
>> internals and are searching google about how to speed up R.
>>
>> Despite common misperception, R is not slow when it comes to iterative
>> access.  R is fast when it comes to matrices.  R is very slow when it
>> comes to iterative access into data frames.  Such access occurs when a
>> user uses "data$varname[index]", which is a very common operation.  To
>> illustrate, run the following program:
>>
>> R<- 1000; C<- 1000
>>
>> example<- function(m) {
>>   cat("rows: "); cat(system.time( for (r in 1:R) m[r,20]<-
>> sqrt(abs(m[r,20])) + rnorm(1) ), "\n")
>>   cat("columns: "); cat(system.time(for (c in 1:C) m[20,c]<-
>> sqrt(abs(m[20,c])) + rnorm(1)), "\n")
>>   if (is.data.frame(m)) { cat("df: columns as names: ");
>> cat(system.time(for (c in 1:C) m[[c]][20]<- sqrt(abs(m[[c]][20])) +
>> rnorm(1)), "\n") }
>> }
>>
>> cat("\n**** Now as matrix\n")
>> example( matrix( rnorm(C*R), nrow=R ) )
>>
>> cat("\n**** Now as data frame\n")
>> example( as.data.frame( matrix( rnorm(C*R), nrow=R ) ) )
>>
>>
>> The following are the reported timing under R 2.12.0 on a Mac Pro 3,1
>> with ample RAM:
>>
>> matrix, columns: 0.01s
>> matrix, rows: 0.175s
>> data frame, columns: 53s
>> data frame, rows: 56s
>> data frame, names: 58s
>>
>> Data frame access is about 5,000 times slower than matrix column
>> access, and 300 times slower than matrix row access.  R's data frame
>> operational speed is an amazing 40 data accesses per seconds.  I have
>> not seen access numbers this low for decades.
>>
>>
>> How to avoid it?  Not easy.  One way is to create multiple matrices,
>> and group them as an object.  of course, this loses a lot of features
>> of R.  Another way is to copy all data used in calculations out of the
>> data frame into a matrix, do the operations, and then copy them back.
>> not ideal, either.
>>
>> In my opinion, this is an R design flow.  Data frames are the
>> fundamental unit of much statistical analysis, and should be fast.  I
>> think R lacks any indexing into data frames.  Turning on indexing of
>> data frames should at least be an optional feature.
>>
>>
>> I hope this message post helps others.
>>
>> /iaw
>>
>> ----
>> Ivo Welch (ivo.welch@gmail.com)
>> http://www.ivo-welch.info/
>>
>> ______________________________**________________
>> R-help@r-project.org mailing list
>> https://stat.ethz.ch/mailman/**listinfo/r-help<https://stat.ethz.ch/mailman/listinfo/r-help>
>> PLEASE do read the posting guide http://www.R-project.org/**
>> posting-guide.html <http://www.R-project.org/posting-guide.html>
>> and provide commented, minimal, self-contained, reproducible code.
>>
>

	[[alternative HTML version deleted]]

