[R] extracting from data frame multiple times

[Thomas Lumley]

On Fri, 14 Oct 2005, Ido M. Tamir wrote:

> Hello,
> i am trying to subset a dataframe multiple times:
> something like:
>
> stats <- by(df, list(items), ttestData)
>
> ttestData <- function(df){
>    t.test( df[,c(2,3,4), df[,c(5,6,7)]
> }
>
> While this works for small data, it is to slow for my
> actual data: 500000 rows dataframe with
> about 135000 different indices, subsetting the
> dataframe into chunks of 5 on average.
>
> Do you have any suggestions how I could speed this up?

The first step is to find out what is too slow, using Rprof().  It may be 
the t.test or it may be the by().

If it is the by() you could put the numeric data into two matrices
   x1<-df[,2:4]
   x2<-df[,5:7]
order them so that the same "item" entries were adjacent, compute the 
start and end indices for each group, and do something like
lapply(1:howevermany, function(i) t.test(x1[start[i]:end[i],],x2[start[i]:end[i]))
Even just turning df into a matrix might help

If it is the repeated t.test() calls that are too slow you need to speed 
them up.  You can probably rowsum() to compute means and variances for all 
the t-tests at once.

 	-thomas