All interesting suggestions.

I guess a better example of the code would have been a good idea.  So,
I'll put a relevant snippet here.

Rows are cases.  There are multiple cases for each ID, marked with a
date.  I'm trying to calculate a time recency weighted score for a
covariate, added as a new column in the data.frame.

So, for each row, I need to see which ID it belongs to, then get all the
scores prior to this row's date, then compute the recency weighted summary.

Right now, I do this in an obvious, but very very slow way.

Here is my slow code:
======================
for(i in 1:nrow(d)){
    for(j in which( d$id == d$id[i] & d$date[j] < d$date[i]) ){
        days_since = as.numeric( d$date[i] - d$date[j] )
        w <- exp( -days_since/decay )
        temp <- temp + w * as.numeric(d[j,'score'])
        wTemp <- wTemp + w
    }

    temp <- temp / wTemp
    d$newScore[i,] <- temp
}
======================

One immediate thought was to turn the "date" into an integer.  That
should save a few cycles of date math.

I need to do this process for a bunch of scores.  A grid search over
different time decay levels might be nice.  So any speedup to this
routine will save me a ton of time.

Ideas?

----

Noah Silverman, M.S., C.Phil
UCLA Department of Statistics
8117 Math Sciences Building
Los Angeles, CA 90095

On 11/21/13, 5:51 AM, Rainer M Krug wrote:
>
>
> On 11/21/13, 12:34 , Jim Holtman wrote:
> > you need to show the statement in context with the rest of the
> > script.  you need to tell us what you want to do, not how you want
> > to do it.
>
> Agreed - a few details will result in guesses (see my guess below)
>
>
> > Sent from my iPad
>
> > On Nov 20, 2013, at 15:16, Noah Silverman
> > <noahsilverman@g.ucla.edu> wrote:
>
> >> Hello,
> >>
> >> I have a fairly large data.frame.  (About 150,000 rows of 100
> >> variables.) There are case IDs, and multiple entries for each ID,
> >> with a date stamp.  (i.e. records of peoples activity.)
> >>
> >>
> >> I need to iterate over each person (record ID) in the data set,
> >> and then process their data for each date.  The processing part
> >> is fast, the date part is fast.  Locating the records is slow.
> >> I've even tried using data.table, with ID set as the index, and
> >> it is still slow.
> >>
> >> The line with the slow process (According to Rprof) is:
> >>
> >>
> >> j <- which( d$id == person )
>
> Possibly use
>
> d_by_id <- split(d, d$id)
>
> which splits the data.frame d into a listt, where each list represents
> the data.frame of one id.
>
> But: Just a guess.
>
> Cheers,
>
> Rainer
>
> >>
> >> (I then process all the records indexed by j, which seems fast
> >> enough.)
> >>
> >> where d is my data.frame or data.table
> >>
> >> I thought that using the data.table indexing would speed things
> >> up, but not in this case.
> >>
> >> Any ideas on how to speed this up?
> >>
> >>
> >> Thanks!
> >>
> >> -- Noah Silverman, M.S., C.Phil UCLA Department of Statistics
> >> 8117 Math Sciences Building Los Angeles, CA 90095
> >>
> >> ______________________________________________
> >> R-help@r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the
> >> posting guide http://www.R-project.org/posting-guide.html and
> >> provide commented, minimal, self-contained, reproducible code.
>
> > ______________________________________________ R-help@r-project.org
> > mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do
> > read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
>


	[[alternative HTML version deleted]]

