[R] qu: predict with lmer (lme4) or other ways to get classification accuracy

[Spencer Graves]

<see inline>

T. Florian Jaeger wrote:
> Spencer,
> 
> Thanks a lot (and sorry that I wasn't more precise). It seems to work. I 
> assume this gives me the predicted log-odds for each case? 
SG:  Correct.

So, I could
> get the classification accuracy as follows? (where data.lmer is my data; 
> REL is the outcome variable)
> 
> data.lmer$lo.fit.lmer <- predict.lmerBin(fit)
> pred.fit.lmer <- exp(data.lmer$lo.fit.lmer) / 
> (exp(data.lmer$lo.fit.lmer) + 1) > 0.5
> prop.table(table(pred.fit.lmer==data.lmer$REL))     # overall accuracy
> prop.table(table(data.lmer$REL, pred.fit.lmer),1)    # accuracy by 
> actual outcome

SG:  This looks plausible, but I don't do enough "classification" work 
to feel comfortable commenting further on this.
> 
> Also, predict.lmerBin doesn't consider the random effects?  I was
> thinking of something like the following (there are two random effects 
> RA and RB, which are already coded as a factor with levels 1...k and 
> 1...l in the same order that you get with ranef( lmer.object); all cases 
> in data don't have missing values):
> 
> fit <- lmer(dependent ~ fixed1 + (1|RA) + (1|RB), data=data)
> lo.fit <- predict.lmerBin(fit)                      # predicted log-odds
> ransp <- unlist(ranef(fit)["RA"])           
> rannp <- unlist(ranef(fit)["RB"])
> 
> for (i in 1:length(lo.fit)) {
>     lo.fit[i]<- lo.fit[i] + ransp[data$RA[i]] + rannp[data$RB[i]]
> }

SG:  This looks plausible, but since you still have not given me a 
self-contained example, I can't comment further.  Have you considered 
'coef(fit)'?  that might also help you.
> 
> I am sure that's very unelegant, but I hope it does the right thing? It 
> looks ok in some tests I ran.
> 
> cheers,
> 
> florian
> 
> On 5/7/06, *Spencer Graves* <spencer.graves at pdf.com 
> <mailto:spencer.graves at pdf.com>> wrote:
> 
>               Checking 'help("lmer")' leads me also to
>     'help("lmer-class")', both
>     of which are helpful.  Unfortunately, I know of no general "predict"
>     method that's available for an object of class 'lmer'.  A preliminary
>     function of that nature is as follows:
> 
> 
>     predict.lmerBin <- function(object, X){
>     # object has class "lmer"
>     # X = model matrix with columns
>     #     matching object at X
>        if(missing(X))
>          X <- object at X
>     #
>        b <- fixef(object)
>        X %*% b
>     }
> 
> 
>               ## To use this, you need to know how to use
>     # 'preduct.glm'.
> 
>     # Example using data(Contraception), discussed in
>     (mlmR <- vignette("MlmSoftRev")) # opens in Adobe Acrobat
>     # To open a script file companion to vignette("MlmSoftRev")
>     #edit(mlmR) # with Rgui
>     #Stangle(mlmR$file) # with ESS
> 
>     fitBin <- lmer(use ~ urban+age+livch+(1|district),
>                     Contraception, binomial)
>     predict.lmerBin(fitBin)
> 
>               Does this answer your question?
>               Spencer Graves
>     p.s.  If your example had been simple and self-contained, it would have
>     been to reply, because I could copy a few lines of R code from your
>     email into R, tested a few ideas, and craft a reply in a very few
>     minutes, if not seconds.  Without that, crafting a sensible reply takes
>     more time, partly because it's less clear what you need to know to move
>     to the next step, and partly because after I guess what you are really
>     asking, I must next hunt for a suitable example.
> 
>     T. Florian Jaeger wrote:
>      > Hi,
>      >
>      > I am using lmer (from the package lme4) to predict a binary response
>      > variable (REL) from a bunch of fixed effects and two random effects
>      > (Speaker_ID and NPhead_lemma):
>      >
>      > fit <- lmer(REL ~
>      >       SPEAKER_GENDER +
>      >       log(SPEECHRATE) +
>      >       SQSPEECHRATE +
>      >         .....
>      >         + (1|Speaker_ID) + (1|NPhead_lemma),
>      >       family="binomial",
>      >       data=data.lmer,
>      >       method="Laplace", model=T, x=T)
>      >
>      > I would like to get classification accuracies for the derived model,
>      > or even do some evaluation (cross-validation), but I cannot find
>     a way
>      > to get to the predicted values. As far as I can tell no predict
>     method
>      > has been implemented for lmer, right? I also was trying to figure
>     out
>      > whether lmer stores the predicted values somewhere (I read the
>      > documentation and implementation summary,
>      > /library/lme4/doc/Implementation.pdf).
>      >
>      > Many thanks for your help and my apologies if I overlooked something
>      > really simple.
>      >
>      > Florian
>      >
>      > --
>      > T. Florian Jaeger
>      > Ph.D. student
>      > Linguistics Department,
>      > P: +1 (650) 725 2323
>      > F: +1 (650) 723 5666
>      > U: http://www.stanford.edu/~tiflo/
>      >
>      > ______________________________________________
>      > R-help at stat.math.ethz.ch <mailto:R-help at stat.math.ethz.ch>
>     mailing list
>      > https://stat.ethz.ch/mailman/listinfo/r-help
>      > PLEASE do read the posting guide!
>     http://www.R-project.org/posting-guide.html
> 
> 
> 
> 
> -- 
> T. Florian Jaeger                              
> Ph.D. student
> Linguistics Department,                              
> P: +1 (650) 725 2323
> F: +1 (650) 723 5666
> U: http://www.stanford.edu/~tiflo/