[R-sig-ME] Reliability via mixed effects modelling

[Viechtbauer Wolfgang (STAT)]

Hi Mike,

I haven't read your mail in detail, but I know that some work has been done on that issue. See, for example:

Laenen, A. et al. (2006). Generalized reliability estimation using repeated measurements. British Journal of Mathematical and Statistical Psychology, 59, 113-131. http://onlinelibrary.wiley.com/doi/10.1348/000711005X66068/full

Laenen, A. et al. (2007). A Measure for the Reliability of a Rating Scale Based on Longitudinal Clinical Trial Data. Psychometrika, 72, 443-448. http://www.springerlink.com/content/em0034pw7u55547q/

Molenberghs, G. et al. (2007). Estimating Reliability and Generalizability from Hierarchical Biomedical Data. Journal of Biopharmaceutical Statistics, 17, 595-627. http://www.informaworld.com/smpp/content~db=all~content=a780211328

Laenen, A. et al. (2009). Reliability of a longitudinal sequence of scale ratings. Psychometrika, 74, 49-64. http://www.springerlink.com/content/v237801006831302/

Best,

-- 
Wolfgang Viechtbauer 
Department of Psychiatry and Neuropsychology 
School for Mental Health and Neuroscience 
Maastricht University, P.O. Box 616 
6200 MD Maastricht, The Netherlands 
Tel: +31 (43) 368-5248 
Fax: +31 (43) 368-8689 
Web: http://www.wvbauer.com 

> -----Original Message-----
> From: r-sig-mixed-models-bounces at r-project.org [mailto:r-sig-mixed-models-
> bounces at r-project.org] On Behalf Of Mike Lawrence
> Sent: Thursday, April 07, 2011 01:25
> To: r-sig-mixed-models at r-project.org
> Subject: [R-sig-ME] Reliability via mixed effects modelling
> 
> Hi folks,
> 
> In my research I typically have human participants play simple video
> games and measure the speed and accuracy of their responses to certain
> stimuli. This usually yields many observations per condition of
> interest per participant, so mixed effects modelling (specifying
> participant as a random effect and condition as a fixed effect)
> becomes rather useful.
> 
> I'm wondering, however, if I might gain even more utility from mixed
> effects models by getting them to help me compute the reliability of
> the fixed effects I'm measuring. That is, normally reliability might
> be measured by something like test-retest, where you run your
> participants through one run of the experiment, compute a condition
> effect for each participant, then repeat and see how well the first
> and second estimated condition effects correlate across participants.
> Alternatively, one could employ a "split-half" procedure whereby only
> one session is conducted after which each of the multiple observations
> from each participant in each condition is randomly as "A" or "B"; one
> can then compute condition effects within A trials and B trials
> separately within each participant and finally compute the correlation
> between the condition effects in A and B across participants.
> 
> Finally, I'm fairly certain that if one were to obtain an estimate of
> the expected within-participant variance of the condition effect and
> an estimate of the expected between-participant variance of the
> condition effect, the formula:
> 
> r  = 1/(1+within_variance/between_variance)
> 
> will achieve an estimate of reliability that does not rely on
> correlation. (I believe this latter approach may be somehow
> mathematically related to intra-class correlation, but I have not been
> able to see precisely how)
> 
> With the latter approach in mind, I notice that if I permit a mixed
> effects model to estimate unique condition effects within each
> participant, as in:
> 
> fit = lmer(
>     dv ~ condition + (condition | participant)
> )
> 
> then ranef( fit , postVar=TRUE ) will return information that strikes
> me may be useful for estimating reliability of the effect of
> condition. I've coded a function that I believe computes the variances
> needed for the above non-correlational computation of reliability and
> then bootstraps confidence intervals on these variances and the
> resulting reliability estimate:
> 
> https://gist.github.com/906741
> 
> Does this make sense at all? Or should I go back to computing
> reliability the traditional correlation way?
> 
> 
> Mike
> 
> --
> Mike Lawrence
> Graduate Student
> Department of Psychology
> Dalhousie University
> 
> Looking to arrange a meeting? Check my public calendar:
> http://tr.im/mikes_public_calendar
> 
> ~ Certainty is folly... I think. ~
> 
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models