[R-sig-ME] Reliability via mixed effects modelling

[Mike Lawrence]

Hi folks,

In my research I typically have human participants play simple video
games and measure the speed and accuracy of their responses to certain
stimuli. This usually yields many observations per condition of
interest per participant, so mixed effects modelling (specifying
participant as a random effect and condition as a fixed effect)
becomes rather useful.

I'm wondering, however, if I might gain even more utility from mixed
effects models by getting them to help me compute the reliability of
the fixed effects I'm measuring. That is, normally reliability might
be measured by something like test-retest, where you run your
participants through one run of the experiment, compute a condition
effect for each participant, then repeat and see how well the first
and second estimated condition effects correlate across participants.
Alternatively, one could employ a "split-half" procedure whereby only
one session is conducted after which each of the multiple observations
from each participant in each condition is randomly as "A" or "B"; one
can then compute condition effects within A trials and B trials
separately within each participant and finally compute the correlation
between the condition effects in A and B across participants.

Finally, I'm fairly certain that if one were to obtain an estimate of
the expected within-participant variance of the condition effect and
an estimate of the expected between-participant variance of the
condition effect, the formula:

r  = 1/(1+within_variance/between_variance)

will achieve an estimate of reliability that does not rely on
correlation. (I believe this latter approach may be somehow
mathematically related to intra-class correlation, but I have not been
able to see precisely how)

With the latter approach in mind, I notice that if I permit a mixed
effects model to estimate unique condition effects within each
participant, as in:

fit = lmer(
    dv ~ condition + (condition | participant)
)

then ranef( fit , postVar=TRUE ) will return information that strikes
me may be useful for estimating reliability of the effect of
condition. I've coded a function that I believe computes the variances
needed for the above non-correlational computation of reliability and
then bootstraps confidence intervals on these variances and the
resulting reliability estimate:

https://gist.github.com/906741

Does this make sense at all? Or should I go back to computing
reliability the traditional correlation way?


Mike

--
Mike Lawrence
Graduate Student
Department of Psychology
Dalhousie University

Looking to arrange a meeting? Check my public calendar:
http://tr.im/mikes_public_calendar

~ Certainty is folly... I think. ~