[R] mixing nested and crossed factors using lme

[Simon Blomberg]

Hi all,

I have an experiment where 5 raters assessed the quality of 24 web sites. (each rater rated each site once). I want to come up with a measure of reliability of the ratings for the web sites ie to what extent does each rater give the same (or similar) rating to each web site. My idea was to fit a random effects model using lme and from that, calculate the intraclass correlation as a measure of rater reliability. So I analysed it using a two-factor, unreplicated random effects model, with sitenumber and rater as crossed random effects. The call to lme was:

fit <- lme(score ~ 1, random=list(level=pdBlocked(list(pdIdent(~sitenumber-1), pdIdent(~rater-1)))),data=scores)

which makes sense, and I then calculated the intraclass correlation using the within sitenumber (ie among raters) variance, the within rater variance and the residual variance (using VarCorr(fit)): 
rho = var_among/(var_among + var_within + var_resid)

Now, the raters can be divided into two groups. I would like to include group as a fixed factor, to test whether there is a difference in scores between groups (I know the number of raters is too small. This is a pilot study and I am really just trying to get the analysis right in case we want to expand the study). raters therefore are nested within group. My initial guess was:

fit2 <- lme(score ~ group, random=list(level=pdBlocked(list(pdIdent(~sitenumber-1), pdIdent(~rater-1)))),data=scores)

but this fit implies (I think) that sitenumber AND rater are nested within group (sitenumber is not nested within group), and results in the wrong denominator degrees of freedom for the test of group (118). I then tried a slightly different approach:

fit3 <- lme(score ~ group, random=list(level=pdIdent(~sitenumber-1), rater=pdIdent(~group-1)),data=scores)

which gives the correct denominator degrees of freedom for the group effect (3). But it implies that rater is nested within level. Given that level is a dummy factor with only one level, I don't think this matters and I can "pretend" that it is really a crossed design. Is that correct? Or am I way off beam here? Can anyone suggest a more appropriate lme call? The more I think about it the more confused I get.

Thanks in advance,

Simon.

Simon Blomberg, PhD
Depression & Anxiety Consumer Research Unit
Centre for Mental Health Research
Australian National University
http://www.anu.edu.au/cmhr/
Simon.Blomberg at anu.edu.au  +61 (2) 6125 3379