[R-sig-ME] Using lmer to determine inter-rater reliability

[Kevin E. Thorpe]

Hello.

I have a data frame with following structure.

 > str(vision)
'data.frame':	268 obs. of  9 variables:
  $ Child          : Factor w/ 67 levels "C01-05","C01-10",..: 43 43 43 
43 44 44 44 44 42 42 ...
  $ Test           : Factor w/ 4 levels "1","2","3","4": 1 2 3 4 1 2 3 4 
1 2 ...
  $ Rater          : Factor w/ 4 levels "F","L","P","S": 4 1 4 1 1 4 1 4 
4 1 ...
  $ Binoc          : int  100 100 100 100 0 0 0 0 40 0 ...
  $ Yield          : int  100 100 100 80 100 20 50 30 100 100 ...
  $ Tries          : int  5 5 5 6 5 10 10 10 5 5 ...
  $ Result         : Factor w/ 5 levels "Pass","ReferBI",..: 1 1 1 1 3 2 
3 2 3 3 ...
  $ ResultCollapsed: Factor w/ 3 levels "Pass","Refer",..: 1 1 1 1 2 2 2 
2 2 2 ...
  $ Test1          : Factor w/ 16 levels "F:1","F:2","F:3",..: 13 2 15 4 
1 14 3 16 13 2 ...

In these data, each subject is rated by 2 (of 4) raters twice. The Test1 
variable was created from Test and Rater with
(Rater:Test)[drop=TRUE] to explicitly create the nesting.

I then fit the following model.

 > binoc.lmer1 <- lmer(Binoc~1+(1|Child) + (1|Rater) + 
(1|Test1),data=vision)
 > binoc.lmer1
Linear mixed model fit by REML ['lmerMod']
Formula: Binoc ~ 1 + (1 | Child) + (1 | Rater) + (1 | Test1)
    Data: vision
REML criterion at convergence: 2592.62
Random effects:
  Groups   Name        Std.Dev.
  Child    (Intercept) 29.226
  Test1    (Intercept)  2.292
  Rater    (Intercept)  5.823
  Residual             26.330
Number of obs: 264, groups:  Child, 66; Test1, 16; Rater, 4
Fixed Effects:
(Intercept)
       51.68


Now my questions.

1. Have I fit the right model?

2. If so, would the right estimate for the rater ICC be
	Rater/(Rater + Residual)
    or
	(Rater + Test1)/(Rater + Test1 + Residual)

3. Would Test1/(Test1 + Residual) give an estimate of intra-rater 
reliability?

Thanks for you time.

Kevin

-- 
Kevin E. Thorpe
Head of Biostatistics,  Applied Health Research Centre (AHRC)
Li Ka Shing Knowledge Institute of St. Michael's
Assistant Professor, Dalla Lana School of Public Health
University of Toronto
email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016