[R-sig-ME] Using lmer to determine inter-rater reliability

[Kevin E. Thorpe]

Hello again.

I posted this question a couple of weeks ago and since I have received 
no suggestions, I assume it's because I have not given enough information.

If more information is needed, please let me know what you need and I'll 
do my best to provide it.

Thanks,

Kevin

On 08/31/2015 11:53 AM, Kevin E. Thorpe wrote:
> Hello.
>
> I have a data frame with following structure.
>
>  > str(vision)
> 'data.frame':    268 obs. of  9 variables:
>   $ Child          : Factor w/ 67 levels "C01-05","C01-10",..: 43 43 43
> 43 44 44 44 44 42 42 ...
>   $ Test           : Factor w/ 4 levels "1","2","3","4": 1 2 3 4 1 2 3 4
> 1 2 ...
>   $ Rater          : Factor w/ 4 levels "F","L","P","S": 4 1 4 1 1 4 1 4
> 4 1 ...
>   $ Binoc          : int  100 100 100 100 0 0 0 0 40 0 ...
>   $ Yield          : int  100 100 100 80 100 20 50 30 100 100 ...
>   $ Tries          : int  5 5 5 6 5 10 10 10 5 5 ...
>   $ Result         : Factor w/ 5 levels "Pass","ReferBI",..: 1 1 1 1 3 2
> 3 2 3 3 ...
>   $ ResultCollapsed: Factor w/ 3 levels "Pass","Refer",..: 1 1 1 1 2 2 2
> 2 2 2 ...
>   $ Test1          : Factor w/ 16 levels "F:1","F:2","F:3",..: 13 2 15 4
> 1 14 3 16 13 2 ...
>
> In these data, each subject is rated by 2 (of 4) raters twice. The Test1
> variable was created from Test and Rater with
> (Rater:Test)[drop=TRUE] to explicitly create the nesting.
>
> I then fit the following model.
>
>  > binoc.lmer1 <- lmer(Binoc~1+(1|Child) + (1|Rater) +
> (1|Test1),data=vision)
>  > binoc.lmer1
> Linear mixed model fit by REML ['lmerMod']
> Formula: Binoc ~ 1 + (1 | Child) + (1 | Rater) + (1 | Test1)
>     Data: vision
> REML criterion at convergence: 2592.62
> Random effects:
>   Groups   Name        Std.Dev.
>   Child    (Intercept) 29.226
>   Test1    (Intercept)  2.292
>   Rater    (Intercept)  5.823
>   Residual             26.330
> Number of obs: 264, groups:  Child, 66; Test1, 16; Rater, 4
> Fixed Effects:
> (Intercept)
>        51.68
>
>
> Now my questions.
>
> 1. Have I fit the right model?
>
> 2. If so, would the right estimate for the rater ICC be
>      Rater/(Rater + Residual)
>     or
>      (Rater + Test1)/(Rater + Test1 + Residual)
>
> 3. Would Test1/(Test1 + Residual) give an estimate of intra-rater
> reliability?
>
> Thanks for you time.
>
> Kevin
>


-- 
Kevin E. Thorpe
Head of Biostatistics,  Applied Health Research Centre (AHRC)
Li Ka Shing Knowledge Institute of St. Michael's
Assistant Professor, Dalla Lana School of Public Health
University of Toronto
email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016