[R] Cohen's Kappa for beginners

[William Revelle]

Peter, Scot, Jim, and Jason,

Thanks for the various solutions to Jason's problem.  In the next 
release of psych  I will a)  document how to use non-numeric 
categorical variables (e.g., red, blue, etc.)  when using 
cohen.kappa,  b) suggest that cbind does the job (Peter's solution), 
c) make it work with data.frames without the need to do the recode 
that Scot did.

For the record, here is the output you get by using cbind:

>  ck <- cohen.kappa(cbind(x,y))   
>  ck
Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and 
confidence boundaries
                   lower estimate upper
unweighted kappa  0.098      0.6  1.10
weighted kappa   -0.693      0.0  0.69

  Number of subjects = 4

>  ck$agree     #show the table of matches
         x2f
x1f      blue  red yellow
   blue   0.25 0.00   0.00
   red    0.00 0.50   0.00
   yellow 0.25 0.00   0.00
>


Bill



At 7:57 AM -0600 5/25/10, Peter Ehlers wrote:
>cohen.kappa(cbind(x,y)) works for me.
>
>  -Peter Ehlers
>
>On 2010-05-25 7:42, Scot W. McNary wrote:
>>Hi,
>>
>>It doesn't seem happy with non-numeric data in the data frame version.
>>Maybe a recode would work?
>>
>>  > x1 <- c(1, 2, 3, 1)
>>  > y1 <- c(1, 3, 3, 1)
>>  > cohen.kappa(data.frame(x = x1, y = y1))
>>Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)
>>
>>Cohen Kappa and Weighted Kappa correlation coefficients and confidence
>>boundaries
>>lower estimate upper
>>unweighted kappa 0.098 0.60 1.1
>>weighted kappa 0.601 0.86 1.1
>>
>>The unweighted kappa is .60.
>>
>>I also tried to create the p x p table version, thinking that it would
>>be unhappy with less than a p x p table. The example dataset produces a
>>3 x 2 table, so this pads it out to a 3 x 3:
>>
>>  > x <- factor(x)
>>  > y <- factor(y, levels = c("blue", "red", "yellow"))
>>  > table(x,y)
>>y
>>x blue red yellow
>>blue 1 0 0
>>red 0 2 0
>>yellow 1 0 0
>>  > cohen.kappa(table(x,y))
>>Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)
>>
>>Cohen Kappa and Weighted Kappa correlation coefficients and confidence
>>boundaries
>>lower estimate upper
>>unweighted kappa 0.098 0.6 1.10
>>weighted kappa -0.693 0.0 0.69
>>
>>Number of subjects = 4>
>>
>>The unweighted kappa is the same as above and as found by Jim with other
>>packages.
>>
>>Scot
>>
>>On 5/25/2010 7:31 AM, Jim Lemon wrote:
>>>On 05/25/2010 06:01 PM, Jason Priem wrote:
>>>>Hi,
>>>>I've got two vectors with ratings from two coders, like this:
>>>>
>>>>x<-c("red", "yellow", "blue", "red") #coder number 1
>>>>y<-c("red", "blue", "blue", "red") #coder number 2
>>>>
>>>>I want to find Cohen's Kappa using the wkappa function in the psych
>>>>package. The only example in the docs is using a matrix, which I'm
>>>>afraid I don't understand--I don't know how to get there from what I've
>>>>got. Could anyone point me in the right direction? Thanks!
>>>
>>>Hi Jason,
>>>You're not alone. I tried to work out how to run this and it took a
>>>while. Both kappa2 in the "irr" package:
>>>
>>>kappa2(cbind(x,y))
>>>
>>>and classAgreement in the "e1071" package:
>>>
>>>classAgreement(table(x,y))
>>>
>>>produce a kappa of 0.6. I was unable to work out how to use the wkappa
>>>function in the "psych" package:
>>>
>>>wkappa(table(x,y))
>>>
>>>The above led to a kappa of -0.8. Although wkappa is presented as a
>>>test of the reliability of ratings of nominal level data, the example
>>>uses numeric data.
>>>
>>>Jim
>>>
>
>______________________________________________
>R-help at r-project.org mailing list
>https://stat.ethz.ch/mailman/listinfo/r-help
>PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
>and provide commented, minimal, self-contained, reproducible code.