[R] Cohen's Kappa for beginners

[Scot W. McNary]

Hi,

It doesn't seem happy with non-numeric data in the data frame version.  
Maybe a recode would work?

 > x1 <- c(1, 2, 3, 1)
 > y1 <- c(1, 3, 3, 1)
 > cohen.kappa(data.frame(x = x1, y = y1))
Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence 
boundaries
                  lower estimate upper
unweighted kappa 0.098     0.60   1.1
weighted kappa   0.601     0.86   1.1

The unweighted kappa is .60.

I also tried to create the p x p table version, thinking that it would 
be unhappy with less than a p x p table. The example dataset produces a 
3 x 2 table, so this pads it out to a 3 x 3:

 > x <- factor(x)
 > y <- factor(y, levels = c("blue", "red", "yellow"))
 > table(x,y)
         y
x        blue red yellow
   blue      1   0      0
   red       0   2      0
   yellow    1   0      0
 > cohen.kappa(table(x,y))
Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence 
boundaries
                   lower estimate upper
unweighted kappa  0.098      0.6  1.10
weighted kappa   -0.693      0.0  0.69

  Number of subjects = 4>

The unweighted kappa is the same as above and as found by Jim with other 
packages.

Scot

On 5/25/2010 7:31 AM, Jim Lemon wrote:
> On 05/25/2010 06:01 PM, Jason Priem wrote:
>> Hi,
>> I've got two vectors with ratings from two coders, like this:
>>
>> x<-c("red", "yellow", "blue", "red") #coder number 1
>> y<-c("red", "blue", "blue", "red") #coder number 2
>>
>> I want to find Cohen's Kappa using the wkappa function in the psych
>> package. The only example in the docs is using a matrix, which I'm
>> afraid I don't understand--I don't know how to get there from what I've
>> got. Could anyone point me in the right direction? Thanks!
>
> Hi Jason,
> You're not alone. I tried to work out how to run this and it took a 
> while. Both kappa2 in the "irr" package:
>
> kappa2(cbind(x,y))
>
> and classAgreement in the "e1071" package:
>
> classAgreement(table(x,y))
>
> produce a kappa of 0.6. I was unable to work out how to use the wkappa 
> function in the "psych" package:
>
> wkappa(table(x,y))
>
> The above led to a kappa of -0.8. Although wkappa is presented as a 
> test of the reliability of ratings of nominal level data, the example 
> uses numeric data.
>
> Jim
>

-- 
Scot McNary
smcnary at charm dot net