[R] Cohen's kappa, unequal score ranges

[Scot W. McNary]

Julia,

I was able to get your toy data to work, using the n * 2 data frame, 
which is what ckappa takes as input.

 > chk <- data.frame(matrix(c(
+    3,   4,
+    4,   3,
+    2,   1,
+    2,   1,
+    5,   1,
+    2,   4), byrow = TRUE, ncol = 2))  
 > chk
  X1 X2
1  3  4
2  4  3
3  2  1
4  2  1
5  5  1
6  2  4

 > sk <- ckappa(chk)
 > sk
$table
  1 2 3 4 5
1 0 0 0 0 0
2 2 0 0 1 0
3 0 0 0 1 0
4 0 0 1 0 0
5 1 0 0 0 0

$kappa
[1] -0.0909091

 > str(sk)
List of 2
 $ table: num [1:5, 1:5] 0 2 0 0 1 0 0 0 0 0 ...
  ..- attr(*, "dimnames")=List of 2
  .. ..$ : chr [1:5] "1" "2" "3" "4" ...
  .. ..$ : chr [1:5] "1" "2" "3" "4" ...
 $ kappa: num -0.091

Hope this helps,

Scot

Julia Myatt wrote:
> Hi Jim,
>
> Thanks for your help, I tried that function which did result in an output. The only issue now is my summary looks like this:
>
>       Length Class  Mode   
> table 16     -none- numeric
> kappa  1     -none- numeric
>
> Not what I was expecting, this was the case when my data was in the following format:
>
>   ob1 ob2
> 1   3   4
> 2   4   3
> 3   2   1
> 4   2   1
> 5   5   1
> 6   2   4
>
> I then tried entering my data as a matrix like the one below (all this data is made up until I get it to work!):
>
>  p1 p2 p3 p4 p5
> p1  3  1  0  4  2
> p2  3  5  0  7  1
> p3  2  3  4  6  3
> p4  4  1  0  4  0
> p5  8  2  0  2  5
>
> and this resulted in the same type of output:
>
>  Length Class  Mode   
> table 36     -none- numeric
> kappa  1     -none- numeric
>
> If that is the kappa output (1) something isn't right as I purposely made the data without good interrater reliability! I was wondering if you have anymore tips as to what I am doing wrong? I'm afraid I'm very new to R so don't even know the basics!!
>
> Thanks for all your help,
>
> Julia.
>
>
> ________________________________________
> From: Jim Lemon [jim at bitwrit.com.au]
> Sent: 23 December 2009 02:15
> To: Julia Myatt
> Cc: r-help at r-project.org
> Subject: Re: [R] Cohen's kappa, unequal score ranges
>
> On 12/23/2009 01:58 AM, Julia Myatt wrote:
>   
>> Hi,
>>
>> I am having problems getting cohen's kappa to work. I have been using the function:
>>
>>
>>     
>>> <-ckappa(x,y)
>>>
>>>       
>> from the psy package.
>>
>> I am trying to test for inter-observer reliability, I have 2 observers and 26 categories, however, the two observers might not necessarily have the same range of categories (I have unequal score ranges). However, I thought R could cope with this.
>>
>> Each time I try the function above I get the following output:
>>
>>
>>     
>>> Error in ckappa(ob1, ob2) : unused argument(s) (ob2)
>>>
>>>       
>> I got this even when I altered the data to have equal score ranges, therefore I am not sure what this means or what I should do!!
>>
>> Any help in this area would be much appreciated, or anything about the best way to deal with inter-observer reliability (my data is all categorical),
>>
>>
>>     
> Hi Julia,
> The ckappa function only takes one argument, a rater by object matrix or
> dataframe of scores. Try:
>
> ckappa(cbind(ob1,ob2))
>
> Jim
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>   

-- 
Scot McNary
smcnary at charm dot net