[R] Different approach to set up Cohen Kappa

[Jim Lemon]

On 08/10/2011 06:38 PM, gavfung wrote:
> Hi, I just started learning R, and one of the most frequent thing that I need
> to calculate is cohen kappa in my psychology lab and I figure being able to
> do inter rater reliability is a great way for me to explore R.  There are
> two different scenario in which I need help with. (By the way, I have
> installed irr and concord.)
>
> 1) In the lab I am working in, we go through transcripts and find certain
> words to code, and we usually compare the codes between two raters. In this
> case, the two rater can agree (both rater coded "apple" with "A"), have a
> mismatch (one rater coded "apple" with "A" but another coded it as "B") or a
> miss (a rater coded "apple" with "A" but the other rater did not code it at
> all). There are several codes with similar procedures and as the codes are
> tallied together, a chart is constructed, similar to the one attached(
> http://r.789695.n4.nabble.com/file/n3732320/example1.xls example1.xls )
> title "example1.xls." The maroon color represents the cells with data, and
> the lavender cells is just the total in each row/column.
>
> My question in this case is How would I calculate the Cohen Kappa for the
> cell shaded in maroon?
>
I would advise not using the cohen.kappa function in the concord 
package, even though it will handle "not coded" responses. The package 
has not been maintained since most of the functions were merged with the 
"irr" package. If the misses were symmetric between the raters, I would 
simply ignore them. It is only when raters differ on misses that you can 
include them without inflating the test statistic. If this example is 
relevant, it might help you:

gf.scores<-matrix(c("A","A","A","A","A","A","A","M","A","M",
  "B","B","B","B","B","B","B","B","M","B","M","B","B","M","B","M",
  "C","C","C","C","C","C","M","C","M","C","M","C","C","M",
  "C","M","C","M"),ncol=2,byrow=TRUE)
gfraters<-rep(1:2,22)
library(e1071)
classAgreement(table(gfraters,gf.scores))

Jim
> 2)I helped run participants over the summer for a psych summer internship
> and after coding them, I will enter the data as shown in the attachment
> title "example2.xls (
> http://r.789695.n4.nabble.com/file/n3732320/example2.xls example2.xls )
> There is also another research assistant that entered the data and I want to
> find a way to check whether we are reliable or not, and want to calculate
> reliability for the following:TimeI, TimeA, TriesI, and TriesA.
>
> Once again, i would need to convert the excel file into csv, but aside from
> that, I am lost as to what I need to do.  Any help is appreciated!  Thank
> you!
>
> --
> View this message in context: http://r.789695.n4.nabble.com/Different-approach-to-set-up-Cohen-Kappa-tp3732320p3732320.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.