[R] Measures of agreement

[Andrew Perrin]

Greetings.

I've been experimenting with some algorithms for document classification
(specifically, a Naive Bayes classifier and a kNN classifier) and I would
now like to calculate some inter-rater reliability scores. I have the data
in a PostgreSQL database, such that for each document, each measure (there
are 9) has three variables: ap_(measure), nb_(measure), and
knn_(measure). ap is me (Andrew Perrin), nb is Naive Bayes, and knn is
knn.

I have two questions:
1.) I have used the code in the Using R for Psychology... paper to
calculate Cohen's Kappa (kappaFor2). It returns a (fairly low) kappa, but
also some warnings I don't understand:
> kappaFor2(ap.nb.df$ap.sub,ap.nb.df$nb.sub)
     kappa       S.E.     z.stat    p.value 
0.09411765 0.33707660 0.27921738 0.78007800 
Warning messages: 
1: longer object length
        is not a multiple of shorter object length in: tm1 * tm2 
2: longer object length
        is not a multiple of shorter object length in: tm1 * tm2 
3: longer object length
        is not a multiple of shorter object length in: tm1 + tm2 


2.) I'd be interested in other measures of reliability, specifically ones
from the NLP literature such as precision, recall, and F1. These seem more
interesting for my uses, if for no other reason than what I'm really
interested in is comparing the success of nb and knn at approaching the ap
categories. Are there any packages that provide such measures?

Many thanks.

----------------------------------------------------------------------
Andrew J Perrin - andrew_perrin at unc.edu - http://www.unc.edu/~aperrin
 Assistant Professor of Sociology, U of North Carolina, Chapel Hill
      269 Hamilton Hall, CB#3210, Chapel Hill, NC 27599-3210 USA


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._