[R] How to compare areas under ROC curves calculated with ROCR package
Bernardo Rangel tura
tura at centroin.com.br
Fri Mar 24 10:23:06 CET 2006
At 11:54 AM 3/23/2006, Frank Samuelson wrote:
>The seROC routine you included is an very good approximation to the
>standard error of the Mann-Whitney-Wilcoxon/Area under the ROC curve
>statistic. It is derived from negative exponential models, but works
>very well in general (e.g. Hanley and McNeil, Diagnostic Radiology,
>1982, v. 143, p. 29).
>A more general estimator of the variance is given by Campbell,
>Douglas and Bailey, Proc. Computers in Cardiology, 1988, p.267)
>I've implemented that in R code included below. It is not an unbiased
>estimator, but it is very close.
>The cROC function is probably not what you want, however.
>It assumes that the data from the two different area measures
>are independent. You said your measures are "from the same dataset."
>Your different AUC measures will be highly correlated.
>There are a number of methods to deal with correlated ROC curves
>If you are interested in performing hypothesis testing on the difference
>in AUC of two parameters, I would suggest a permutation test.
>Permuting the ranks of the data between parameters is
>simple and works well.
Laurent, Jarek, Frank
My routines seROC and cROC was developed based in the paper
A method of comparing the areas under Receiver
Operating Characteristic curves derived from the same cases.
Hanley JA & McNeil BJ, Radiology 1983 148 839-43
The seROC is calculation of standart error of ROC
and cROC is calculation of statistical significance for two AUC in same sample.
I don´t documented this rotine but it assume that
the data from the two different area measures are DEPENDENT.
A dependency of two measure is fix with
incorporation of Pearson correlation among this measures.
I wait that this information helps yours to fix this problem.
Well... If you s think I can help yours just talk.
Bernardo Rangel Tura, MD, MSc
National Institute of Cardiology Laranjeiras
Rio de Janeiro Brazil
More information about the R-help