[R] About Mcneil Hanley test for a portion of AUC!

[Frank E Harrell Jr]

Dukka k.c. wrote:
> Dear all
> 
>  I am trying to compare the performances of several methods using the AUC0.1
> and
> not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only
> goes to
> 0.1 not 1)
> 
> I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
> and I referred to the original paper for the calculation of "r" which is an
> argument of the function
> cROC. I can only find the value of "r" for the whole AUC's .
> 
> 
> 
> 
>> seROC<-function(AUC,na,nn){
>> a<-AUC
>> q1<-a/(2-a)
>> q2<-(2*a^2)/(1+a)
>> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
>> se
>> }
>>
>> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
>> se1<-seROC(AUC1,na1,nn1)
>> se2<-seROC(AUC2,na2,nn2)
>>
>> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
>> zad<-(AUC1-AUC2)/sed
>> p<-dnorm(zad)
>> a<-list(zad,p)
>> a
> 
> Could somebody kindly suggest me how to calculate the value of "r" or
> some ways to calculate the statistical significance measure for the
> differences of auc for a part of
> the curve like AUC0.1.
> 
> Thank You

The ROC area is not a sensitive enough measure for comparing two 
competing predictors.  Its power is too low.  See for example the 
following papers.  Note that Pencina et al's approach is now in the 
Hmisc package (function improveProb; documentation to be coming soon). 
Likelihood ratio tests are even more powerful.

@Article{pen08eva,
   author = 		 {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and 
{D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},
   title = 		 {Evaluating the added predictive ability of a new marker: 
{From} area under the {ROC} curve to reclassification and beyond},
   journal = 	 Stat in Med,
   year = 		 2008,
   volume =		 27,
   pages =        {157-172},
   annote =		 {discrimination;model performance;AUC;C-index;risk 
prediction;biomarker;small differences in ROC area can still be very 
meaningful;example of insignificant test for difference in ROC areas 
with very significant results from new method;Yates' discrimination 
slope;reclassification table;limiting version of this based on whether 
and amount by which probabilities rise for events and lower for 
non-events when compare new model to old;comparing two models}
}

@Article{coo07use,
   author = 		 {Cook, Nancy R.},
   title = 		 {Use and misues of the receiver operating characteristic 
curve in risk prediction},
   journal = 	 {Circulation},
   year = 		 2007,
   volume =		 115,
   pages =		 {928-935},
   annote =		 {reclassification table;problems with c index;problems 
with ROC area;example of large change in predicted risk in 
cardiovascular disease with tiny change in ROC area;possible limits to c 
index when calibration is perfect;importance of calibration accuracy and 
changes in predicted risk when new variables are added}
}


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University