[R] ROC curve in R

[Dylan Beaudette]

On Thursday 26 July 2007 06:01, Frank E Harrell Jr wrote:
> Note that even though the ROC curve as a whole is an interesting
> 'statistic' (its area is a linear translation of the
> Wilcoxon-Mann-Whitney-Somers-Goodman-Kruskal rank correlation
> statistics), each individual point on it is an improper scoring rule,
> i.e., a rule that is optimized by fitting an inappropriate model.  Using
> curves to select cutoffs is a low-precision and arbitrary operation, and
> the cutoffs do not replicate from study to study.  Probably the worst
> problem with drawing an ROC curve is that it tempts analysts to try to
> find cutoffs where none really exist, and it makes analysts ignore the
> whole field of decision theory.
>
> Frank Harrell

Frank,

This thread has caught may attention for a couple reasons, possibly related to 
my novice-level experience. 

1. in a logistic regression study, where i am predicting the probability of 
the response being 1 (for example) - there exists a continuum of probability 
values - and a finite number of {1,0} realities when i either look within the 
original data set, or with a new 'verification' data set. I understand that 
drawing a line through the probabilities returned from the logistic 
regression is a loss of information, but there are times when a 'hard' 
decision requiring prediction of {1,0} is required. I have found that the 
ROCR package (not necessarily the ROC Curve) can be useful in identifying the 
probability cutoff where accuracy is maximized. Is this an unreasonable way 
of using logistic regression as a predictor? 

2. The ROC curve can be a helpful way of communicating false positives / false 
negatives to other users who are less familiar with the output and 
interpretation of logistic regression. 


3. I have been using the area under the ROC Curve, kendall's tau, and cohen's 
kappa to evaluate the accuracy of a logistic regression based prediction, the 
last two statistics based on a some probability cutoff identified before 
hand. 


How does the topic of decision theory relate to some of the circumstances 
described above? Is there a better way to do some of these things?

Cheers,

Dylan



>
> gyadav at ccilindia.co.in wrote:
> > http://search.r-project.org/cgi-bin/namazu.cgi?query=ROC&max=20&result=no
> >rmal&sort=score&idxname=Rhelp02a&idxname=functions&idxname=docs
> >
> > there is a lot of help try help.search("ROC curve") gave
> > Help files with alias or concept or title matching 'ROC curve' using
> > fuzzy matching:
> >
> >
> >
> > granulo(ade4)                             Granulometric Curves
> > plot.roc(analogue)                        Plot ROC curves and associated
> > diagnostics
> > roc(analogue)                             ROC curve analysis
> > colAUC(caTools)                           Column-wise Area Under ROC
> > Curve (AUC)
> > DProc(DPpackage)                          Semiparametric Bayesian ROC
> > curve analysis
> > cv.enet(elasticnet)                       Computes K-fold cross-validated
> > error curve for elastic net
> > ROC(Epi)                                  Function to compute and draw
> > ROC-curves.
> > lroc(epicalc)                             ROC curve
> > cv.lars(lars)                             Computes K-fold cross-validated
> > error curve for lars
> > roc.demo(TeachingDemos)                   Demonstrate ROC curves by
> > interactively building one
> >
> > HTH
> > see the help and examples those will suffice
> >
> > Type 'help(FOO, package = PKG)' to inspect entry 'FOO(PKG) TITLE'.
> >
> >
> >
> > Regards,
> >
> > Gaurav Yadav
> > +++++++++++
> > Assistant Manager, CCIL, Mumbai (India)
> > Mob: +919821286118 Email: emailtogauravyadav at gmail.com
> > Bhagavad Gita:  Man is made by his Belief, as He believes, so He is
> >
> >
> >
> > "Rithesh M. Mohan" <rithesh.m at brickworkindia.com>
> > Sent by: r-help-bounces at stat.math.ethz.ch
> > 07/26/2007 11:26 AM
> >
> > To
> > <R-help at stat.math.ethz.ch>
> > cc
> >
> > Subject
> > [R] ROC curve in R
> >
> >
> >
> >
> >
> >
> > Hi,
> >
> >
> >
> > I need to build ROC curve in R, can you please provide data steps / code
> > or guide me through it.
> >
> >
> >
> > Thanks and Regards
> >
> > Rithesh M Mohan
> >
> >
> >                  [[alternative HTML version deleted]]
>
> -
> Frank E Harrell Jr   Professor and Chair           School of Medicine
>                       Department of Biostatistics   Vanderbilt University
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html and provide commented, minimal,
> self-contained, reproducible code.