[R] Fw: Logistic regresion - Interpreting (SENS) and (SPEC)

[Frank E Harrell Jr]

Gad Abraham wrote:
>> This approach leaves much to be desired.  I hope that its 
>> practitioners start gauging it by the mean squared error of predicted 
>> probabilities.
> 
> Is the logic here is that low MSE of predicted probabilities equals a 
> better calibrated model? What about discrimination? Perfect calibration 

Almost.  I was addressed more the wish for the use of strategies that 
maximize precision while keeping bias to a minimim.

> implies perfect discrimination, but I often find that you can have two 

That doesn't follow.  You can have perfect calibration in the large with 
no discrimination.

> competing models, the first with higher discrimination (AUC) and worse 
> calibration, and the the second the other way round. Which one is the 
> better model?

I judge models on the basis of both discrimination (best measured with 
log likelihood measures, 2nd best AUC) and calibration.  It's a 
two-dimensional issue and we don't always know how to weigh the two. 
For many purposes calibration is a must.  In those we don't look at 
discrimination until calibration-in-the-small is verified at high 
resolution.

Frank

> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University