[R] How to validate model?

[Frank E Harrell Jr]

Pedro.Rodriguez at sungard.com wrote:
> Usually one validates scorecards with the ROC curve, Pietra Index, KS
> test, etc. You may be interested in the WP 14 from BIS (www.bis.org).
> 
> Regards,
> 
> Pedro

No, the validation should be done using an absolute reliability 
(calibration) curve.  You need to verify that at all levels of predicted 
risk there is agreement with the true probability of failure.  An ROC 
curve does not do that, and I doubt the others do.  A 
resampling-corrected loess calibration curve is a good approach as 
implemented in the Design package's calibrate function.

Frank

> 
> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
> On Behalf Of Maithili Shiva
> Sent: Tuesday, October 07, 2008 8:22 AM
> To: r-help at r-project.org
> Subject: [R] How to validate model?
> 
> Hi!
> 
> I am working on scorecard model and I have arrived at the regression
> equation. I have used logistic regression using R.
> 
> My question is how do I validate this model? I do have hold out sample
> of 5000 customers.
> 
> Please guide me. Problem is I had never used Logistic regression earlier
> neither I am used to credit scoring models.
> 
> Thanks in advance
> 
> Maithili
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University