[R] interpreting bootstrap corrected slope [rms package]

[Frank Harrell]

Adam - the very low amount of optimism suggests that you have a large sample
size and that your model was completely pre-specified.  If you did any
feature/variable selection or made any model changes in a way that was not
blinded to Y then you are not using the software correctly.  But you are
right the slope decrement indicates a bit of overfitting on an absolute
calibration scale.  The harm done by this can be partially interpreted by
the Emax value of 0.05 indicated the maximum absolute calibration error is
estimated to be 0.05 on the probability scale.  If your exceedence
probabilities for the middle Y category have a wide range then 0.05 isn't so
bad.

Frank


apeer wrote:
> 
> Dear List:
> 
> Below is the validation output of a fitted ordinal logistic model
> using the bootstrap in the rms package.  My interpretation is that
> most of the corrected indices indicate little overfitting, however the
> slope seems to indicate that the model is too optimistic.  Given that
> most of the corrected indices seem reasonable, would it be appropriate
> to use this model on future data if the corrected intercept and slope
> estimates are used?
> 
>           index.orig training    test optimism index.corrected   n
> Dxy          0.9932   0.9940  0.9905   0.0035          0.9897 363
> R2            0.9291   0.9364  0.9163   0.0202          0.9089 363
> Intercept   0.0000   0.0000  0.0233  -0.0233          0.0233 363
> Slope       1.0000   1.0000  0.7836   0.2164          0.7836 363
> Emax       0.0000   0.0000  0.0582   0.0582          0.0582 363
> D              0.9118   0.9190  0.8915   0.0275          0.8844 363
> U             -0.0110  -0.0110  0.0124  -0.0234          0.0124 363
> Q              0.9228   0.9299  0.8791   0.0508          0.8720 363
> B              0.0205   0.0172  0.0239  -0.0067          0.0272 363
> 
> 
> Any input is much appreciated.
> 
> Thanks,
> Adam
> 
> ______________________________________________
> R-help@ mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 


-----
Frank Harrell
Department of Biostatistics, Vanderbilt University
--
View this message in context: http://r.789695.n4.nabble.com/interpreting-bootstrap-corrected-slope-rms-package-tp3928314p3928467.html
Sent from the R help mailing list archive at Nabble.com.