[R] Goodness of fit of binary logistic model
David Winsemius
dwinsemius at comcast.net
Fri Aug 5 20:40:43 CEST 2011
On Aug 5, 2011, at 2:29 PM, Paul Smith wrote:
> On Fri, Aug 5, 2011 at 7:07 PM, David Winsemius <dwinsemius at comcast.net
> > wrote:
>>>>>>> I have just estimated this model:
>>>>>>> -----------------------------------------------------------
>>>>>>> Logistic Regression Model
>>>>>>>
>>>>>>> lrm(formula = Y ~ X16, x = T, y = T)
>>>>>>>
>>>>>>> Model Likelihood Discrimination Rank
>>>>>>> Discrim.
>>>>>>> Ratio Test Indexes
>>>>>>> Indexes
>>>>>>>
>>>>>>> Obs 82 LR chi2 5.58 R2 0.088 C
>>>>>>> 0.607
>>>>>>> 0 46 d.f. 1 g 0.488 Dxy
>>>>>>> 0.215
>>>>>>> 1 36 Pr(> chi2) 0.0182 gr 1.629 gamma
>>>>>>> 0.589
>>>>>>> max |deriv| 9e-11 gp 0.107
>>>>>>> tau-a
>>>>>>> 0.107
>>>>>>> Brier 0.231
>>>>>>>
>>>>>>> Coef S.E. Wald Z Pr(>|Z|)
>>>>>>> Intercept -1.3218 0.5627 -2.35 0.0188
>>>>>>> X16=1 1.3535 0.6166 2.20 0.0282
>>>>>>> -----------------------------------------------------------
>>>>>>>
>>>>>>> Analyzing the goodness of fit:
>>>>>>>
>>>>>>> -----------------------------------------------------------
>>>>>>>>
>>>>>>>> resid(model.lrm,'gof')
>>>>>>>
>>>>>>> Sum of squared errors Expected value|H0
>>>>>>> SD
>>>>>>> 1.890393e+01 1.890393e+01 6.073415e-16
>>>>>>> Z P
>>>>>>> -8.638125e+04 0.000000e+00
>>>>>>> -----------------------------------------------------------
>>>>>>>
>>>>>>>> From the above calculated p-value (0.000000e+00), one should
>>>>>>>> discard
>>>>>>>
>>>>>>> this model. However, there is something that is puzzling me:
>>>>>>> If the
>>>>>>> 'Expected value|H0' is so coincidental with the 'Sum of squared
>>>>>>> errors', why should one discard the model? I am certainly
>>>>>>> missing
>>>>>>> something.
>>>>>>
>>>>>> It's hard to tell what you are missing, since you have not
>>>>>> described
>>>>>> your
>>>>>> reasoning at all. So I guess what is at error is your
>>>>>> expectation that
>>>>>> we
>>>>>> would have drawn all of the unstated inferences that you draw
>>>>>> when
>>>>>> offered
>>>>>> the output from lrm. (I certainly did not draw the inference
>>>>>> that "one
>>>>>> should discard the model".)
>>>>>>
>>>>>> resid is a function designed for use with glm and lm models.
>>>>>> Why aren't
>>>>>> you
>>>>>> using residuals.lrm?
>>>>>
>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> residuals.lrm(model.lrm,'gof')
>>>>>
>>>>> Sum of squared errors Expected value|H0 SD
>>>>> 1.890393e+01 1.890393e+01 6.073415e-16
>>>>> Z P
>>>>> -8.638125e+04 0.000000e+00
>>>>
>>>> Great. Now please answer the more fundamental question. Why do
>>>> you think
>>>> this mean "discard the model"?
>>>
>>> Before answering that, let me tell you
>>>
>>> resid(model.lrm,'gof')
>>>
>>> calls residuals.lrm() -- so both approaches produce the same
>>> results.
>>> (See the examples given by ?residuals.lrm)
>>>
>>> To answer your question, I invoke the reasoning given by Frank
>>> Harrell at:
>>>
>>>
>>> http://r.789695.n4.nabble.com/Hosmer-Lemeshow-goodness-of-fit-td3508127.html
>>>
>>> He writes:
>>>
>>> «The test in the rms package's residuals.lrm function is the le
>>> Cessie
>>> - van Houwelingen - Copas - Hosmer unweighted sum of squares test
>>> for
>>> global goodness of fit. Like all statistical tests, a large P-value
>>> has no information other than there was not sufficient evidence to
>>> reject the null hypothesis. Here the null hypothesis is that the
>>> true
>>> probabilities are those specified by the model. »
>>>
>>
>> How does that apply to your situation? You have a small (one might
>> even say
>> infinitesimal) p-value.
>>
>>
>>>> From Harrell's argument does not follow that if the p-value is zero
>>>
>>> one should reject the null hypothesis?
>>
>> No, it doesn't follow at all, since that is not what he said. You are
>> committing a common logical error. If A then B does _not_ imply If
>> Not-A
>> then Not-B.
>>
>>> Please, correct if it is not
>>> correct what I say, and please direct me towards a way of
>>> establishing
>>> the goodness of fit of my model.
>>
>> You need to state your research objectives and describe the science
>> in your
>> domain. They you need to describe your data gathering methods and
>> your
>> analytic process. Then there might be a basis for further comment.
>
> I will try to read the original paper where this goodness of fit test
> is proposed to clarify my doubts. In any case, in the paper
>
> @article{barnes2008model,
> title={A model to predict outcomes for endovascular aneurysm repair
> using preoperative variables},
> author={Barnes, M. and Boult, M. and Maddern, G. and Fitridge, R.},
> journal={European Journal of Vascular and Endovascular Surgery},
> volume={35},
> number={5},
> pages={571--579},
> year={2008},
> publisher={Elsevier}
> }
>
> it is written:
>
> «Table 5 lists the results of the global goodness of fit test
> for each outcome model using the le Cessie-van Houwe-
> lingen-Copas-Hosmer unweighted sum of squares test.
> In the table a ‘good’ fit is indicated by large p-values
> ( p > 0.05). Lack of fit is indicated by low p-values
> ( p < 0.05). All p-values indicate that the outcome models
> have reasonable fit, with the exception of the outcome
> model for conversion to open repairs ( p ¼ 0.04). The
> low p-value suggests a lack of fit and it may be worth
> refining the model for conversion to open repair.»
>
> In short, according to these authors, low p-values seem to suggest
> lack of fit.
Sorry for the blank message.
So the topic is outcomes from surgery surgery? The gof approach to
model assessment is just one way of looking at model comparison. The
real question is not "is this the right fit", but should rather be
"have I included as many relevant variables (for which I have data) as
I need to". You included exactly one variable. That would imply that
you had no prior knowledge about predictors of outcomes from surgery.
On the the face of it seems highly implausible. Why are you even
contemplating a gof test in such a situation? Notice that those
authors said the effort should be made to "refine the model", not that
it "should be discarded".
David Winsemius, MD
West Hartford, CT
More information about the R-help
mailing list