[R] Re: HOWTO compare univariate binomial glm lrm models which are not nested

[Frank E Harrell Jr]

Prof Brian Ripley wrote:
> Compare them by `goodness for purpose': you have not told us the purpose.
> Please do read some of the extensive literature on model comparison.
> 
> On Sat, 16 Apr 2005, Jan Verbesselt wrote:
> 
>> Thanks a lot for the input!
>>
>> I forgot to add family=binomial, for a binomial glm. Now the AIC's are
>> positive!
>>
>> I was planning to use AIC (from the binomial glm) and c-index (lrm) to
>> compare and rank different uni-variate (one continue explanatory 
>> variable)
>> logistic models to evaluate the 'performance' of the different 
>> explanatory
>> variables in the different models.
>>
>> What is the best technique to compare these lrm.models, which are not
>> nested? I found in literature that ranking based on different parameters
>> (goodness of fit and predictability) that these can be used to compare
>> uni-variate models.
>>
>> Thanks in advance,
>> Regards,
>> Jan-
>>

In addition to Brian's comment, AIC may be of use.  You can't really use 
c-index (ROC area) as it is not sensitive enough for comparing two 
models.  But whatever you use, the bad news is that you can't use the 
results to compare more than 2 or 3 completely pre-chosen models or you 
will invalidate inference and estimates if you use these comparisons to 
build a final model.

Frank

>>
>> _______________________________________________________________________
>> ir. Jan Verbesselt
>> Research Associate
>> Lab of Geomatics Engineering K.U. Leuven
>> Vital Decosterstraat 102. B-3000 Leuven Belgium
>> Tel: +32-16-329750   Fax: +32-16-329760
>> http://gloveg.kuleuven.ac.be/
>> _______________________________________________________________________
>>
>> -----Original Message-----
>> From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk]
>> Sent: Friday, April 15, 2005 5:06 PM
>> To: Jan Verbesselt
>> Cc: r-help at stat.math.ethz.ch
>> Subject: Re: [R] negetative AIC values: How to compare models with 
>> negative
>> AIC's
>>
>> AICs (like log-likelihoods) can be positive or negative.
>> However, you fitted a Gaussian and not a binomial glm (as lrm does if
>> m.arson is binary).
>>
>> For a discrete response with the usual dominating measure (counting
>> measure) the log-likelihood is negative and hence the AIC is positive,
>> but not in general (and it is matter of convention even there).
>>
>> In any case, Akaike only suggested comparing AIC for nested models, no 
>> one
>> suggests comparing continuous and discrete models.
>>
>> On Fri, 15 Apr 2005, Jan Verbesselt wrote:
>>
>>>
>>> Dear,
>>>
>>> When fitting the following model
>>> knots <- 5
>>> lrm.NDWI <- lrm(m.arson ~ rcs(NDWI,knots)
>>>
>>> I obtain the following result:
>>>
>>> Logistic Regression Model
>>>
>>> lrm(formula = m.arson ~ rcs(NDWI, knots))
>>>
>>>
>>> Frequencies of Responses
>>>  0   1
>>> 666  35
>>>
>>>       Obs  Max Deriv Model L.R.       d.f.          P          C
>>
>> Dxy
>>
>>> Gamma      Tau-a         R2      Brier
>>>       701      5e-07      34.49          4          0      0.777
>>
>> 0.553
>>
>>> 0.563      0.053      0.147      0.045
>>>
>>>          Coef     S.E.    Wald Z P
>>> Intercept   -4.627   3.188 -1.45  0.1467
>>> NDWI         5.333  20.724  0.26  0.7969
>>> NDWI'        6.832  74.201  0.09  0.9266
>>> NDWI''      10.469 183.915  0.06  0.9546
>>> NDWI'''   -190.566 254.590 -0.75  0.4541
>>>
>>> When analysing the glm fit of the same model
>>>
>>> Call:  glm(formula = m.arson ~ rcs(NDWI, knots), x = T, y = T)
>>>
>>> Coefficients:
>>>            (Intercept)     rcs(NDWI, knots)NDWI    rcs(NDWI, knots)NDWI'
>>> rcs(NDWI, knots)NDWI''  rcs(NDWI, knots)NDWI'''
>>>                0.02067                  0.08441                 -0.54307
>>> 3.99550                -17.38573
>>>
>>> Degrees of Freedom: 700 Total (i.e. Null);  696 Residual
>>> Null Deviance:      33.25
>>> Residual Deviance: 31.76        AIC: -167.7
>>>
>>> A negative AIC occurs!
>>>
>>> How can the negative AIC from different models be compared with each
>>
>> other?
>>
>>> Is this result logical? Is the lowest AIC still correct?
>>
>>
>> -- 
>> Brian D. Ripley,                  ripley at stats.ox.ac.uk
>> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>> University of Oxford,             Tel:  +44 1865 272861 (self)
>> 1 South Parks Road,                     +44 1865 272866 (PA)
>> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>>
>>
>>
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University