[R] Re: HOWTO compare univariate binomial glm lrm models which are not nested

[Prof Brian Ripley]

Compare them by `goodness for purpose': you have not told us the purpose.
Please do read some of the extensive literature on model comparison.

On Sat, 16 Apr 2005, Jan Verbesselt wrote:

> Thanks a lot for the input!
>
> I forgot to add family=binomial, for a binomial glm. Now the AIC's are
> positive!
>
> I was planning to use AIC (from the binomial glm) and c-index (lrm) to
> compare and rank different uni-variate (one continue explanatory variable)
> logistic models to evaluate the 'performance' of the different explanatory
> variables in the different models.
>
> What is the best technique to compare these lrm.models, which are not
> nested? I found in literature that ranking based on different parameters
> (goodness of fit and predictability) that these can be used to compare
> uni-variate models.
>
> Thanks in advance,
> Regards,
> Jan-
>
>
> _______________________________________________________________________
> ir. Jan Verbesselt
> Research Associate
> Lab of Geomatics Engineering K.U. Leuven
> Vital Decosterstraat 102. B-3000 Leuven Belgium
> Tel: +32-16-329750   Fax: +32-16-329760
> http://gloveg.kuleuven.ac.be/
> _______________________________________________________________________
>
> -----Original Message-----
> From: Prof Brian Ripley [mailto:ripley at stats.ox.ac.uk]
> Sent: Friday, April 15, 2005 5:06 PM
> To: Jan Verbesselt
> Cc: r-help at stat.math.ethz.ch
> Subject: Re: [R] negetative AIC values: How to compare models with negative
> AIC's
>
> AICs (like log-likelihoods) can be positive or negative.
> However, you fitted a Gaussian and not a binomial glm (as lrm does if
> m.arson is binary).
>
> For a discrete response with the usual dominating measure (counting
> measure) the log-likelihood is negative and hence the AIC is positive,
> but not in general (and it is matter of convention even there).
>
> In any case, Akaike only suggested comparing AIC for nested models, no one
> suggests comparing continuous and discrete models.
>
> On Fri, 15 Apr 2005, Jan Verbesselt wrote:
>
>>
>> Dear,
>>
>> When fitting the following model
>> knots <- 5
>> lrm.NDWI <- lrm(m.arson ~ rcs(NDWI,knots)
>>
>> I obtain the following result:
>>
>> Logistic Regression Model
>>
>> lrm(formula = m.arson ~ rcs(NDWI, knots))
>>
>>
>> Frequencies of Responses
>>  0   1
>> 666  35
>>
>>       Obs  Max Deriv Model L.R.       d.f.          P          C
> Dxy
>> Gamma      Tau-a         R2      Brier
>>       701      5e-07      34.49          4          0      0.777
> 0.553
>> 0.563      0.053      0.147      0.045
>>
>>          Coef     S.E.    Wald Z P
>> Intercept   -4.627   3.188 -1.45  0.1467
>> NDWI         5.333  20.724  0.26  0.7969
>> NDWI'        6.832  74.201  0.09  0.9266
>> NDWI''      10.469 183.915  0.06  0.9546
>> NDWI'''   -190.566 254.590 -0.75  0.4541
>>
>> When analysing the glm fit of the same model
>>
>> Call:  glm(formula = m.arson ~ rcs(NDWI, knots), x = T, y = T)
>>
>> Coefficients:
>>            (Intercept)     rcs(NDWI, knots)NDWI    rcs(NDWI, knots)NDWI'
>> rcs(NDWI, knots)NDWI''  rcs(NDWI, knots)NDWI'''
>>                0.02067                  0.08441                 -0.54307
>> 3.99550                -17.38573
>>
>> Degrees of Freedom: 700 Total (i.e. Null);  696 Residual
>> Null Deviance:      33.25
>> Residual Deviance: 31.76        AIC: -167.7
>>
>> A negative AIC occurs!
>>
>> How can the negative AIC from different models be compared with each
> other?
>> Is this result logical? Is the lowest AIC still correct?
>
> -- 
> Brian D. Ripley,                  ripley at stats.ox.ac.uk
> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
> University of Oxford,             Tel:  +44 1865 272861 (self)
> 1 South Parks Road,                     +44 1865 272866 (PA)
> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>
>
>

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595