[R-sig-ME] AIC Comparison for MLM with Different Distributions

Ben Bolker bbo|ker @end|ng |rom gm@||@com
Tue Apr 21 19:08:39 CEST 2020


   (Note that this is a public list - not just me!)

   AICs are comparable across a pretty wide spectrum of models, with 
some caveats.

    * they're asymptotic measures (and the AICc correction was derived 
for linear models, so may or may not be exactly applicable to other 
model types)

    * comparing models with parameters on the boundary (e.g. comparing a 
zero-inflated vs a non-zero-inflated model) is not exactly correct

    * there's a "level of focus" question when comparing models that 
differ in their random effects

    I believe most of this is discussed in the GLMM FAQ.

On 4/21/20 12:25 PM, Kate R wrote:
> Hi Ben,
>
> Thank you again for your help before!
>
> We will be using other model assessments (including R^2) as well, but 
> I wanted to check if the AICs can be compared for (hurdled) gamma 
> models with (zero-inflated) beta models when fit in glmmTMB? Likewise, 
> can the AICs be compared for(zero-inflated / hurdle) negative binomial 
> and (zero-inflated) beta models?For the beta models, we have 
> transformed raw counts/durations into percentages.
>
> Many thanks,
> K
>
> On Sat, Mar 7, 2020 at 9:17 AM Ben Bolker <bbolker using gmail.com 
> <mailto:bbolker using gmail.com>> wrote:
>
>
>       Only when the response variable is transformed.
>       [please keep r-sig-mixed-models in the cc: list if possible when
>     following up on questions ... ]
>
>       cheers
>         Ben Bolker
>
>     On 2020-03-07 12:16 p.m., Kate R wrote:
>     >
>     > Hi Ben,
>     >
>     > Thank you for your reply! Would I also apply the Jacobian
>     correction to
>     > the Gamma with log-link, or is it only used when the response
>     variable
>     > is transformed?
>     >
>     > Many thanks again!
>     > Katie
>     >
>     >
>     >     ------------------------------
>     >
>     >     Message: 2
>     >     Date: Wed, 4 Mar 2020 09:55:55 -0500
>     >     From: Ben Bolker <bbolker using gmail.com
>     <mailto:bbolker using gmail.com> <mailto:bbolker using gmail.com
>     <mailto:bbolker using gmail.com>>>
>     >     To: r-sig-mixed-models using r-project.org
>     <mailto:r-sig-mixed-models using r-project.org>
>     >     <mailto:r-sig-mixed-models using r-project.org
>     <mailto:r-sig-mixed-models using r-project.org>>
>     >     Subject: Re: [R-sig-ME] AIC Comparison for MLM with Different
>     >             Distributions
>     >     Message-ID: <eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
>     <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>
>     >     <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
>     <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>>>
>     >     Content-Type: text/plain; charset="utf-8"
>     >
>     >
>     >       I agree with Thierry's big-picture comment that you should
>     generally
>     >     use broader/qualitative criteria to decide on a model rather
>     than
>     >     testing all possibilities.  The only exception I can think
>     of is if you
>     >     are *only* interested in predictive accuracy (not in inference
>     >     [confidence intervals/p-values etc.]), and you make sure to use
>     >     cross-validation or a testing set to evaluate out-of-sample
>     predictive
>     >     error (although AIC *should* generally give a reasonable
>     approximation
>     >     to relative out-of-sample error).
>     >
>     >       Beyond that, if you still want to compute AIC (e.g. your
>     supervisor or
>     >     a reviewer is forcing to do it, and you don't think you're
>     in a position
>     >     to push back effectively):
>     >
>     >       * as long as you include the Jacobian correction when you
>     transform
>     >     the predictor variable (i.e. #2), these log-likelihoods (and
>     AICs)
>     >     should in principle be comparable (FWIW the robustness of
>     the derivation
>     >     of AIC is much weaker for non-nested models; Brian Ripley
>     [of MASS fame]
>     >     holds a minority opinion that one should *not* use AICs to
>     compare
>     >     non-nested models)
>     >
>     >       * computing log-likelihoods/AICs by hand is in principle a
>     good idea,
>     >     but is often difficulty for multi-level models, as various
>     integrals or
>     >     approximations of integrals are involved.  The lmer and glmer
>     >     likelihoods (1-4) are definitely comparable. To compare
>     across platforms
>     >     I often try to think of a simplified model that *can* be
>     fitted in both
>     >     platforms (e.g. in this case I think a proportional-odds ordinal
>     >     regression where the response has only two levels should be
>     equivalent
>     >     to a binomial model with cloglog link ...)
>     >
>     >       cheers
>     >       Ben Bolker
>     >
>     >     On 2020-03-03 5:29 p.m., Kate R wrote:
>     >     > Hi all,
>     >     >
>     >     > Thank you in advance for your time and consideration! I am a
>     >     > non-mathematically-inclined graduate student in
>     communication just
>     >     learning
>     >     > multilevel modeling.
>     >     >
>     >     > I am trying to compare the AIC for 5 different models:
>     >     >
>     >     >
>     >     >    1. model.mn5 <- lmer(anxious ~ num.cm <http://num.cm>
>     <http://num.cm> + num.pmc
>     >     + (1|userid), data = df,
>     >     >    REML = F)
>     >     >    2. model.mn5.log <- lmer(log(anxious) ~ num.cm
>     <http://num.cm> <http://num.cm>
>     >     + num.pmc + (1|userid),
>     >     >    data = df, REML = F)
>     >     >    3. model.mn5.gamma.log <- glmer(anxious ~ num.cm
>     <http://num.cm>
>     >     <http://num.cm> + num.pmc + (1|userid),
>     >     >    data = df, family = Gamma(link="log"))
>     >     >    4. model.mn5.gamma.id <http://model.mn5.gamma.id>
>     <http://model.mn5.gamma.id> <-
>     >     glmer(anxious ~ num.cm <http://num.cm> <http://num.cm> +
>     num.pmc + (1|userid),
>     >     >    data = df, family = Gamma(link="identity"))
>     >     >    5. model.ord5 <- clmm(anxious ~ num.cm <http://num.cm>
>     <http://num.cm> +
>     >     num.pmc + (1|userid), data =
>     >     >    df, na.action = na.omit)
>     >     >
>     >     > (num.cm <http://num.cm> <http://num.cm> is the group mean
>     and num.pmc is the
>     >     group-mean-centered score of
>     >     > the predictor)
>     >     >
>     >     > Despite many posts on various help forums, I understand
>     that it's
>     >     possible
>     >     > to compare non-nested models with different distributions
>     as long
>     >     as all
>     >     > terms, including constants, are retained (i.e. see Burnham &
>     >     Anderson, Ch
>     >     > 6.7 <https://www.springer.com/gp/book/9780387953649>), but
>     that
>     >     different R
>     >     > packages or model classes might handle constants
>     differently or use
>     >     > different algorithms (see point 7
>     >     <https://robjhyndman.com/hyndsight/aic/>),
>     >     > thus making it difficult to directly compare AIC values.
>     To avoid
>     >     > this non-comparability pitfall, it was suggested in one
>     post to
>     >     calculate
>     >     > your own log-likelihood (though I'm having trouble finding
>     this
>     >     post again).
>     >     >
>     >
>     >
>     >
>     >
>     >     > Please could you help with the following:
>     >     >
>     >     >    - What is the best practice for comparing the AICs for
>     these 5
>     >     models?
>     >     >    - What is the R-code for manually calculating the
>     >     log-likelihood and/or
>     >     >    the AIC to retain all terms, including constants?
>     >     >    - Can you compare ordinal models (clmm) with the
>     continuous models?
>     >     >    - Do you recommend any other methods and/or packages
>     for comparing
>     >     >    models with different distributions and/or links?
>     >     >
>     >     > Many thanks in advance for your time and consideration! I
>     greatly
>     >     > appreciate any suggestions.
>     >     >
>     >     > Kind regards,
>     >     > K
>     >     >
>     >     >       [[alternative HTML version deleted]]
>     >     >
>     >     > _______________________________________________
>     >     > R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>
>     >     <mailto:R-sig-mixed-models using r-project.org
>     <mailto:R-sig-mixed-models using r-project.org>> mailing list
>     >     > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>     >     >
>     >
>     >
>     >
>     >
>

	[[alternative HTML version deleted]]



More information about the R-sig-mixed-models mailing list