[R-sig-ME] AIC Comparison for MLM with Different Distributions
Ben Bolker
bbo|ker @end|ng |rom gm@||@com
Tue Apr 21 19:08:39 CEST 2020
(Note that this is a public list - not just me!)
AICs are comparable across a pretty wide spectrum of models, with
some caveats.
* they're asymptotic measures (and the AICc correction was derived
for linear models, so may or may not be exactly applicable to other
model types)
* comparing models with parameters on the boundary (e.g. comparing a
zero-inflated vs a non-zero-inflated model) is not exactly correct
* there's a "level of focus" question when comparing models that
differ in their random effects
I believe most of this is discussed in the GLMM FAQ.
On 4/21/20 12:25 PM, Kate R wrote:
> Hi Ben,
>
> Thank you again for your help before!
>
> We will be using other model assessments (including R^2) as well, but
> I wanted to check if the AICs can be compared for (hurdled) gamma
> models with (zero-inflated) beta models when fit in glmmTMB? Likewise,
> can the AICs be compared for(zero-inflated / hurdle) negative binomial
> and (zero-inflated) beta models?For the beta models, we have
> transformed raw counts/durations into percentages.
>
> Many thanks,
> K
>
> On Sat, Mar 7, 2020 at 9:17 AM Ben Bolker <bbolker using gmail.com
> <mailto:bbolker using gmail.com>> wrote:
>
>
> Only when the response variable is transformed.
> [please keep r-sig-mixed-models in the cc: list if possible when
> following up on questions ... ]
>
> cheers
> Ben Bolker
>
> On 2020-03-07 12:16 p.m., Kate R wrote:
> >
> > Hi Ben,
> >
> > Thank you for your reply! Would I also apply the Jacobian
> correction to
> > the Gamma with log-link, or is it only used when the response
> variable
> > is transformed?
> >
> > Many thanks again!
> > Katie
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Wed, 4 Mar 2020 09:55:55 -0500
> > From: Ben Bolker <bbolker using gmail.com
> <mailto:bbolker using gmail.com> <mailto:bbolker using gmail.com
> <mailto:bbolker using gmail.com>>>
> > To: r-sig-mixed-models using r-project.org
> <mailto:r-sig-mixed-models using r-project.org>
> > <mailto:r-sig-mixed-models using r-project.org
> <mailto:r-sig-mixed-models using r-project.org>>
> > Subject: Re: [R-sig-ME] AIC Comparison for MLM with Different
> > Distributions
> > Message-ID: <eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
> <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>
> > <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
> <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>>>
> > Content-Type: text/plain; charset="utf-8"
> >
> >
> > I agree with Thierry's big-picture comment that you should
> generally
> > use broader/qualitative criteria to decide on a model rather
> than
> > testing all possibilities. The only exception I can think
> of is if you
> > are *only* interested in predictive accuracy (not in inference
> > [confidence intervals/p-values etc.]), and you make sure to use
> > cross-validation or a testing set to evaluate out-of-sample
> predictive
> > error (although AIC *should* generally give a reasonable
> approximation
> > to relative out-of-sample error).
> >
> > Beyond that, if you still want to compute AIC (e.g. your
> supervisor or
> > a reviewer is forcing to do it, and you don't think you're
> in a position
> > to push back effectively):
> >
> > * as long as you include the Jacobian correction when you
> transform
> > the predictor variable (i.e. #2), these log-likelihoods (and
> AICs)
> > should in principle be comparable (FWIW the robustness of
> the derivation
> > of AIC is much weaker for non-nested models; Brian Ripley
> [of MASS fame]
> > holds a minority opinion that one should *not* use AICs to
> compare
> > non-nested models)
> >
> > * computing log-likelihoods/AICs by hand is in principle a
> good idea,
> > but is often difficulty for multi-level models, as various
> integrals or
> > approximations of integrals are involved. The lmer and glmer
> > likelihoods (1-4) are definitely comparable. To compare
> across platforms
> > I often try to think of a simplified model that *can* be
> fitted in both
> > platforms (e.g. in this case I think a proportional-odds ordinal
> > regression where the response has only two levels should be
> equivalent
> > to a binomial model with cloglog link ...)
> >
> > cheers
> > Ben Bolker
> >
> > On 2020-03-03 5:29 p.m., Kate R wrote:
> > > Hi all,
> > >
> > > Thank you in advance for your time and consideration! I am a
> > > non-mathematically-inclined graduate student in
> communication just
> > learning
> > > multilevel modeling.
> > >
> > > I am trying to compare the AIC for 5 different models:
> > >
> > >
> > > 1. model.mn5 <- lmer(anxious ~ num.cm <http://num.cm>
> <http://num.cm> + num.pmc
> > + (1|userid), data = df,
> > > REML = F)
> > > 2. model.mn5.log <- lmer(log(anxious) ~ num.cm
> <http://num.cm> <http://num.cm>
> > + num.pmc + (1|userid),
> > > data = df, REML = F)
> > > 3. model.mn5.gamma.log <- glmer(anxious ~ num.cm
> <http://num.cm>
> > <http://num.cm> + num.pmc + (1|userid),
> > > data = df, family = Gamma(link="log"))
> > > 4. model.mn5.gamma.id <http://model.mn5.gamma.id>
> <http://model.mn5.gamma.id> <-
> > glmer(anxious ~ num.cm <http://num.cm> <http://num.cm> +
> num.pmc + (1|userid),
> > > data = df, family = Gamma(link="identity"))
> > > 5. model.ord5 <- clmm(anxious ~ num.cm <http://num.cm>
> <http://num.cm> +
> > num.pmc + (1|userid), data =
> > > df, na.action = na.omit)
> > >
> > > (num.cm <http://num.cm> <http://num.cm> is the group mean
> and num.pmc is the
> > group-mean-centered score of
> > > the predictor)
> > >
> > > Despite many posts on various help forums, I understand
> that it's
> > possible
> > > to compare non-nested models with different distributions
> as long
> > as all
> > > terms, including constants, are retained (i.e. see Burnham &
> > Anderson, Ch
> > > 6.7 <https://www.springer.com/gp/book/9780387953649>), but
> that
> > different R
> > > packages or model classes might handle constants
> differently or use
> > > different algorithms (see point 7
> > <https://robjhyndman.com/hyndsight/aic/>),
> > > thus making it difficult to directly compare AIC values.
> To avoid
> > > this non-comparability pitfall, it was suggested in one
> post to
> > calculate
> > > your own log-likelihood (though I'm having trouble finding
> this
> > post again).
> > >
> >
> >
> >
> >
> > > Please could you help with the following:
> > >
> > > - What is the best practice for comparing the AICs for
> these 5
> > models?
> > > - What is the R-code for manually calculating the
> > log-likelihood and/or
> > > the AIC to retain all terms, including constants?
> > > - Can you compare ordinal models (clmm) with the
> continuous models?
> > > - Do you recommend any other methods and/or packages
> for comparing
> > > models with different distributions and/or links?
> > >
> > > Many thanks in advance for your time and consideration! I
> greatly
> > > appreciate any suggestions.
> > >
> > > Kind regards,
> > > K
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>
> > <mailto:R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org>> mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> > >
> >
> >
> >
> >
>
[[alternative HTML version deleted]]
More information about the R-sig-mixed-models
mailing list