[R-sig-ME] AIC Comparison for MLM with Different Distributions

[Kate R]

Hi Ben,

Thank you again for your help before!

We will be using other model assessments (including R^2) as well, but I
wanted to check if the AICs can be compared for (hurdled) gamma models with
(zero-inflated) beta models when fit in glmmTMB? Likewise, can the AICs be
compared for (zero-inflated / hurdle) negative binomial and (zero-inflated)
beta models? For the beta models, we have transformed raw counts/durations
into percentages.

Many thanks,
K

On Sat, Mar 7, 2020 at 9:17 AM Ben Bolker <bbolker using gmail.com> wrote:

>
>   Only when the response variable is transformed.
>   [please keep r-sig-mixed-models in the cc: list if possible when
> following up on questions ... ]
>
>   cheers
>     Ben Bolker
>
> On 2020-03-07 12:16 p.m., Kate R wrote:
> >
> > Hi Ben,
> >
> > Thank you for your reply! Would I also apply the Jacobian correction to
> > the Gamma with log-link, or is it only used when the response variable
> > is transformed?
> >
> > Many thanks again!
> > Katie
> >
> >
> >     ------------------------------
> >
> >     Message: 2
> >     Date: Wed, 4 Mar 2020 09:55:55 -0500
> >     From: Ben Bolker <bbolker using gmail.com <mailto:bbolker using gmail.com>>
> >     To: r-sig-mixed-models using r-project.org
> >     <mailto:r-sig-mixed-models using r-project.org>
> >     Subject: Re: [R-sig-ME] AIC Comparison for MLM with Different
> >             Distributions
> >     Message-ID: <eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
> >     <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>>
> >     Content-Type: text/plain; charset="utf-8"
> >
> >
> >       I agree with Thierry's big-picture comment that you should
> generally
> >     use broader/qualitative criteria to decide on a model rather than
> >     testing all possibilities.  The only exception I can think of is if
> you
> >     are *only* interested in predictive accuracy (not in inference
> >     [confidence intervals/p-values etc.]), and you make sure to use
> >     cross-validation or a testing set to evaluate out-of-sample
> predictive
> >     error (although AIC *should* generally give a reasonable
> approximation
> >     to relative out-of-sample error).
> >
> >       Beyond that, if you still want to compute AIC (e.g. your
> supervisor or
> >     a reviewer is forcing to do it, and you don't think you're in a
> position
> >     to push back effectively):
> >
> >       * as long as you include the Jacobian correction when you transform
> >     the predictor variable (i.e. #2), these log-likelihoods (and AICs)
> >     should in principle be comparable (FWIW the robustness of the
> derivation
> >     of AIC is much weaker for non-nested models; Brian Ripley [of MASS
> fame]
> >     holds a minority opinion that one should *not* use AICs to compare
> >     non-nested models)
> >
> >       * computing log-likelihoods/AICs by hand is in principle a good
> idea,
> >     but is often difficulty for multi-level models, as various integrals
> or
> >     approximations of integrals are involved.  The lmer and glmer
> >     likelihoods (1-4) are definitely comparable. To compare across
> platforms
> >     I often try to think of a simplified model that *can* be fitted in
> both
> >     platforms (e.g. in this case I think a proportional-odds ordinal
> >     regression where the response has only two levels should be
> equivalent
> >     to a binomial model with cloglog link ...)
> >
> >       cheers
> >       Ben Bolker
> >
> >     On 2020-03-03 5:29 p.m., Kate R wrote:
> >     > Hi all,
> >     >
> >     > Thank you in advance for your time and consideration! I am a
> >     > non-mathematically-inclined graduate student in communication just
> >     learning
> >     > multilevel modeling.
> >     >
> >     > I am trying to compare the AIC for 5 different models:
> >     >
> >     >
> >     >    1. model.mn5 <- lmer(anxious ~ num.cm <http://num.cm> + num.pmc
> >     + (1|userid), data = df,
> >     >    REML = F)
> >     >    2. model.mn5.log <- lmer(log(anxious) ~ num.cm <http://num.cm>
> >     + num.pmc + (1|userid),
> >     >    data = df, REML = F)
> >     >    3. model.mn5.gamma.log <- glmer(anxious ~ num.cm
> >     <http://num.cm> + num.pmc + (1|userid),
> >     >    data = df, family = Gamma(link="log"))
> >     >    4. model.mn5.gamma.id <http://model.mn5.gamma.id> <-
> >     glmer(anxious ~ num.cm <http://num.cm> + num.pmc + (1|userid),
> >     >    data = df, family = Gamma(link="identity"))
> >     >    5. model.ord5 <- clmm(anxious ~ num.cm <http://num.cm> +
> >     num.pmc + (1|userid), data =
> >     >    df, na.action = na.omit)
> >     >
> >     > (num.cm <http://num.cm> is the group mean and num.pmc is the
> >     group-mean-centered score of
> >     > the predictor)
> >     >
> >     > Despite many posts on various help forums, I understand that it's
> >     possible
> >     > to compare non-nested models with different distributions as long
> >     as all
> >     > terms, including constants, are retained (i.e. see Burnham &
> >     Anderson, Ch
> >     > 6.7 <https://www.springer.com/gp/book/9780387953649>), but that
> >     different R
> >     > packages or model classes might handle constants differently or use
> >     > different algorithms (see point 7
> >     <https://robjhyndman.com/hyndsight/aic/>),
> >     > thus making it difficult to directly compare AIC values. To avoid
> >     > this non-comparability pitfall, it was suggested in one post to
> >     calculate
> >     > your own log-likelihood (though I'm having trouble finding this
> >     post again).
> >     >
> >
> >
> >
> >
> >     > Please could you help with the following:
> >     >
> >     >    - What is the best practice for comparing the AICs for these 5
> >     models?
> >     >    - What is the R-code for manually calculating the
> >     log-likelihood and/or
> >     >    the AIC to retain all terms, including constants?
> >     >    - Can you compare ordinal models (clmm) with the continuous
> models?
> >     >    - Do you recommend any other methods and/or packages for
> comparing
> >     >    models with different distributions and/or links?
> >     >
> >     > Many thanks in advance for your time and consideration! I greatly
> >     > appreciate any suggestions.
> >     >
> >     > Kind regards,
> >     > K
> >     >
> >     >       [[alternative HTML version deleted]]
> >     >
> >     > _______________________________________________
> >     > R-sig-mixed-models using r-project.org
> >     <mailto:R-sig-mixed-models using r-project.org> mailing list
> >     > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >     >
> >
> >
> >
> >
>

	[[alternative HTML version deleted]]