[R-sig-ME] AIC Comparison for MLM with Different Distributions
Ben Bolker
bbo|ker @end|ng |rom gm@||@com
Sat Mar 7 18:17:48 CET 2020
Only when the response variable is transformed.
[please keep r-sig-mixed-models in the cc: list if possible when
following up on questions ... ]
cheers
Ben Bolker
On 2020-03-07 12:16 p.m., Kate R wrote:
>
> Hi Ben,
>
> Thank you for your reply! Would I also apply the Jacobian correction to
> the Gamma with log-link, or is it only used when the response variable
> is transformed?
>
> Many thanks again!
> Katie
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 4 Mar 2020 09:55:55 -0500
> From: Ben Bolker <bbolker using gmail.com <mailto:bbolker using gmail.com>>
> To: r-sig-mixed-models using r-project.org
> <mailto:r-sig-mixed-models using r-project.org>
> Subject: Re: [R-sig-ME] AIC Comparison for MLM with Different
> Distributions
> Message-ID: <eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com
> <mailto:eae1791a-e56a-7c32-b872-f6fa93157857 using gmail.com>>
> Content-Type: text/plain; charset="utf-8"
>
>
> I agree with Thierry's big-picture comment that you should generally
> use broader/qualitative criteria to decide on a model rather than
> testing all possibilities. The only exception I can think of is if you
> are *only* interested in predictive accuracy (not in inference
> [confidence intervals/p-values etc.]), and you make sure to use
> cross-validation or a testing set to evaluate out-of-sample predictive
> error (although AIC *should* generally give a reasonable approximation
> to relative out-of-sample error).
>
> Beyond that, if you still want to compute AIC (e.g. your supervisor or
> a reviewer is forcing to do it, and you don't think you're in a position
> to push back effectively):
>
> * as long as you include the Jacobian correction when you transform
> the predictor variable (i.e. #2), these log-likelihoods (and AICs)
> should in principle be comparable (FWIW the robustness of the derivation
> of AIC is much weaker for non-nested models; Brian Ripley [of MASS fame]
> holds a minority opinion that one should *not* use AICs to compare
> non-nested models)
>
> * computing log-likelihoods/AICs by hand is in principle a good idea,
> but is often difficulty for multi-level models, as various integrals or
> approximations of integrals are involved. The lmer and glmer
> likelihoods (1-4) are definitely comparable. To compare across platforms
> I often try to think of a simplified model that *can* be fitted in both
> platforms (e.g. in this case I think a proportional-odds ordinal
> regression where the response has only two levels should be equivalent
> to a binomial model with cloglog link ...)
>
> cheers
> Ben Bolker
>
> On 2020-03-03 5:29 p.m., Kate R wrote:
> > Hi all,
> >
> > Thank you in advance for your time and consideration! I am a
> > non-mathematically-inclined graduate student in communication just
> learning
> > multilevel modeling.
> >
> > I am trying to compare the AIC for 5 different models:
> >
> >
> > 1. model.mn5 <- lmer(anxious ~ num.cm <http://num.cm> + num.pmc
> + (1|userid), data = df,
> > REML = F)
> > 2. model.mn5.log <- lmer(log(anxious) ~ num.cm <http://num.cm>
> + num.pmc + (1|userid),
> > data = df, REML = F)
> > 3. model.mn5.gamma.log <- glmer(anxious ~ num.cm
> <http://num.cm> + num.pmc + (1|userid),
> > data = df, family = Gamma(link="log"))
> > 4. model.mn5.gamma.id <http://model.mn5.gamma.id> <-
> glmer(anxious ~ num.cm <http://num.cm> + num.pmc + (1|userid),
> > data = df, family = Gamma(link="identity"))
> > 5. model.ord5 <- clmm(anxious ~ num.cm <http://num.cm> +
> num.pmc + (1|userid), data =
> > df, na.action = na.omit)
> >
> > (num.cm <http://num.cm> is the group mean and num.pmc is the
> group-mean-centered score of
> > the predictor)
> >
> > Despite many posts on various help forums, I understand that it's
> possible
> > to compare non-nested models with different distributions as long
> as all
> > terms, including constants, are retained (i.e. see Burnham &
> Anderson, Ch
> > 6.7 <https://www.springer.com/gp/book/9780387953649>), but that
> different R
> > packages or model classes might handle constants differently or use
> > different algorithms (see point 7
> <https://robjhyndman.com/hyndsight/aic/>),
> > thus making it difficult to directly compare AIC values. To avoid
> > this non-comparability pitfall, it was suggested in one post to
> calculate
> > your own log-likelihood (though I'm having trouble finding this
> post again).
> >
>
>
>
>
> > Please could you help with the following:
> >
> > - What is the best practice for comparing the AICs for these 5
> models?
> > - What is the R-code for manually calculating the
> log-likelihood and/or
> > the AIC to retain all terms, including constants?
> > - Can you compare ordinal models (clmm) with the continuous models?
> > - Do you recommend any other methods and/or packages for comparing
> > models with different distributions and/or links?
> >
> > Many thanks in advance for your time and consideration! I greatly
> > appreciate any suggestions.
> >
> > Kind regards,
> > K
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org
> <mailto:R-sig-mixed-models using r-project.org> mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
>
>
>
More information about the R-sig-mixed-models
mailing list