[R-sig-ME] AIC Comparison for MLM with Different Distributions

[Ben Bolker]

  I agree with Thierry's big-picture comment that you should generally
use broader/qualitative criteria to decide on a model rather than
testing all possibilities.  The only exception I can think of is if you
are *only* interested in predictive accuracy (not in inference
[confidence intervals/p-values etc.]), and you make sure to use
cross-validation or a testing set to evaluate out-of-sample predictive
error (although AIC *should* generally give a reasonable approximation
to relative out-of-sample error).

  Beyond that, if you still want to compute AIC (e.g. your supervisor or
a reviewer is forcing to do it, and you don't think you're in a position
to push back effectively):

  * as long as you include the Jacobian correction when you transform
the predictor variable (i.e. #2), these log-likelihoods (and AICs)
should in principle be comparable (FWIW the robustness of the derivation
of AIC is much weaker for non-nested models; Brian Ripley [of MASS fame]
holds a minority opinion that one should *not* use AICs to compare
non-nested models)

  * computing log-likelihoods/AICs by hand is in principle a good idea,
but is often difficulty for multi-level models, as various integrals or
approximations of integrals are involved.  The lmer and glmer
likelihoods (1-4) are definitely comparable. To compare across platforms
I often try to think of a simplified model that *can* be fitted in both
platforms (e.g. in this case I think a proportional-odds ordinal
regression where the response has only two levels should be equivalent
to a binomial model with cloglog link ...)

  cheers
  Ben Bolker

On 2020-03-03 5:29 p.m., Kate R wrote:
> Hi all,
> 
> Thank you in advance for your time and consideration! I am a
> non-mathematically-inclined graduate student in communication just learning
> multilevel modeling.
> 
> I am trying to compare the AIC for 5 different models:
> 
> 
>    1. model.mn5 <- lmer(anxious ~ num.cm + num.pmc + (1|userid), data = df,
>    REML = F)
>    2. model.mn5.log <- lmer(log(anxious) ~ num.cm + num.pmc + (1|userid),
>    data = df, REML = F)
>    3. model.mn5.gamma.log <- glmer(anxious ~ num.cm + num.pmc + (1|userid),
>    data = df, family = Gamma(link="log"))
>    4. model.mn5.gamma.id <- glmer(anxious ~ num.cm + num.pmc + (1|userid),
>    data = df, family = Gamma(link="identity"))
>    5. model.ord5 <- clmm(anxious ~ num.cm + num.pmc + (1|userid), data =
>    df, na.action = na.omit)
> 
> (num.cm is the group mean and num.pmc is the group-mean-centered score of
> the predictor)
> 
> Despite many posts on various help forums, I understand that it's possible
> to compare non-nested models with different distributions as long as all
> terms, including constants, are retained (i.e. see Burnham & Anderson, Ch
> 6.7 <https://www.springer.com/gp/book/9780387953649>), but that different R
> packages or model classes might handle constants differently or use
> different algorithms (see point 7 <https://robjhyndman.com/hyndsight/aic/>),
> thus making it difficult to directly compare AIC values. To avoid
> this non-comparability pitfall, it was suggested in one post to calculate
> your own log-likelihood (though I'm having trouble finding this post again).
> 




> Please could you help with the following:
> 
>    - What is the best practice for comparing the AICs for these 5 models?
>    - What is the R-code for manually calculating the log-likelihood and/or
>    the AIC to retain all terms, including constants?
>    - Can you compare ordinal models (clmm) with the continuous models?
>    - Do you recommend any other methods and/or packages for comparing
>    models with different distributions and/or links?
> 
> Many thanks in advance for your time and consideration! I greatly
> appreciate any suggestions.
> 
> Kind regards,
> K
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>