[R-sig-ME] Advice on comparing non-nested random slope models
Paul Buerkner
paul.buerkner at gmail.com
Mon Mar 27 02:30:05 CEST 2017
Hi Craig,
in short, significance does not tell you anything about model fit. You may
find models to have the best fit without any particular predictor being
significant for this model. Similarily average "effect sizes" are not a
good indicator of model fit.
Information criteria are, in my opinion, the right way to go. For an
improved version of the AIC, I recommend going Bayesian and computing the
so called LOO (leave-one-out cross validation) or the WAIC (widely
applicable information criterion) as implemented in the R package loo. For
the bayesian GLMM model fitting (and convenient LOO computation), you could
use the R packages brms or rstanarm.
Best,
Paul
2017-03-26 22:15 GMT+02:00 Craig DeMars <cdemars at ualberta.ca>:
> Hello,
>
> This is a bit of a follow-up to a question last week on selecting among
> GLMM models. Is there a recommended strategy for comparing non-nested,
> random slope models? I have seen a similar question posted here
> http://stats.stackexchange.com/questions/116935/comparing-non-nested-
> models-with-aic but it doesn't seem to answer the problem - and maybe
> there
> is no "answer". Zuur et al. (2010) discuss model selection but only in a
> nested framework. Bolker et al. (2009) suggest AIC can be used in GLMMs but
> caution against boundary issues and don't specifically mention any issues
> with comparing different random effects structures (as Zuur does).
>
> The context of my question comes from an analysis where we have 5 *a
> priori*
> hypotheses describing different climate effects on juvenile recruitment in
> an ungulate species. The data set has 21 populations (or herds) with
> repeated annual measurements of recruitment and the climate variables
> measured at the herd scale. To generate SE's that reflect herd as the
> sampling unit, explanatory variables are specified as random slopes within
> herd (as recommended by Schielzeth & Forstmeier 2009; Year is also
> specified as a random intercept). Because there are only 21 herds, models
> are fairly simple with only 2-3 explanatory variables (3 may by pushing
> it...????). I can't post the data but it isn't really relevant to the
> question (I think).
>
> Initially, we looked at AIC to compare models. At the bottom of this
> email, I have pasted the output from two models, each representing separate
> hypotheses, to illustrate "the problem". The first model yields an AIC
> value of 2210.7. The second model yields an AIC of 2479.5. Using AIC, Model
> 1 would be the "best" model. However, examining the parameter estimates
> within each model makes me think twice about declaring Model 1 (or the
> hypothesis it represents) as the most parsimonious explanation for the
> data. In Model 1, two of the thee fixed effects estimates have small effect
> sizes and all estimates are "non-significant" (if one considers
> p-values....). In Model 2, two of the three fixed effect estimates have
> larger effect sizes are would be considered "significant. Is this an
> example of the difficulty in using AIC to compare non-nested mixed
> models.....or am I missing something in my interpretation? I haven't come
> across this type of result when model selecting among GLMs.
>
> Any suggestions on how best to compare competing hypotheses represented by
> non-nested GLMMs? Should one just compare relative effect sizes of
> parameter estimates among models?
> Any help would be appreciated.
>
> Thanks,
> Craig
>
> *Model 1:*
> Generalized linear mixed model fit by maximum likelihood (Laplace
> Approximation) ['glmerMod']
> Family: binomial ( logit )
> Formula: (Calves/Cows) ~ spr.indvi.ab + green.rate.ab + trend + (1 | Year)
> + (spr.indvi.ab + green.rate.ab + trend | Herd)
> Data: bou.dat
> Weights: Cows
>
> *AIC * BIC logLik deviance df.resid
> *2210.7* 2265.0 -1090.3 2180.7 262
>
> Scaled residuals:
> Min 1Q Median 3Q Max
> -3.8700 -1.0800 -0.1057 1.0405 6.8353
>
> Random effects:
> Groups Name Variance Std.Dev. Corr
> Year (Intercept) 0.10517 0.3243
> Herd (Intercept) 0.29832 0.5462
> spr.indvi.ab 0.04331 0.2081 0.38
> green.rate.ab 0.03741 0.1934 0.68 0.62
> trend 0.62661 0.7916 -0.59 0.20 -0.46
> Number of obs: 277, groups: Year, 22; Herd, 21
>
> Fixed effects:
> Estimate Std. Error z value Pr(>|z|)
> (Intercept) -1.62160 0.15798 -10.265 <2e-16 ***
> spr.indvi.ab 0.04019 0.09793 0.410 0.682
> green.rate.ab 0.04704 0.05555 0.847 0.397
> trend -0.29676 0.23092 -1.285 0.199
> ---
> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
> Correlation of Fixed Effects:
> (Intr) spr.n. grn.r.
> spr.indvi.b -0.113
> green.rat.b 0.347 0.438
> trend -0.606 0.349 -0.200
>
> *Model 2:*
> Generalized linear mixed model fit by maximum likelihood (Laplace
> Approximation) ['glmerMod']
> Family: binomial ( logit )
> Formula: (Calves/Cows) ~ win.bb + tot.sn.ybb + trend + (1 | Year) + (
> win.bb
> + tot.sn.ybb | Herd)
> Data: bou.dat
> Weights: Cows
>
> * AIC* BIC logLik deviance df.resid
> *2479.5 * 2519.4 -1228.8 2457.5 266
>
> Scaled residuals:
> Min 1Q Median 3Q Max
> -4.5720 -1.1801 -0.1364 1.3704 8.3271
>
> Random effects:
> Groups Name Variance Std.Dev. Corr
> Year (Intercept) 0.10694 0.3270
> Herd (Intercept) 0.13496 0.3674
> win.bb 0.05351 0.2313 -0.13
> tot.sn.ybb 0.06200 0.2490 0.23 0.34
> Number of obs: 277, groups: Year, 22; Herd, 21
>
> Fixed effects:
> Estimate Std. Error z value Pr(>|z|)
> (Intercept) -1.851656 0.127702 -14.500 < 2e-16 ***
> win.bb -0.364019 0.101386 -3.590 0.00033 ***
> tot.sn.ybb 0.275271 0.118111 2.331 0.01977 *
> trend -0.007568 0.115706 -0.065 0.94785
> ---
> Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
> Correlation of Fixed Effects:
> (Intr) win.bb tt.sn.
> win.bb 0.048
> tot.sn.ybb 0.269 0.083
> trend -0.242 -0.269 -0.131
> --
> Craig DeMars, Ph.D.
> Postdoctoral Fellow
> Department of Biological Sciences
> University of Alberta
> Phone: 780-221-3971 <(780)%20221-3971>
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
[[alternative HTML version deleted]]
More information about the R-sig-mixed-models
mailing list