[R-sig-ME] Advice on comparing non-nested random slope models
Craig DeMars
cdemars at ualberta.ca
Mon Mar 27 16:16:24 CEST 2017
Thanks. Unfortunately, it doesn't look like the Vuong test has been
implemented for mixed models yet, at least not in R.....
On Mon, Mar 27, 2017 at 4:36 AM, Poe, John <jdpo223 at g.uky.edu> wrote:
> You might try a Vuong test. It's a likelihood ratio test that allows for
> nonnested models.
>
>
> On Mar 26, 2017 8:30 PM, "Paul Buerkner" <paul.buerkner at gmail.com> wrote:
>
> Hi Craig,
>
> in short, significance does not tell you anything about model fit. You may
> find models to have the best fit without any particular predictor being
> significant for this model. Similarily average "effect sizes" are not a
> good indicator of model fit.
>
> Information criteria are, in my opinion, the right way to go. For an
> improved version of the AIC, I recommend going Bayesian and computing the
> so called LOO (leave-one-out cross validation) or the WAIC (widely
> applicable information criterion) as implemented in the R package loo. For
> the bayesian GLMM model fitting (and convenient LOO computation), you could
> use the R packages brms or rstanarm.
>
> Best,
> Paul
>
> 2017-03-26 22:15 GMT+02:00 Craig DeMars <cdemars at ualberta.ca>:
>
> > Hello,
> >
> > This is a bit of a follow-up to a question last week on selecting among
> > GLMM models. Is there a recommended strategy for comparing non-nested,
> > random slope models? I have seen a similar question posted here
> > http://stats.stackexchange.com/questions/116935/comparing-non-nested-
> > models-with-aic but it doesn't seem to answer the problem - and maybe
> > there
> > is no "answer". Zuur et al. (2010) discuss model selection but only in a
> > nested framework. Bolker et al. (2009) suggest AIC can be used in GLMMs
> but
> > caution against boundary issues and don't specifically mention any issues
> > with comparing different random effects structures (as Zuur does).
> >
> > The context of my question comes from an analysis where we have 5 *a
> > priori*
> > hypotheses describing different climate effects on juvenile recruitment
> in
> > an ungulate species. The data set has 21 populations (or herds) with
> > repeated annual measurements of recruitment and the climate variables
> > measured at the herd scale. To generate SE's that reflect herd as the
> > sampling unit, explanatory variables are specified as random slopes
> within
> > herd (as recommended by Schielzeth & Forstmeier 2009; Year is also
> > specified as a random intercept). Because there are only 21 herds,
> models
> > are fairly simple with only 2-3 explanatory variables (3 may by pushing
> > it...????). I can't post the data but it isn't really relevant to the
> > question (I think).
> >
> > Initially, we looked at AIC to compare models. At the bottom of this
> > email, I have pasted the output from two models, each representing
> separate
> > hypotheses, to illustrate "the problem". The first model yields an AIC
> > value of 2210.7. The second model yields an AIC of 2479.5. Using AIC,
> Model
> > 1 would be the "best" model. However, examining the parameter estimates
> > within each model makes me think twice about declaring Model 1 (or the
> > hypothesis it represents) as the most parsimonious explanation for the
> > data. In Model 1, two of the thee fixed effects estimates have small
> effect
> > sizes and all estimates are "non-significant" (if one considers
> > p-values....). In Model 2, two of the three fixed effect estimates have
> > larger effect sizes are would be considered "significant. Is this an
> > example of the difficulty in using AIC to compare non-nested mixed
> > models.....or am I missing something in my interpretation? I haven't come
> > across this type of result when model selecting among GLMs.
> >
> > Any suggestions on how best to compare competing hypotheses represented
> by
> > non-nested GLMMs? Should one just compare relative effect sizes of
> > parameter estimates among models?
> > Any help would be appreciated.
> >
> > Thanks,
> > Craig
> >
> > *Model 1:*
> > Generalized linear mixed model fit by maximum likelihood (Laplace
> > Approximation) ['glmerMod']
> > Family: binomial ( logit )
> > Formula: (Calves/Cows) ~ spr.indvi.ab + green.rate.ab + trend + (1 |
> Year)
> > + (spr.indvi.ab + green.rate.ab + trend | Herd)
> > Data: bou.dat
> > Weights: Cows
> >
> > *AIC * BIC logLik deviance df.resid
> > *2210.7* 2265.0 -1090.3 2180.7 262
> >
> > Scaled residuals:
> > Min 1Q Median 3Q Max
> > -3.8700 -1.0800 -0.1057 1.0405 6.8353
> >
> > Random effects:
> > Groups Name Variance Std.Dev. Corr
> > Year (Intercept) 0.10517 0.3243
> > Herd (Intercept) 0.29832 0.5462
> > spr.indvi.ab 0.04331 0.2081 0.38
> > green.rate.ab 0.03741 0.1934 0.68 0.62
> > trend 0.62661 0.7916 -0.59 0.20 -0.46
> > Number of obs: 277, groups: Year, 22; Herd, 21
> >
> > Fixed effects:
> > Estimate Std. Error z value Pr(>|z|)
> > (Intercept) -1.62160 0.15798 -10.265 <2e-16 ***
> > spr.indvi.ab 0.04019 0.09793 0.410 0.682
> > green.rate.ab 0.04704 0.05555 0.847 0.397
> > trend -0.29676 0.23092 -1.285 0.199
> > ---
> > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> >
> > Correlation of Fixed Effects:
> > (Intr) spr.n. grn.r.
> > spr.indvi.b -0.113
> > green.rat.b 0.347 0.438
> > trend -0.606 0.349 -0.200
> >
> > *Model 2:*
> > Generalized linear mixed model fit by maximum likelihood (Laplace
> > Approximation) ['glmerMod']
> > Family: binomial ( logit )
> > Formula: (Calves/Cows) ~ win.bb + tot.sn.ybb + trend + (1 | Year) + (
> > win.bb
> > + tot.sn.ybb | Herd)
> > Data: bou.dat
> > Weights: Cows
> >
> > * AIC* BIC logLik deviance df.resid
> > *2479.5 * 2519.4 -1228.8 2457.5 266
> >
> > Scaled residuals:
> > Min 1Q Median 3Q Max
> > -4.5720 -1.1801 -0.1364 1.3704 8.3271
> >
> > Random effects:
> > Groups Name Variance Std.Dev. Corr
> > Year (Intercept) 0.10694 0.3270
> > Herd (Intercept) 0.13496 0.3674
> > win.bb 0.05351 0.2313 -0.13
> > tot.sn.ybb 0.06200 0.2490 0.23 0.34
> > Number of obs: 277, groups: Year, 22; Herd, 21
> >
> > Fixed effects:
> > Estimate Std. Error z value Pr(>|z|)
> > (Intercept) -1.851656 0.127702 -14.500 < 2e-16 ***
> > win.bb -0.364019 0.101386 -3.590 0.00033 ***
> > tot.sn.ybb 0.275271 0.118111 2.331 0.01977 *
> > trend -0.007568 0.115706 -0.065 0.94785
> > ---
> > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> >
> > Correlation of Fixed Effects:
> > (Intr) win.bb tt.sn.
> > win.bb 0.048
> > tot.sn.ybb 0.269 0.083
> > trend -0.242 -0.269 -0.131
> > --
> > Craig DeMars, Ph.D.
> > Postdoctoral Fellow
> > Department of Biological Sciences
> > University of Alberta
> > Phone: 780-221-3971 <(780)%20221-3971> <(780)%20221-3971>
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
>
>
--
Craig DeMars, Ph.D.
Postdoctoral Fellow
Department of Biological Sciences
University of Alberta
Phone: 780-221-3971
[[alternative HTML version deleted]]
More information about the R-sig-mixed-models
mailing list