[R-sig-ME] Advice on comparing non-nested random slope models

[Craig DeMars]

Hello,

This is a bit of a follow-up to a question last week on selecting among
GLMM models. Is there a recommended strategy for comparing non-nested,
random slope models? I have seen a similar question posted here
http://stats.stackexchange.com/questions/116935/comparing-non-nested-
models-with-aic but it doesn't seem to answer the problem - and maybe there
is no "answer".  Zuur et al. (2010) discuss model selection but only in a
nested framework. Bolker et al. (2009) suggest AIC can be used in GLMMs but
caution against boundary issues and don't specifically mention any issues
with comparing different random effects structures (as Zuur does).

The context of my question comes from an analysis where we have 5 *a priori*
hypotheses describing different climate effects on juvenile recruitment in
an ungulate species.  The data set has 21 populations (or herds) with
repeated annual measurements of recruitment and the climate variables
measured at the herd scale. To generate SE's that reflect herd as the
sampling unit, explanatory variables are specified as random slopes within
herd (as recommended by Schielzeth & Forstmeier 2009; Year is also
specified as a random intercept).  Because there are only 21 herds, models
are fairly simple with only 2-3 explanatory variables (3 may by pushing
it...????). I can't post the data but it isn't really relevant to the
question (I think).

Initially, we looked at AIC to compare models.  At the bottom of this
email, I have pasted the output from two models, each representing separate
hypotheses, to illustrate "the problem".  The first model yields an AIC
value of 2210.7. The second model yields an AIC of 2479.5. Using AIC, Model
1 would be the "best" model. However, examining the parameter estimates
within each model makes me think twice about declaring  Model 1 (or the
hypothesis it represents) as the most parsimonious explanation for the
data. In Model 1, two of the thee fixed effects estimates have small effect
sizes and all estimates are "non-significant" (if one considers
p-values....). In Model 2, two of the three fixed effect estimates have
larger effect sizes are would be considered "significant.  Is this an
example of the difficulty in using AIC to compare non-nested mixed
models.....or am I missing something in my interpretation? I haven't come
across this type of result when model selecting among GLMs.

Any suggestions on how best to compare competing hypotheses represented by
non-nested GLMMs? Should one just compare relative effect sizes of
parameter estimates among models?
Any help would be appreciated.

Thanks,
Craig

*Model 1:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: (Calves/Cows) ~ spr.indvi.ab + green.rate.ab + trend + (1 | Year)
+      (spr.indvi.ab + green.rate.ab + trend | Herd)
   Data: bou.dat
Weights: Cows

     *AIC  *    BIC   logLik deviance df.resid
  *2210.7*   2265.0  -1090.3   2180.7      262

Scaled residuals:
    Min      1Q  Median      3Q     Max
-3.8700 -1.0800 -0.1057  1.0405  6.8353

Random effects:
 Groups Name          Variance Std.Dev. Corr
 Year   (Intercept)   0.10517  0.3243
 Herd   (Intercept)   0.29832  0.5462
        spr.indvi.ab  0.04331  0.2081    0.38
        green.rate.ab 0.03741  0.1934    0.68  0.62
        trend         0.62661  0.7916   -0.59  0.20 -0.46
Number of obs: 277, groups:  Year, 22; Herd, 21

Fixed effects:
              Estimate Std. Error z value Pr(>|z|)
(Intercept)   -1.62160    0.15798 -10.265   <2e-16 ***
spr.indvi.ab   0.04019    0.09793   0.410    0.682
green.rate.ab  0.04704    0.05555   0.847    0.397
trend         -0.29676    0.23092  -1.285    0.199
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr) spr.n. grn.r.
spr.indvi.b -0.113
green.rat.b  0.347  0.438
trend       -0.606  0.349 -0.200

*Model 2:*
Generalized linear mixed model fit by maximum likelihood (Laplace
Approximation) ['glmerMod']
 Family: binomial  ( logit )
Formula: (Calves/Cows) ~ win.bb + tot.sn.ybb + trend + (1 | Year) + (win.bb
+      tot.sn.ybb | Herd)
   Data: bou.dat
Weights: Cows

    * AIC*      BIC   logLik deviance df.resid
  *2479.5 *  2519.4  -1228.8   2457.5      266

Scaled residuals:
    Min      1Q  Median      3Q     Max
-4.5720 -1.1801 -0.1364  1.3704  8.3271

Random effects:
 Groups Name        Variance Std.Dev. Corr
 Year   (Intercept) 0.10694  0.3270
 Herd   (Intercept) 0.13496  0.3674
        win.bb      0.05351  0.2313   -0.13
        tot.sn.ybb  0.06200  0.2490    0.23  0.34
Number of obs: 277, groups:  Year, 22; Herd, 21

Fixed effects:
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -1.851656   0.127702 -14.500  < 2e-16 ***
win.bb      -0.364019   0.101386  -3.590  0.00033 ***
tot.sn.ybb   0.275271   0.118111   2.331  0.01977 *
trend       -0.007568   0.115706  -0.065  0.94785
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
           (Intr) win.bb tt.sn.
win.bb      0.048
tot.sn.ybb  0.269  0.083
trend      -0.242 -0.269 -0.131
-- 
Craig DeMars, Ph.D.
Postdoctoral Fellow
Department of Biological Sciences
University of Alberta
Phone: 780-221-3971 <(780)%20221-3971>

	[[alternative HTML version deleted]]