On Fri, Feb 28, 2014 at 10:04 AM, Emilia Ellsiepen <
emilia.ellsiepen@gmail.com> wrote:

> Dear list members,
>
> in analyzing a data set using lmers with maximal random effect
> structure and subsequent likelihood-ratio tests (LRTs) following Barr
> et al. 2013, I ran into the following problem: In some of the LRTs, it
> turned out that the simpler model (only main effects) has a higher
> likelihood than the more complicated model (including interaction),
> resulting in Chi=0. If I simplify the models by taking out the
> interactions in the two random effect terms, the LRT for the
> interaction has a highly significant result.
>
> Now, I figured this might either be a strong case for Barr et al., or
> something is going wrong, as the data do suggest an interaction, and
> the t-value for the interaction coefficiant in the model summary of
> the maximal random effect model is 1.69 - which I would have expected
> to be a (close to) marginal effect. Also, a lot of researchers who do
> use maximal random effect models rather interpret the t-values
> directly instead of running LRTs, which then might lead to different
> interpretations of the same data.
>

It shows that the more complex model has not converged to the optimum
parameter values.  This can be because the optimizer being used is a bad
choice (in recent versions of the lme4 package the default was a
Nelder-Mead optimizer that can declare convergence to values that are not
the optimal values) or it can be because the model is too complex.  We say
that such models are over-parameterized.

This is why the Barr et al. advice is dangerous.  In model selection there
are two basic strategies: forward and backward.  Forward selection starts
with a simple model and adds terms until they are no longer justified.
 Backward selection starts with the most complex model and drops terms.  It
is well known that backward selection is problematic when you can't fit the
most complex model.  Yet Barr et al. say unequivocally that you must use
backward selection.  The result will be that many researchers, especially
in linguistics, will encounter these problems of complex models providing
worse fits than simpler models.

I wish that Barr et al. would have provided software that is guaranteed to
fit even the most complex model to a global optimum when they stated their
"one size fits all" strategy.  they didn't and those with experience in
numerical optimization can tell you why.  It is not possible to guarantee
convergence to an optimum in over-parameterized models.

This is why papers like Barr et al. that, based on a single simulation
study, state absolute rules about the complex and subtle process of model
selection, are dangerous.

My questions are:
> 1) Is the LRT reported below reliable, indicating there is no
> interaction, or does the non-expected difference in logLik suggest
> that something went wrong?
> 2) Is it possible that the control parameter that increases the number
> of evaluations has something to do with it?
>
>
> Here are some details:
> The design is 2x2 within subjects and items, the factors are sum-coded.
> I used REML=F to straightforwardly perform the LRTs afterwards and
> increased the number of evaluations for the model to converge
>
> m1 = lmer(score ~ Order*Voice +
> (Order*Voice|subject)+(Order*Voice|sentence) ,REML=F,
> control=lmerControl(optCtrl=list(maxfun=50000)),rawdata)
> mi = lmer(score ~ Order+Voice +
> (Order*Voice|subject)+(Order*Voice|sentence) ,REML=F,
> control=lmerControl(optCtrl=list(maxfun=50000)),rawdata)
>
> anova(m1,mi)
>
> Models:
> mi: score ~ Order + Voice + (Order * Voice | subject) + (Order *
> mi:     Voice | sentence)
> m1: score ~ Order * Voice + (Order * Voice | subject) + (Order *
> m1:     Voice | sentence)
>    Df    AIC    BIC  logLik deviance Chisq Chi Df Pr(>Chisq)
> mi 24 1633.0 1742.9 -792.48   1585.0
> m1 25 1664.1 1778.5 -807.03   1614.1     0      1          1
>
>
> summary(m1)
>
> Linear mixed model fit by maximum likelihood ['lmerMod']
> Formula: score ~ Order * Voice + (Order * Voice | subject) + (Order *
>     Voice | sentence)
>
>       AIC       BIC    logLik  deviance
> 1664.0644 1778.5457 -807.0322 1614.0644
>
> Random effects:
>  Groups   Name          Variance  Std.Dev. Corr
>  subject  (Intercept)   0.0387850 0.19694
>           Order1        0.0005166 0.02273  0.65
>           Voice1        0.0326954 0.18082  0.71 0.55
>           Order1:Voice1 0.0266763 0.16333  0.34 0.76 0.57
>  sentence (Intercept)   0.0683897 0.26151
>           Order1        0.0779307 0.27916  1.00
>           Voice1        0.0096621 0.09830  0.61 0.60
>           Order1:Voice1 0.1013066 0.31829  0.55 0.55 0.34
>  Residual               0.4272769 0.65366
> Number of obs: 720, groups: subject, 36; sentence, 20
>
> Fixed effects:
>               Estimate Std. Error t value
> (Intercept)    0.26184    0.07135   3.670
> Order1        -0.12645    0.06711  -1.884
> Voice1        -0.10761    0.04455  -2.415
> Order1:Voice1  0.13562    0.08000   1.695
>
> Correlation of Fixed Effects:
>             (Intr) Order1 Voice1
> Order1      0.776
> Voice1      0.469  0.294
> Order1:Voc1 0.458  0.468  0.282
>
> ----------------------------------------------------
> Emilia Ellsiepen
> Institut für Linguistik
> Goethe-Universität Frankfurt
>
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
>

	[[alternative HTML version deleted]]

