[R-sig-ME] likelihood-ratio tests in conflict with coefficiants in maximal random effect model

[Martin Maechler]

>>>>> Emilia Ellsiepen <emilia.ellsiepen at gmail.com>
>>>>>     on Mon, 3 Mar 2014 10:47:15 +0100 writes:

    > 2014-02-28 18:29 GMT+01:00 Ben Bolker <bbolker at gmail.com>:
    >> On 14-02-28 11:27 AM, Douglas Bates wrote:
    >>> On Fri, Feb 28, 2014 at 10:04 AM, Emilia Ellsiepen <
    >>> emilia.ellsiepen at gmail.com> wrote:
    >>> 
    >>>> Dear list members,
    >>>> 
    >>>> in analyzing a data set using lmers with maximal random effect
    >>>> structure and subsequent likelihood-ratio tests (LRTs) following Barr
    >>>> et al. 2013, I ran into the following problem: In some of the LRTs, it
    >>>> turned out that the simpler model (only main effects) has a higher
    >>>> likelihood than the more complicated model (including interaction),
    >>>> resulting in Chi=0. If I simplify the models by taking out the
    >>>> interactions in the two random effect terms, the LRT for the
    >>>> interaction has a highly significant result.>
    >> 
    >> [snip]
    >> 
    >>> It shows that the more complex model has not converged to the optimum
    >>> parameter values.  This can be because the optimizer being used is a bad
    >>> choice (in recent versions of the lme4 package the default was a
    >>> Nelder-Mead optimizer that can declare convergence to values that are not
    >>> the optimal values) or it can be because the model is too complex.  We say
    >>> that such models are over-parameterized.
    >>> 
    >>> This is why the Barr et al. advice is dangerous.  In model selection there
    >>> are two basic strategies: forward and backward.  Forward selection starts
    >>> with a simple model and adds terms until they are no longer justified.
    >>> Backward selection starts with the most complex model and drops terms.  It
    >>> is well known that backward selection is problematic when you can't fit the
    >>> most complex model.  Yet Barr et al. say unequivocally that you must use
    >>> backward selection.  The result will be that many researchers, especially
    >>> in linguistics, will encounter these problems of complex models providing
    >>> worse fits than simpler models.
    >>> 
    >>> I wish that Barr et al. would have provided software that is guaranteed to
    >>> fit even the most complex model to a global optimum when they stated their
    >>> "one size fits all" strategy.  they didn't and those with experience in
    >>> numerical optimization can tell you why.  It is not possible to guarantee
    >>> convergence to an optimum in over-parameterized models.
    >>> 
    >> 
    >> [snip]
    >> 
    >> I would be very interested to know whether the more thorough
    >> convergence tests that we have implemented in the development version of
    >> lme4 would correctly report that there are convergence problems with the
    >> model containing the maximal RE structure ...
    >> 
    >> Ben Bolker
    >> 

    > Yes, it does! After installing the development version, I get the
    > following error message:

    > Warning messages:
    > 1: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv,  :
    > Model failed to converge with max|grad| = 12.213 (tol = 0.001)
    > 2: In checkConv(attr(opt, "derivs"), opt$par, ctrl = control$checkConv,  :
    > Model failed to converge: degenerate  Hessian with 3 negative eigenvalues

Aah, great!  
That's very good news.  Thank you, Emilia, for reporting here.
We (lme4 developers) might consider changing the warning into an
error in such an extreme case of non-convergence (12.2 compared
to 0.001). OTOH, good scientists would care about warnings.

So --- just for the readership who'll read this in the future:
Doug Bates was right on spot, and I just like to underline his
comments about the danger of doing science based on data
analysis which "blindly" follows strict 'cooking recipes' rules.
Such rules are something that should be discouraged (if taken strictly) 
rather than taught much more often.

Yes, it would be nice and satisfying if one could always do
backward selection in such cases, but life's reality is not
always nice.

Martin

--
Martin <Maechler at stat.math.ethz.ch>  http://stat.ethz.ch/people/maechler
Seminar für Statistik, ETH Zürich  HG G 16      Rämistrasse 101
CH-8092 Zurich, SWITZERLAND
phone: +41-44-632-3408       fax: ...-1228      <><