[R-sig-ME] Likelihood Ratio Test for non-nested mixed-effect-model comparison

[Merkle, Edgar C.]

Ben, Francesco,

I have developed a package, nonnest2, that implements the Vuong tests (multiple test statistics) for many types of R models including glm() models.  I believe we will soon be able to add models estimated via lmer().  For an overview of the Vuong test statistics (mostly in the context of structural equation models, but the idea is the same regardless), see my paper here:

http://arxiv.org/pdf/1402.6720v3

We have also been interested in getting the tests to work with models estimated via glmer().  The problem is that the tests require scores of the fitted model (casewise first derivatives of the log-likelihood evaluated at the ML estimates), and it is difficult to obtain scores when the likelihood involves an intractable integral.  My students and I have tentatively concluded that there is no way to use the lme4 source in order to obtain the scores because pirls generally avoids dealing directly with the likelihood.  But I would be happy to hear hints otherwise.

Best,
Ed Merkle

________________________________________
From: Ben Bolker [bbolker at gmail.com]
Sent: Friday, October 23, 2015 1:18 PM
To: r-sig-mixed-models at r-project.org
Subject: Re: [R-sig-ME] Likelihood Ratio Test for non-nested mixed-effect-model comparison

  The Vuong test <https://en.wikipedia.org/wiki/Vuong's_closeness_test>
provides a likelihood-based framework for distinguishing between
non-nested models.  I'm not immediately aware of an implementation that
works with mixed models in R, but you could look around (library("sos");
findFn("vuong") and screen the results for applicability ...)

  As a crude alternative you could do something parametric bootstrap-y
by simulating from each fitted model, fitting each realization with both
fitted models, and comparing the goodness of fit distributions (that's
intentionally vague, I haven't thought about it very much ...)

  Ben Bolker

On 15-10-23 11:40 AM, Francesco Sigona wrote:
> Hi all,
>
> I need to compare two mixed-effects-models that would explain a
> dependent variable by means of two completely different sets of fixed
> factors (my random intercept is the same in both models).
>
> I understand that the anova() cannot be used to perform a comparison via
> LRTest in my case, because my models are non-nested.
>
> Thus, I would take the model with the lowest AIC (o BIC) value, but I am
> worried about the statistical significance, so I would prefer a LRTest
> and the related p-value (as provided by anova() for nested models) to
> support my model selection.
>
> My problem is that I don't know how to compare two non-nested
> mixed-effect-models via a LRTest.
>
> Any suggestion?
>
> Thank you in advance.
>
> Francesco
>