[R-sig-ME] Model selection, LRT test

Sun Jun 19 16:09:50 CEST 2011

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11-06-19 07:35 AM, Reinhold Kliegl wrote:
> I also did a quick scan of the paper. Here is a quote from page 156:
> 
> "In richly parameterised models, such as those containing random
> effects, estimating the number of degrees of freedom (or alternatively
> the number of ‘effective’ parameters) is neither straightforward nor
> necessarily unambiguous (Hodges & Sargent 2001). Hence, attention here
> is restricted to models that contain fixed effects only."
> 
> I suspect this puts us back to square 1. Or is there evidence in the
> paper to the contrary?
> 
> Reinhold Kliegl

  I'm a bit confused by this statement since I don't see what one needs
the degrees of freedom for in the parametric bootstrap/simulated null
distribution case ... they do some comparisons with simpler asymptotic
approaches (AIC, Vuong's test, etc.) that would require knowing (at
least differences among models in) degrees of freedom. Maybe they just
wanted to keep the paper short?

  Ben Bolker

> 
> On Fri, Jun 17, 2011 at 11:47 PM, Ben Bolker <bbolker at gmail.com> wrote:
>  Based on a very quick glance at the paper, I suspect that the
> parametric bootstrap approach illustrated in recent releases of the lme4
> package (library("lme4"); help("simulate-mer")) should be easily
> adaptable (if my guess is correct and what they are doing is evaluating
> deviance of model B based on simulations conditional on model A fits and
> vice versa?)
> 
>  Ben Bolker
> 
> On 11-06-18 01:44 PM, Arnaud Mosnier wrote:
>>>> Thanks for this reference, I will try to find this paper.
>>>> I would also be interested by any comments from the statistics masters
>>>> that we can find on this list.
>>>> If it's correct ... is there any package (function) allowing to apply
>>>> their suggested approach ("simulation-based approach").
>>>>
>>>> Arnaud
>>>>
>>>> 2011/6/17 luca borger <luca.borger at cebc.cnrs.fr>:
>>>>>> Thanks to remind me that LRT are only for nested model
>>>>>
>>>>> actually, Lewis et al. (2011, Abstract below) just published a paper where they claim that LRTs can be used also for non nested models ("This fact is well-established in the statistical literature, but not widely used in ecological studies."). Has anyone read this paper? I'd be interested to hear any comments (incl. if you believe the authors' approach could be used also for GLMMs).
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Luca
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://onlinelibrary.wiley.com/doi/10.1111/j.2041-210X.2010.00063.x/abstract
>>>>>
>>>>> A unified approach to model selection using the likelihood ratio test
>>>>> Fraser Lewis, Adam Butler, Lucy Gilbert
>>>>>
>>>>> Methods in Ecology and Evolution
>>>>> Volume 2, Issue 2, pages 155162, April 2011
>>>>> DOI: 10.1111/j.2041-210X.2010.00063.x
>>>>>
>>>>> Summary
>>>>> 1.Ecological count data typically exhibit complexities such as overdispersion and zero-inflation, and are often weakly associated with a relatively large number of correlated covariates. The use of an appropriate statistical model for inference is therefore essential. A common selection criteria for choosing between nested models is the likelihood ratio test (LRT). Widely used alternatives to the LRT are based on information-theoretic metrics such as the Akaike Information Criterion.
>>>>>
>>>>> 2.It is widely believed that the LRT can only be used to compare the performance of nested models  i.e. in situations where one model is a special case of another. There are many situations in which it is important to compare non-nested models, so, if true, this would be a substantial drawback of using LRTs for model comparison. In reality, however, it is actually possible to use the LRT for comparing both nested and non-nested models. This fact is well-established in the statistical literature, but not widely used in ecological studies.
>>>>>
>>>>> 3.The main obstacle to the use of the LRT with non-nested models has, until relatively recently, been the fact that it is difficult to explicitly write down a formula for the distribution of the LRT statistic under the null hypothesis that one of the models is true. With modern computing power it is possible to overcome this difficulty by using a simulation-based approach.
>>>>>
>>>>> 4.To demonstrate the practical application of the LRT to both nested and non-nested model comparisons, a case study involving data on questing tick (Ixodes ricinus) abundance is presented. These data contain complexities typical in ecological analyses, such as zero-inflation and overdispersion, for which comparison between models of differing structure  e.g. non-nested models  is of particular importance.
>>>>>
>>>>> 5.Choosing between competing statistical models is an essential part of any applied ecological analysis. The LRT is a standard statistical test for comparing nested models. By use of simulation the LRT can also be used in an analogous fashion to compare non-nested models, thereby providing a unified approach for model comparison within the null hypothesis testing paradigm. A simple practical guide is provided in how to apply this approach to the key models required in the analyses of count data.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Arnaud Mosnier <a.mosnier at gmail.com>
>>>>> To: Andrew Miles <rstuff.miles at gmail.com>
>>>>> Date: Fri, 17 Jun 2011 12:44:50 -0400
>>>>> Subject: Re: [R-sig-ME] Model selection, LRT test
>>>>>
>>>>> Andrew,
>>>>>
>>>>> Thanks to remind me that LRT are only for nested model ... this is not
>>>>> the case in my situation.
>>>>>
>>>>> Arnaud
>>>>>
>>>>> 2011/6/17 Andrew Miles <rstuff.miles at gmail.com>:
>>>>>> What types of models are you running?
>>>>>>
>>>>>> There may be two issues at play.
>>>>>>
>>>>>> 1. Likelihood ratio tests are only for nested models (i.e. models where the
>>>>>> variables in one model are a subset of the variables in the other) so by
>>>>>> definition there will always be a difference in degrees of freedom.
>>>>>>
>>>>>> 2. With mixed models you can only use a likelihood ratio test when the model
>>>>>> returns a deviance score - so not for generalized linear mixed models in
>>>>>> most cases (though I believe that you can use LRT's if they are estimated
>>>>>> using a type of numerical integration, but not any sort of quasi-likelihood
>>>>>> like PQL)
>>>>>>
>>>>>> Andrew
>>>>>>
>>>>>> On Fri, Jun 17, 2011 at 11:12 AM, Arnaud Mosnier <a.mosnier at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Dear list,
>>>>>>>
>>>>>>> If I do not make a mistake, use of Likelihood ratio test is precluded
>>>>>>> when two models have the same number of degree of freedom.
>>>>>>> Is there a way to test which one is the best when both are close in
>>>>>>> AIC value (difference < 5) or do I have to conclude that they are
>>>>>>> "equivalent" ?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Arnaud
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>
>>>>>
>>>>> __________ Information from ESET Mail Security, version of virus signature database 6217 (20110617) __________
>>>>>
>>>>> The message was checked by ESET Mail Security.
>>>>> http://www.eset.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> __________ Information from ESET Mail Security, version of virus signature database 6218 (20110617) __________
>>>>>
>>>>> The message was checked by ESET Mail Security.
>>>>> http://www.eset.com
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> 
>>
_______________________________________________
R-sig-mixed-models at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk3+Ay4ACgkQc5UpGjwzenPqTwCgl/xIEdelNfiIHjs7F/e3Nldn
ziMAoJJriT84AsfNJJSxBQNncfHSgxEu
=vRar
-----END PGP SIGNATURE-----