[R-sig-ME] Model selection, LRT test
John Maindonald
john.maindonald at anu.edu.au
Mon Jun 20 08:46:11 CEST 2011
With some reinterpretation, the general point that Hodges and Sargent
make is correct. Degrees of freedom are one way, not always effective,
to get a handle on uncertainty in variance and variance ratio estimates.
Uncertainty in degrees of freedom is not, at the end of the day, the nub
of the issue.
The real point at issue is that where models have two or more variances
that have to be estimated, distributions for t-statistics (and other such
statistics) that involve two or more of those variances will depend on
assumptions about the distribution(s) of one or more functions of those
variances, often variance ratios. Opportunities for Behrens-Fisher type
effects can only increase as one moves from t-statistics for comparing
groups where variances may be unequal to multi-level models and
beyond.
One can of course plug the variance estimates in and use those for the
simulation. One is not then accounting for uncertainty in the variance
ratio. It is straightforward to set up a simulation that will demonstrate
the serious underestimate of uncertainties that may result.
Standard types of nonparametric bootstrap may suffer from a different
incarnation of the same problem. Unless a plausible prior is built in,
they will lock in the various ratio that on gets from fitting the model to
the data.
John Maindonald email: john.maindonald at anu.edu.au
phone : +61 2 (6125)3473 fax : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.
http://www.maths.anu.edu.au/~johnm
On 20/06/2011, at 12:09 AM, Ben Bolker wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11-06-19 07:35 AM, Reinhold Kliegl wrote:
>> I also did a quick scan of the paper. Here is a quote from page 156:
>>
>> "In richly parameterised models, such as those containing random
>> effects, estimating the number of degrees of freedom (or alternatively
>> the number of ‘effective’ parameters) is neither straightforward nor
>> necessarily unambiguous (Hodges & Sargent 2001). Hence, attention here
>> is restricted to models that contain fixed effects only."
>>
>> I suspect this puts us back to square 1. Or is there evidence in the
>> paper to the contrary?
>>
>> Reinhold Kliegl
>
> I'm a bit confused by this statement since I don't see what one needs
> the degrees of freedom for in the parametric bootstrap/simulated null
> distribution case ... they do some comparisons with simpler asymptotic
> approaches (AIC, Vuong's test, etc.) that would require knowing (at
> least differences among models in) degrees of freedom. Maybe they just
> wanted to keep the paper short?
>
> Ben Bolker
>
>>
>> On Fri, Jun 17, 2011 at 11:47 PM, Ben Bolker <bbolker at gmail.com> wrote:
>> Based on a very quick glance at the paper, I suspect that the
>> parametric bootstrap approach illustrated in recent releases of the lme4
>> package (library("lme4"); help("simulate-mer")) should be easily
>> adaptable (if my guess is correct and what they are doing is evaluating
>> deviance of model B based on simulations conditional on model A fits and
>> vice versa?)
>>
>> Ben Bolker
>>
>> On 11-06-18 01:44 PM, Arnaud Mosnier wrote:
>>>>> Thanks for this reference, I will try to find this paper.
>>>>> I would also be interested by any comments from the statistics masters
>>>>> that we can find on this list.
>>>>> If it's correct ... is there any package (function) allowing to apply
>>>>> their suggested approach ("simulation-based approach").
>>>>>
>>>>> Arnaud
>>>>>
>>>>> 2011/6/17 luca borger <luca.borger at cebc.cnrs.fr>:
>>>>>>> Thanks to remind me that LRT are only for nested model
>>>>>>
>>>>>> actually, Lewis et al. (2011, Abstract below) just published a paper where they claim that LRTs can be used also for non nested models ("This fact is well-established in the statistical literature, but not widely used in ecological studies."). Has anyone read this paper? I'd be interested to hear any comments (incl. if you believe the authors' approach could be used also for GLMMs).
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Luca
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://onlinelibrary.wiley.com/doi/10.1111/j.2041-210X.2010.00063.x/abstract
>>>>>>
>>>>>> A unified approach to model selection using the likelihood ratio test
>>>>>> Fraser Lewis, Adam Butler, Lucy Gilbert
>>>>>>
>>>>>> Methods in Ecology and Evolution
>>>>>> Volume 2, Issue 2, pages 155162, April 2011
>>>>>> DOI: 10.1111/j.2041-210X.2010.00063.x
>>>>>>
>>>>>> Summary
>>>>>> 1.Ecological count data typically exhibit complexities such as overdispersion and zero-inflation, and are often weakly associated with a relatively large number of correlated covariates. The use of an appropriate statistical model for inference is therefore essential. A common selection criteria for choosing between nested models is the likelihood ratio test (LRT). Widely used alternatives to the LRT are based on information-theoretic metrics such as the Akaike Information Criterion.
>>>>>>
>>>>>> 2.It is widely believed that the LRT can only be used to compare the performance of nested models i.e. in situations where one model is a special case of another. There are many situations in which it is important to compare non-nested models, so, if true, this would be a substantial drawback of using LRTs for model comparison. In reality, however, it is actually possible to use the LRT for comparing both nested and non-nested models. This fact is well-established in the statistical literature, but not widely used in ecological studies.
>>>>>>
>>>>>> 3.The main obstacle to the use of the LRT with non-nested models has, until relatively recently, been the fact that it is difficult to explicitly write down a formula for the distribution of the LRT statistic under the null hypothesis that one of the models is true. With modern computing power it is possible to overcome this difficulty by using a simulation-based approach.
>>>>>>
>>>>>> 4.To demonstrate the practical application of the LRT to both nested and non-nested model comparisons, a case study involving data on questing tick (Ixodes ricinus) abundance is presented. These data contain complexities typical in ecological analyses, such as zero-inflation and overdispersion, for which comparison between models of differing structure e.g. non-nested models is of particular importance.
>>>>>>
>>>>>> 5.Choosing between competing statistical models is an essential part of any applied ecological analysis. The LRT is a standard statistical test for comparing nested models. By use of simulation the LRT can also be used in an analogous fashion to compare non-nested models, thereby providing a unified approach for model comparison within the null hypothesis testing paradigm. A simple practical guide is provided in how to apply this approach to the key models required in the analyses of count data.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Arnaud Mosnier <a.mosnier at gmail.com>
>>>>>> To: Andrew Miles <rstuff.miles at gmail.com>
>>>>>> Date: Fri, 17 Jun 2011 12:44:50 -0400
>>>>>> Subject: Re: [R-sig-ME] Model selection, LRT test
>>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> Thanks to remind me that LRT are only for nested model ... this is not
>>>>>> the case in my situation.
>>>>>>
>>>>>> Arnaud
>>>>>>
>>>>>> 2011/6/17 Andrew Miles <rstuff.miles at gmail.com>:
>>>>>>> What types of models are you running?
>>>>>>>
>>>>>>> There may be two issues at play.
>>>>>>>
>>>>>>> 1. Likelihood ratio tests are only for nested models (i.e. models where the
>>>>>>> variables in one model are a subset of the variables in the other) so by
>>>>>>> definition there will always be a difference in degrees of freedom.
>>>>>>>
>>>>>>> 2. With mixed models you can only use a likelihood ratio test when the model
>>>>>>> returns a deviance score - so not for generalized linear mixed models in
>>>>>>> most cases (though I believe that you can use LRT's if they are estimated
>>>>>>> using a type of numerical integration, but not any sort of quasi-likelihood
>>>>>>> like PQL)
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Fri, Jun 17, 2011 at 11:12 AM, Arnaud Mosnier <a.mosnier at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Dear list,
>>>>>>>>
>>>>>>>> If I do not make a mistake, use of Likelihood ratio test is precluded
>>>>>>>> when two models have the same number of degree of freedom.
>>>>>>>> Is there a way to test which one is the best when both are close in
>>>>>>>> AIC value (difference < 5) or do I have to conclude that they are
>>>>>>>> "equivalent" ?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Arnaud
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>
>>>>>>
>>>>>> __________ Information from ESET Mail Security, version of virus signature database 6217 (20110617) __________
>>>>>>
>>>>>> The message was checked by ESET Mail Security.
>>>>>> http://www.eset.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> __________ Information from ESET Mail Security, version of virus signature database 6218 (20110617) __________
>>>>>>
>>>>>> The message was checked by ESET Mail Security.
>>>>>> http://www.eset.com
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>>>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAk3+Ay4ACgkQc5UpGjwzenPqTwCgl/xIEdelNfiIHjs7F/e3Nldn
> ziMAoJJriT84AsfNJJSxBQNncfHSgxEu
> =vRar
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
More information about the R-sig-mixed-models
mailing list