[R-sig-ME] test significance of single random effect
Highland Statistics Ltd.
highstat at highstat.com
Sun Nov 29 12:36:23 CET 2009
> 1. Re: test significance of single random effect
> (Achaz von Hardenberg)
> 2. Re: test significance of single random effect (Ben Bolker)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sat, 28 Nov 2009 15:23:01 +0000
> From: Achaz von Hardenberg <achaz.hardenberg at gmail.com>
> Subject: Re: [R-sig-ME] test significance of single random effect
> To: R Mixed Models <r-sig-mixed-models at r-project.org>
> Message-ID: <E7D9F7E1-84DA-4C1F-B9D4-F6222564975E at pngp.it>
> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
>
> Dear all,
> I am coming back on the recent issue on how to test the significance
> of a single random term in linear mixed models...
>
> In Zuur et al. "Mixed Models and Extentions in Ecology with R"
> Springer, 2009, the authors suggest to compare a lme model (with the
> random effect) with a gls model with the same fixed effects structure,
> and then compare the AICs of the two models or using a likelihood
> ratio test via the ANOVA comand (pages 122 - 128).
>
>
See also Verbeke and Molenberghs (2000)... The correction for testing on
the boundary is described on page 123 (and see also V&M) and is viewed
as quick and dirty.
My question to you...why do you want to test the significance of a
random term in a linear mixed model? Why not include it purely based on
the design of the experiment?
Alain
> I would be interested in hearing the opinion of other members of the
> list on this approach...
>
> Thanks a lot,
>
> Achaz
>
>
> On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:
>
>
>> With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the
>> design matrix and Zt as arguments in LRTSim().
>> Otherwise I didn't get useful results out.
>>
>> That's not too much of a problem.
>> It is not difficult to simulate the null model without random
>> effect, extract logLikelihoods from the (generalized) mixed model
>> and the (generalized) linear model fitted to those pseudo-data, to
>> calculate a distribution of likelihood ratios,
>> which are then maybe off by a constant.
>> What I was mainly uncertain about, is whether the log-likelihood of
>> a mixed model (also fitted to data simulated from the null model
>> without random effect),
>> can be used as a statistic itself?
>> The answer might be a simple NO! of course, or something more
>> involved...
>>
>> Tom
>>
>>
>> Douglas Bates wrote:
>>
>>> On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle
>>> <matthias_gralle at eva.mpg.de> wrote:
>>>
>>>
>>>> I had basically the same problem a short time ago, and resorted to
>>>> lme
>>>> instead of lmer, because one can directly compare lme and lm
>>>> objects using
>>>> anova(). Is that OK, or is this feature of lme depreciated ?
>>>>
>>>>
>>> Is that not possible for linear mixed-effects models fit by lmer
>>> using
>>> REML = FALSE? (Occasionally I lose track of what can be done in
>>> different versions of lme4.) You don't want to compare an lmer model
>>> fit by REML with the log-likelihood of an lm model but you should be
>>> able to compare likelihoods (subject to the caveat that the p-value
>>> for the likelihood ratio test on the boundary of the parameter space
>>> is conservative).
>>>
>>>
>>>
>>>> Ben Bolker wrote:
>>>>
>>>>
>>>>> Have you tried the RLRsim package??
>>>>>
>>>>> Tom Van Dooren wrote:
>>>>>
>>>>>
>>>>>
>>>>>> I tried to find an easy way to test whether the random effect
>>>>>> would be
>>>>>> significant in a (generalized) mixed model with a single random
>>>>>> effect.
>>>>>> It annoyed me that log-likelihoods of lm or glm and lmer are not
>>>>>> necesarily directly comparable -> trouble with calculating
>>>>>> likelihood
>>>>>> ratios.
>>>>>> What do members of this list think of the following simulation
>>>>>> approach?
>>>>>> It basically amounts to simulating a distribution for the log
>>>>>> likelihood,
>>>>>> given the null hypothesis that there is no random effect
>>>>>> variance and that
>>>>>> the fixed effect model is correct.
>>>>>>
>>>>>>
>>>>>> library(lme4)
>>>>>> mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
>>>>>> lm1<- lm(Reaction ~ Days, sleepstudy)
>>>>>>
>>>>>>
>>>>>> LL<-numeric(500)
>>>>>> for(i in 1:500){
>>>>>> resp<-simulate(lm1)
>>>>>> LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
>>>>>> }
>>>>>>
>>>>>> hist(LL)
>>>>>> logLik(mm1)
>>>>>> mean(LL>logLik(mm1))
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> Matthias Gralle, PhD
>>>> Dept. Evolutionary Genetics
>>>> Max Planck Institute for Evolutionary Anthropology
>>>> Deutscher Platz 6
>>>> 04103 Leipzig, Germany
>>>> Tel +49 341 3550 519
>>>> Fax +49 341 3550 555
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>>>>
>>>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>>
>>
>>
>
> Dr. Achaz von Hardenberg
> --------------------------------------------------------------------------------------------------------
> Centro Studi Fauna Alpina - Alpine Wildlife Research Centre
> Servizio Sanitario e della Ricerca Scientifica
> Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao),
> Italy
>
> Present address:
> National Centre for Statistical Ecology
> School of Mathematics, Statistics and Actuarial Science,
> University of Kent, Canterbury, UK
>
>
>
> ------------------------------
>
> Message: 2
> Date: Sat, 28 Nov 2009 21:49:34 -0500
> From: Ben Bolker <bolker at ufl.edu>
> Subject: Re: [R-sig-ME] test significance of single random effect
> To: Achaz von Hardenberg <achaz.hardenberg at gmail.com>
> Cc: R Mixed Models <r-sig-mixed-models at r-project.org>
> Message-ID: <4B11E13E.3030105 at ufl.edu>
> Content-Type: text/plain; charset=ISO-8859-1
>
> I think it will be conservative (in the sense of underestimating the
> significance of the random effect), because of the well-known(?)
> boundary issue (the null hypothesis for random effects, variance==0, is
> on the boundary of the feasible space).
>
> I went a little overboard in testing this: see
> <http://glmm.wikidot.com/random-effects-testing> , and feel free to
> improve it ...
>
> Achaz von Hardenberg wrote:
>
>> Dear all,
>> I am coming back on the recent issue on how to test the significance
>> of a single random term in linear mixed models...
>>
>> In Zuur et al. "Mixed Models and Extentions in Ecology with R"
>> Springer, 2009, the authors suggest to compare a lme model (with the
>> random effect) with a gls model with the same fixed effects structure,
>> and then compare the AICs of the two models or using a likelihood
>> ratio test via the ANOVA comand (pages 122 - 128).
>>
>> I would be interested in hearing the opinion of other members of the
>> list on this approach...
>>
>> Thanks a lot,
>>
>> Achaz
>>
>>
>> On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:
>>
>>
>>> With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the
>>> design matrix and Zt as arguments in LRTSim().
>>> Otherwise I didn't get useful results out.
>>>
>>> That's not too much of a problem.
>>> It is not difficult to simulate the null model without random
>>> effect, extract logLikelihoods from the (generalized) mixed model
>>> and the (generalized) linear model fitted to those pseudo-data, to
>>> calculate a distribution of likelihood ratios,
>>> which are then maybe off by a constant.
>>> What I was mainly uncertain about, is whether the log-likelihood of
>>> a mixed model (also fitted to data simulated from the null model
>>> without random effect),
>>> can be used as a statistic itself?
>>> The answer might be a simple NO! of course, or something more
>>> involved...
>>>
>>> Tom
>>>
>>>
>>> Douglas Bates wrote:
>>>
>>>> On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle
>>>> <matthias_gralle at eva.mpg.de> wrote:
>>>>
>>>>
>>>>> I had basically the same problem a short time ago, and resorted to
>>>>> lme
>>>>> instead of lmer, because one can directly compare lme and lm
>>>>> objects using
>>>>> anova(). Is that OK, or is this feature of lme depreciated ?
>>>>>
>>>>>
>>>> Is that not possible for linear mixed-effects models fit by lmer
>>>> using
>>>> REML = FALSE? (Occasionally I lose track of what can be done in
>>>> different versions of lme4.) You don't want to compare an lmer model
>>>> fit by REML with the log-likelihood of an lm model but you should be
>>>> able to compare likelihoods (subject to the caveat that the p-value
>>>> for the likelihood ratio test on the boundary of the parameter space
>>>> is conservative).
>>>>
>>>>
>>>>
>>>>> Ben Bolker wrote:
>>>>>
>>>>>
>>>>>> Have you tried the RLRsim package??
>>>>>>
>>>>>> Tom Van Dooren wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> I tried to find an easy way to test whether the random effect
>>>>>>> would be
>>>>>>> significant in a (generalized) mixed model with a single random
>>>>>>> effect.
>>>>>>> It annoyed me that log-likelihoods of lm or glm and lmer are not
>>>>>>> necesarily directly comparable -> trouble with calculating
>>>>>>> likelihood
>>>>>>> ratios.
>>>>>>> What do members of this list think of the following simulation
>>>>>>> approach?
>>>>>>> It basically amounts to simulating a distribution for the log
>>>>>>> likelihood,
>>>>>>> given the null hypothesis that there is no random effect
>>>>>>> variance and that
>>>>>>> the fixed effect model is correct.
>>>>>>>
>>>>>>>
>>>>>>> library(lme4)
>>>>>>> mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
>>>>>>> lm1<- lm(Reaction ~ Days, sleepstudy)
>>>>>>>
>>>>>>>
>>>>>>> LL<-numeric(500)
>>>>>>> for(i in 1:500){
>>>>>>> resp<-simulate(lm1)
>>>>>>> LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
>>>>>>> }
>>>>>>>
>>>>>>> hist(LL)
>>>>>>> logLik(mm1)
>>>>>>> mean(LL>logLik(mm1))
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Matthias Gralle, PhD
>>>>> Dept. Evolutionary Genetics
>>>>> Max Planck Institute for Evolutionary Anthropology
>>>>> Deutscher Platz 6
>>>>> 04103 Leipzig, Germany
>>>>> Tel +49 341 3550 519
>>>>> Fax +49 341 3550 555
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>
>>>>>
>>>>>
>>> _______________________________________________
>>> R-sig-mixed-models at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>
>>>
>>>
>>>
>> Dr. Achaz von Hardenberg
>> --------------------------------------------------------------------------------------------------------
>> Centro Studi Fauna Alpina - Alpine Wildlife Research Centre
>> Servizio Sanitario e della Ricerca Scientifica
>> Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao),
>> Italy
>>
>> Present address:
>> National Centre for Statistical Ecology
>> School of Mathematics, Statistics and Actuarial Science,
>> University of Kent, Canterbury, UK
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>
>
>
--
Dr. Alain F. Zuur
First author of:
1. Analysing Ecological Data (2007).
Zuur, AF, Ieno, EN and Smith, GM. Springer. 680 p.
URL: www.springer.com/0-387-45967-7
2. Mixed effects models and extensions in ecology with R. (2009).
Zuur, AF, Ieno, EN, Walker, N, Saveliev, AA, and Smith, GM. Springer.
http://www.springer.com/life+sci/ecology/book/978-0-387-87457-9
3. A Beginner's Guide to R (2009).
Zuur, AF, Ieno, EN, Meesters, EHWG. Springer
http://www.springer.com/statistics/computational/book/978-0-387-93836-3
Other books: http://www.highstat.com/books.htm
Statistical consultancy, courses, data analysis and software
Highland Statistics Ltd.
6 Laverock road
UK - AB41 6FN Newburgh
Tel: 0044 1358 788177
Email: highstat at highstat.com
URL: www.highstat.com
URL: www.brodgar.com
More information about the R-sig-mixed-models
mailing list