[R-sig-ME] test significance of single random effect

[Ben Bolker]

  I think it will be conservative (in the sense of underestimating the
significance of the random effect), because of the well-known(?)
boundary issue (the null hypothesis for random effects, variance==0, is
on the boundary of the feasible space).

  I went a little overboard in testing this: see
<http://glmm.wikidot.com/random-effects-testing> , and feel free to
improve it ...

Achaz von Hardenberg wrote:
> Dear all,
> I am coming back on the recent issue on how to test the significance  
> of a single random term in linear mixed models...
> 
> In Zuur et al.  "Mixed Models and Extentions in Ecology with R"  
> Springer, 2009, the authors suggest to compare a lme model (with the  
> random effect) with a gls model with the same fixed effects structure,  
> and then compare the AICs of the two models or using a likelihood  
> ratio test via the ANOVA comand (pages 122 - 128).
> 
> I would be interested in hearing the opinion of other members of the  
> list on this approach...
> 
> Thanks a lot,
> 
> Achaz
> 
> 
> On 17 Nov 2009, at 20:41, Tom Van Dooren wrote:
> 
>> With REML=FALSE RLRsim seems to work fine in R 2.10, if I use the  
>> design matrix and Zt as arguments in LRTSim().
>> Otherwise I didn't get useful results out.
>>
>> That's not too much of a problem.
>> It is not difficult to simulate the null model without random  
>> effect, extract logLikelihoods from the (generalized) mixed model  
>> and the (generalized) linear model fitted to those pseudo-data, to  
>> calculate a distribution of likelihood ratios,
>> which are then maybe off by a constant.
>> What I was mainly uncertain about, is whether the log-likelihood of  
>> a mixed model (also fitted to data simulated from the null model  
>> without random effect),
>> can be used as a statistic itself?
>> The answer might be a simple NO! of course, or something more  
>> involved...
>>
>> Tom
>>
>>
>> Douglas Bates wrote:
>>> On Tue, Nov 17, 2009 at 3:49 AM, Matthias Gralle
>>> <matthias_gralle at eva.mpg.de> wrote:
>>>
>>>> I had basically the same problem a short time ago, and resorted to  
>>>> lme
>>>> instead of lmer, because one can directly compare lme and lm  
>>>> objects using
>>>> anova(). Is that OK, or is this feature of lme depreciated ?
>>>>
>>> Is that not possible for linear mixed-effects models fit by lmer  
>>> using
>>> REML = FALSE? (Occasionally I lose track of what can be done in
>>> different versions of lme4.)  You don't want to compare an lmer model
>>> fit by REML with the log-likelihood of an lm model but you should be
>>> able to compare likelihoods (subject to the caveat that the p-value
>>> for the likelihood ratio test on the boundary of the parameter space
>>> is conservative).
>>>
>>>
>>>> Ben Bolker wrote:
>>>>
>>>>> Have you tried the RLRsim package??
>>>>>
>>>>> Tom Van Dooren wrote:
>>>>>
>>>>>
>>>>>> I tried to find an easy way to test whether the random effect  
>>>>>> would be
>>>>>> significant in a (generalized) mixed model with a single random  
>>>>>> effect.
>>>>>> It annoyed me that log-likelihoods of lm or glm and lmer are not
>>>>>> necesarily directly comparable -> trouble with calculating  
>>>>>> likelihood
>>>>>> ratios.
>>>>>> What do members of this list think of the following simulation  
>>>>>> approach?
>>>>>> It basically amounts to simulating a distribution for the log  
>>>>>> likelihood,
>>>>>> given the null hypothesis that there is no random effect  
>>>>>> variance and that
>>>>>> the fixed effect model is correct.
>>>>>>
>>>>>>
>>>>>> library(lme4)
>>>>>> mm1 <- lmer(Reaction ~ Days + (1|Subject), sleepstudy)
>>>>>> lm1<- lm(Reaction ~ Days, sleepstudy)
>>>>>>
>>>>>>
>>>>>> LL<-numeric(500)
>>>>>> for(i in 1:500){
>>>>>> resp<-simulate(lm1)
>>>>>> LL[i]<-logLik(lmer(resp[,1] ~ Days + (1|Subject), sleepstudy))
>>>>>> }
>>>>>>
>>>>>> hist(LL)
>>>>>> logLik(mm1)
>>>>>> mean(LL>logLik(mm1))
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> Matthias Gralle, PhD
>>>> Dept. Evolutionary Genetics
>>>> Max Planck Institute for Evolutionary Anthropology
>>>> Deutscher Platz 6
>>>> 04103 Leipzig, Germany
>>>> Tel +49 341 3550 519
>>>> Fax +49 341 3550 555
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>>>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
>>
>>
> 
> Dr. Achaz von Hardenberg
> --------------------------------------------------------------------------------------------------------
> Centro Studi Fauna Alpina - Alpine Wildlife Research Centre
> Servizio Sanitario e della Ricerca Scientifica
> Parco Nazionale Gran Paradiso, Degioz, 11, 11010-Valsavarenche (Ao),  
> Italy
> 
> Present address:
> National Centre for Statistical Ecology
> School of Mathematics, Statistics and Actuarial Science,
> University of Kent,  Canterbury, UK
> 
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models


-- 
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker at ufl.edu / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc