[R-sig-ME] model selection in lme4

John Maindonald john.maindonald at anu.edu.au
Tue Feb 17 01:05:54 CET 2009 

The issue seems to be what kind of generations one wishes to make.  
This determines what conditioning is appropriate, and it determines  
the distribution with respect to which one tries to find the  
expectation that is involved in calculating the AIC or other such  
statistic.  Should one condition wrt to, e.g., the actual numbers of  
plots at the different sites and the actual number of sites, as in the  
data?  Or should these be treated as random?   It all gets too  
horrible to contemplate.  Vaida and Blanchard, and the Liang & Wu &  
Zhou paper, do not do much more than scratch the surface of these  
complications.

The complications are of the same kind as those involved in  
calculating predicted values.  These differ depending on the  
population to which one wishes to generalize.  The SEs vary also, and  
depend on whether one wants the SE of the prediction, or the SE of the  
equivalent observation.  A focus on prediction may be the way to get a  
clear understanding of what should be optimized.

John Maindonald             email: john.maindonald at anu.edu.au
phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
Centre for Mathematics & Its Applications, Room 1194,
John Dedman Mathematical Sciences Building (Building 27)
Australian National University, Canberra ACT 0200.


On 16/02/2009, at 3:15 PM, Simon Blomberg wrote:

> Vaida and Blanchard Biometrika [(2005), 92, 2, pp. 351–370 Conditional
> Akaike information for mixed-effects models] discuss using AIC for  
> model
> selection in mixed-effects models, and make recommendations. There is
> also a follow-up not by Liang, Wu and Zhou. Biometrika (2008), 95, 3,
> pp. 773–778 A note on conditional AIC for linear mixed-effects models.
>
> The general message is that the "type" of AIC statistic will depend on
> your motivation for model selection. Is it the fixed effects part of  
> the
> model that is of most interest? Or are the random effects of specific
> interest too? This "focus" will determine the number of "effective
> parameters" in the penalty term (using results from Hodges, J.S. and
> Sargent, D. J. (2001). Counting degrees of freedom in hierarchical and
> other richly parameterized models. Biometrika 88, 367–79). There is  
> also
> the issue of REML v ML estimation...
>
> Cheers,
>
> Simon.
>
> On Sun, 2009-02-15 at 20:23 -0600, Christopher David Desjardins wrote:
>> For a discussion of BIC, please see Raftery (1995) in Sociological
>> Methodology. Before you commit yourself on the AIC, I do encourage  
>> you to
>> look at your BIC. In the models I've run when there is disagreement  
>> between
>> the BIC and the AIC, it's usually that the AIC selects the overly  
>> complex
>> model and includes unnecessary parameters.
>> Cheers,
>> Chris
>>
>> On Sunday 15 February 2009 19:50:30 Ben Bolker wrote:
>>> It would be better to use AICc, but I'm not sure what I would
>>> use for "number of parameters" for a random effect with n
>>> levels: any number between 0.5 and n seems plausible!
>>> Someone should send Shane Richards (who has done some
>>> very nice work testing (Q)AIC(c) in ecological settings)
>>> and see if he's willing to tackle this one, although I can
>>> imagine he's getting sick of this kind of exercise ...
>>>
>>> Ben Bolker
>>>
>>> Renwick, A. R. wrote:
>>>> Just a quickie Ben,
>>>> Are you saying that you would use AIC rather than AICc even with
>>>> small sample size - due to difficulty in counting residual  
>>>> degrees of
>>>
>>> freedom?
>>>
>>>> Thanks
>>>> Anna
>>>> p.s. this forum really is fantastic
>>>>
>>>> ________________________________________
>>>> From: r-sig-mixed-models-bounces at r-project.org
>>>> [r-sig-mixed-models-bounces at r-project.org] On Behalf Of Ben Bolker
>>>> [bolker at ufl.edu] Sent: 15 February 2009 23:07
>>>> To: Christopher David Desjardins
>>>> Cc: r-sig-mixed-models at r-project.org; tahirajamil at yahoo.com
>>>> Subject: Re: [R-sig-ME] model selection in lme4
>>>>
>>>> Some caution on this advice: you seem to be quoting
>>>> the general advice on AIC/BIC/AICc
>>>>
>>>> 1. The AIC/BIC distinction is between "best prediction"
>>>> and "consistent estimation of true model" dimension, e.g.
>>>>
>>>> Yang, Yuhong. 2005. Can the strengths of AIC and BIC be shared? A
>>>> conflict between model identification and regression estimation.
>>>> Biometrika 92, no. 4 (December 1): 937-950. doi:10.1093/biomet/ 
>>>> 92.4.937.
>>>>
>>>> I favor AIC on these grounds, but you can decide for yourself.
>>>>
>>>> 2. For models with different random effects, AIC and BIC share
>>>> a "degrees of freedom counting" problem with all model selection
>>>> approaches -- there are two aspects here, (1) whether you are
>>>> focused on individual-level prediction or population-level
>>>> prediction (Vaida and Blanchard 2005, Spiegelhalter et al 2002)
>>>> and (2) whether AIC/BIC share the boundary problems that
>>>> also apply to likelihood ratio tests (Greven, Sonja. 2008. Non- 
>>>> Standard
>>>> Problems in Inference for Additive and Linear Mixed Models.  
>>>> Göttingen,
>>>> Germany: Cuvillier Verlag.
>>>> http://www.cuvillier.de/flycms/en/html/30/-UickI3zKPS,3cEY=/Buchdetails.h
>>>> tml?SID=wVZnpL8f0fbc. )
>>>>
>>>> 3. AIC and BIC are asymptotic tests (which can be especially
>>>> problematic with random effects problems, when there are not
>>>> large number of random blocks -- this makes likelihood ratio
>>>> tests NOT OK for fixed-effect comparisons with small numbers
>>>> of blocks (Pinheiro and Bates 2000)).  If you want to use
>>>> AICc then you are back to counting residual degrees of freedom ...
>>>> as far as I know there isn't much guidance available on this
>>>> issue.
>>>>
>>>> My bottom line:
>>>>
>>>> I would go ahead and use (Q)AIC with caution for data sets with  
>>>> large
>>>> (?) numbers of blocks.  With smaller numbers of blocks I would  
>>>> probably
>>>> try to find some kind of randomization/permutation approach to  
>>>> get a
>>>> sense of the relevant size of delta-AIC values ...
>>>>  ... or damn the torpedoes and see if you can get away with  
>>>> straight
>>>> AIC.
>>>>
>>>> Ben Bolker
>>>>
>>>> Christopher David Desjardins wrote:
>>>>> You could use either the BIC or the AIC. My understanding is  
>>>>> that the
>>>>> AIC tends to favor overly complex models whereas the BIC tends to
>>>>> favor parsimonious models. I am generally inclined to always use  
>>>>> the
>>>>> BIC. If you have a small sample size you might also consider  
>>>>> using the
>>>>> AICC which is a correction of the AIC for small sample sizes. That
>>>>> said, in my experience the AICC still selects more complex  
>>>>> models than
>>>>> the BIC. Also if you have nested models you could use the chi- 
>>>>> square
>>>>> tests.
>>>>> Cheers,
>>>>> Chris
>>>>>
>>>>> On Feb 15, 2009, at 4:44 PM, Tahira Jamil wrote:
>>>>>> Hi
>>>>>> I have run  GLMM models in lme4 with different fixed effects and
>>>>>> random effects . But now the problem is model selction Is AIC  
>>>>>> or BIC
>>>>>> results are definitive specially for Gernalized linear mixed  
>>>>>> models
>>>>>> or what critera should I use for model selction. So I can decide
>>>>>> which explantory variable should be in the model because I have  
>>>>>> more
>>>>>> than 10 explantory variables and some are entering in the model  
>>>>>> as
>>>>>> random effect. In some cases If AIC has lower value but BIC is
>>>>>> comparatively high.
>>>>>>  some suggestion for model selection would be highly appricated.
>>>>>>
>>>>>>  WIth best wishes
>>>>>>  T Jamil
>>>>>>  Ph.D student
>>>>>>  Biometris
>>>>>>  Wageningen University and Research centre Netherlands.
>>>>>>
>>>>>> _______________________________________________
>>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>>
>>>>> -----------------
>>>>> Christopher David Desjardins
>>>>> Ph.D. Student
>>>>> Quantitative Methods in Education
>>>>> Department of Educational Psychology
>>>>> University of  Minnesota
>>>>> http://blog.lib.umn.edu/desja004/educationalpsychology/
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-mixed-models at r-project.org mailing list
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>>>> --
>>>> Ben Bolker
>>>> Associate professor, Biology Dep't, Univ. of Florida
>>>> bolker at ufl.edu / www.zoology.ufl.edu/bolker
>>>> GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>>>>
>>>> The University of Aberdeen is a charity registered in Scotland, No
>>>> SC013683.
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> -- 
> Simon Blomberg, BSc (Hons), PhD, MAppStat.
> Lecturer and Consultant Statistician
> School of Biological Sciences
> The University of Queensland
> St. Lucia Queensland 4072
> Australia
> Room 320 Goddard Building (8)
> T: +61 7 3365 2506
> http://www.uq.edu.au/~uqsblomb
> email: S.Blomberg1_at_uq.edu.au
>
> Policies:
> 1.  I will NOT analyse your data for you.
> 2.  Your deadline is your problem.
>
> The combination of some data and an aching desire for
> an answer does not ensure that a reasonable answer can
> be extracted from a given body of data. - John Tukey.
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models

More information about the R-sig-mixed-models mailing list