[R-sig-ME] model selection in lme4

Mon Feb 16 06:53:20 CET 2009

> In that case I might bring up B&A's "tapering effects" argument
> again -- selecting the correct model with a fixed number of parameters
> with non-tapering effects is what BIC is for, not what AIC is for.

I think this may be the case. The data that I have used is real data not 
simulated. However, I can tell that the parameters were unnecessary as they 
didn't explain any variation above and beyond the more simpler models. I 
think it may also be a case of the type of research questions that I ask in 
psychology vs. ecology.

I'm always interested in knowing more about BIC/AIC and I'll check out your 
references.
Thanks!
Chris

On Sunday 15 February 2009 21:02:42 Ben Bolker wrote:
> Took a (very) quick look at Raftery, which all seems sensible
> and well-argued.  However ... the paper contrasts Bayes/BIC
> with classical hypothesis testing.  Many of the points listed on p. 155
> (better assessment of evidence, applicability to non-nested models, take
> model uncertainty into account, allow model averaging, easy to
> implement) apply to AIC as well as BIC.  BIC does have many good
> qualities (approximation to Bayes factor, sensible "flat prior"
> interpretation, statistical consistency, ...).  But the crux of the
> argument between BIC and AIC is the difference in their objective. BIC
> aims to identify the "true model", which essentially assumes that there
> is a sharp cutoff between parameters/processes that are in the model and
> those that are out. Burnham and Anderson have a lot to say about
> tapering effect sizes; they are zealots about AIC, and I often discount
> their enthusiasm, but after much percolation I've decided that AIC
> really does make sense for the kinds of questions I (and many
> ecologists) tend to ask.
>
>    When you say that AIC selects an overly complex model, how
> do you know what the correct model is and which parameters are
> unnecessary?  Is this a case of fitting to simulation output?

>   I have tried to say this more coherently at
> http://emdbolker.wikidot.com/blog:aic-vs-bic
>
>   As an aside, I don't have a vested interest in this, and I don't
> claim that AIC is better for everything ... just that it seems
> most ecologists are working with "true models" that are of
> arbitrarily large dimension with tapering effects, which is where
> AIC should select the model with the best predictive capability ...
>
>   Ben Bolker
>
> Christopher David Desjardins wrote:
> > For a discussion of BIC, please see Raftery (1995) in Sociological
> > Methodology. Before you commit yourself on the AIC, I do encourage you to
> > look at your BIC. In the models I've run when there is disagreement
> > between the BIC and the AIC, it's usually that the AIC selects the overly
> > complex model and includes unnecessary parameters.
> > Cheers,
> > Chris
> >
> > On Sunday 15 February 2009 19:50:30 Ben Bolker wrote:
> >>   It would be better to use AICc, but I'm not sure what I would
> >> use for "number of parameters" for a random effect with n
> >> levels: any number between 0.5 and n seems plausible!
> >> Someone should send Shane Richards (who has done some
> >> very nice work testing (Q)AIC(c) in ecological settings)
> >> and see if he's willing to tackle this one, although I can
> >> imagine he's getting sick of this kind of exercise ...
> >>
> >>   Ben Bolker
> >>
> >> Renwick, A. R. wrote:
> >>> Just a quickie Ben,
> >>> Are you saying that you would use AIC rather than AICc even with
> >>> small sample size - due to difficulty in counting residual degrees of
> >>
> >> freedom?
> >>
> >>> Thanks
> >>> Anna
> >>> p.s. this forum really is fantastic
> >>>
> >>> ________________________________________
> >>> From: r-sig-mixed-models-bounces at r-project.org
> >>> [r-sig-mixed-models-bounces at r-project.org] On Behalf Of Ben Bolker
> >>> [bolker at ufl.edu] Sent: 15 February 2009 23:07
> >>> To: Christopher David Desjardins
> >>> Cc: r-sig-mixed-models at r-project.org; tahirajamil at yahoo.com
> >>> Subject: Re: [R-sig-ME] model selection in lme4
> >>>
> >>>   Some caution on this advice: you seem to be quoting
> >>> the general advice on AIC/BIC/AICc
> >>>
> >>>   1. The AIC/BIC distinction is between "best prediction"
> >>> and "consistent estimation of true model" dimension, e.g.
> >>>
> >>> Yang, Yuhong. 2005. Can the strengths of AIC and BIC be shared? A
> >>> conflict between model identification and regression estimation.
> >>> Biometrika 92, no. 4 (December 1): 937-950.
> >>> doi:10.1093/biomet/92.4.937.
> >>>
> >>>   I favor AIC on these grounds, but you can decide for yourself.
> >>>
> >>>   2. For models with different random effects, AIC and BIC share
> >>> a "degrees of freedom counting" problem with all model selection
> >>> approaches -- there are two aspects here, (1) whether you are
> >>> focused on individual-level prediction or population-level
> >>> prediction (Vaida and Blanchard 2005, Spiegelhalter et al 2002)
> >>> and (2) whether AIC/BIC share the boundary problems that
> >>> also apply to likelihood ratio tests (Greven, Sonja. 2008. Non-Standard
> >>> Problems in Inference for Additive and Linear Mixed Models. Göttingen,
> >>> Germany: Cuvillier Verlag.
> >>> http://www.cuvillier.de/flycms/en/html/30/-UickI3zKPS,3cEY=/Buchdetails
> >>>.h tml?SID=wVZnpL8f0fbc. )
> >>>
> >>>   3. AIC and BIC are asymptotic tests (which can be especially
> >>> problematic with random effects problems, when there are not
> >>> large number of random blocks -- this makes likelihood ratio
> >>> tests NOT OK for fixed-effect comparisons with small numbers
> >>> of blocks (Pinheiro and Bates 2000)).  If you want to use
> >>> AICc then you are back to counting residual degrees of freedom ...
> >>> as far as I know there isn't much guidance available on this
> >>> issue.
> >>>
> >>>   My bottom line:
> >>>
> >>>   I would go ahead and use (Q)AIC with caution for data sets with large
> >>> (?) numbers of blocks.  With smaller numbers of blocks I would probably
> >>> try to find some kind of randomization/permutation approach to get a
> >>> sense of the relevant size of delta-AIC values ...
> >>>    ... or damn the torpedoes and see if you can get away with straight
> >>> AIC.
> >>>
> >>>   Ben Bolker
> >>>
> >>> Christopher David Desjardins wrote:
> >>>> You could use either the BIC or the AIC. My understanding is that the
> >>>> AIC tends to favor overly complex models whereas the BIC tends to
> >>>> favor parsimonious models. I am generally inclined to always use the
> >>>> BIC. If you have a small sample size you might also consider using the
> >>>> AICC which is a correction of the AIC for small sample sizes. That
> >>>> said, in my experience the AICC still selects more complex models than
> >>>> the BIC. Also if you have nested models you could use the chi-square
> >>>> tests.
> >>>> Cheers,
> >>>> Chris
> >>>>
> >>>> On Feb 15, 2009, at 4:44 PM, Tahira Jamil wrote:
> >>>>> Hi
> >>>>> I have run  GLMM models in lme4 with different fixed effects and
> >>>>> random effects . But now the problem is model selction Is AIC or BIC
> >>>>> results are definitive specially for Gernalized linear mixed models
> >>>>> or what critera should I use for model selction. So I can decide
> >>>>> which explantory variable should be in the model because I have more
> >>>>> than 10 explantory variables and some are entering in the model as
> >>>>> random effect. In some cases If AIC has lower value but BIC is
> >>>>> comparatively high.
> >>>>>    some suggestion for model selection would be highly appricated.
> >>>>>
> >>>>>    WIth best wishes
> >>>>>    T Jamil
> >>>>>    Ph.D student
> >>>>>    Biometris
> >>>>>    Wageningen University and Research centre Netherlands.
> >>>>>
> >>>>> _______________________________________________
> >>>>> R-sig-mixed-models at r-project.org mailing list
> >>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >>>>
> >>>> -----------------
> >>>> Christopher David Desjardins
> >>>> Ph.D. Student
> >>>> Quantitative Methods in Education
> >>>> Department of Educational Psychology
> >>>> University of  Minnesota
> >>>> http://blog.lib.umn.edu/desja004/educationalpsychology/
> >>>>
> >>>> _______________________________________________
> >>>> R-sig-mixed-models at r-project.org mailing list
> >>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >>>
> >>> --
> >>> Ben Bolker
> >>> Associate professor, Biology Dep't, Univ. of Florida
> >>> bolker at ufl.edu / www.zoology.ufl.edu/bolker
> >>> GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc
> >>>
> >>> _______________________________________________
> >>> R-sig-mixed-models at r-project.org mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >>>
> >>>
> >>> The University of Aberdeen is a charity registered in Scotland, No
> >>> SC013683.