[R-sig-ME] lmer and p-values (variable selection)

[Liaw, Andy]

From: John Maindonald
> 
> Yes, the effect size is ultimately more important.  But one needs
> to be somewhat sure that the effect is real, and that it is estimated
> appropriately.  p-values can contribute to a story that gives some 
> smaller or larger confidence that claimed effects are real.  They 
> are just one of several routes that contribute to this end. Opinions 
> differ on whether, in any particular circumstance,  they are the 
> best route.
> 
> The discussion that prompted these various comments related to 
> a different use of p-values (and p-value 'alternatives'), one that is 
> even more controversial.  It related to the use of p-values in 
> excluding or including model explanatory terms.  Here, there are 
> several related issues:
> 
> 1) Inference for model parameters should take account of the
> process that has generated the model that is under consideration.  
> This includes any omission of terms that are judged of no statistical 
> consequence.  The standard interpretations of p-values apply, 
> strictly, only if there has been no  elimination/selection of 
> variables.
> 
> 2) In models that have certain types of imbalance, parameter
> estimates can change markedly (even to changing sign), depending
> on what other terms are in the model.  
> 
> 3) Point 2 argues for choosing the model that is on 
> scientific grounds 
> most reasonable, and sticking with it.  If model parameters are
> important to the subsequent discussion, be sure that their estimates
> condition on the 'correct' other set of model variables, 
> i.e., that the
> other variables that are in the model are the ones that are required
> to allow this interpretation.

I'm afraid that all too often the reason models are chosen on 
"statistical ground" is the lack of "scientific ground".  Sort of 
a catch 22, I guess...  Even when "scientific ground" exists, 
what exactly constitute one, and how do we know it's not 
another rabbit (or ozone) hole?

Andy


 
> 4) One may however allow fine tuning that simplifies the model, while 
> changing nothing of consequence (and it really is necessary to check 
> that there are no changes of consequence).  p-values may have a
> limited use in such fine tuning, but for that purpose the 
> p=0.05 cutoff is 
> not appropriate.
> 
> John Maindonald             email: john.maindonald at anu.edu.au
> phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
> Centre for Mathematics & Its Applications, Room 1194,
> John Dedman Mathematical Sciences Building (Building 27)
> Australian National University, Canberra ACT 0200.
> http://www.maths.anu.edu.au/~johnm
> 
> On 29/03/2011, at 10:35 PM, Manuel Spínola wrote:
> 
> > I am not a statistician, but what the p-value is telling me?
> > 
> > Is not more important the effect size?
> > 
> > Best,
> > 
> > Manuel
> > 
> > On 28/03/2011 04:40 p.m., Ben Bolker wrote:
> >> 
> >> On 03/28/2011 06:15 PM, John Maindonald wrote:
> >> 
> >>> Elimination of a term with a p-value greater than say 
> 0.15 or 0.2 is
> >>> however likely to make little differences to estimates of 
> other terms
> >>> in the model.  Thus, it may be a reasonable way to proceed.  For
> >>> this purpose, an anti-conservative (smaller than it should be)  
> >>> p-value will usually serve the purpose.
> >>   Note that naive likelihood ratio tests of random effects 
> are likely to
> >> be conservative (in the simplest case, true p-values are twice the
> >> nominal value) because of boundary issues and those of 
> fixed effects are
> >> probably anticonservative because of finite-size effects 
> (see PB 2000
> >> for examples of both cases.)
> >> 
> >>> John Maindonald             email: john.maindonald at anu.edu.au
> >>> phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
> >>> Centre for Mathematics & Its Applications, Room 1194,
> >>> John Dedman Mathematical Sciences Building (Building 27)
> >>> Australian National University, Canberra ACT 0200.
> >>> http://www.maths.anu.edu.au/~johnm
> >>> 
> >>   Ben
> >> 
> >> _______________________________________________
> >> R-sig-mixed-models at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >> 
> >> 
> > 
> > 
> > -- 
> > Manuel Spínola, Ph.D. 
> > Instituto Internacional en Conservación y Manejo de Vida Silvestre 
> > Universidad Nacional 
> > Apartado 1350-3000 
> > Heredia 
> > COSTA RICA 
> > mspinola at una.ac.cr 
> > mspinola10 at gmail.com 
> > Teléfono: (506) 2277-3598 
> > Fax: (506) 2237-7036 
> > Personal website: Lobito de río 
> > Institutional website: ICOMVIS
> 
> 
> 	[[alternative HTML version deleted]]
> 
> 
Notice:  This e-mail message, together with any attachme...{{dropped:11}}