[R-sig-ME] lmer and p-values (variable selection)

[Dominick Samperi]

On Tue, Mar 29, 2011 at 7:44 PM, Liaw, Andy <andy_liaw at merck.com> wrote:
> From: John Maindonald
>>
>> Yes, the effect size is ultimately more important.  But one needs
>> to be somewhat sure that the effect is real, and that it is estimated
>> appropriately.  p-values can contribute to a story that gives some
>> smaller or larger confidence that claimed effects are real.  They
>> are just one of several routes that contribute to this end. Opinions
>> differ on whether, in any particular circumstance,  they are the
>> best route.
>>
>> The discussion that prompted these various comments related to
>> a different use of p-values (and p-value 'alternatives'), one that is
>> even more controversial.  It related to the use of p-values in
>> excluding or including model explanatory terms.  Here, there are
>> several related issues:
>>
>> 1) Inference for model parameters should take account of the
>> process that has generated the model that is under consideration.
>> This includes any omission of terms that are judged of no statistical
>> consequence.  The standard interpretations of p-values apply,
>> strictly, only if there has been no  elimination/selection of
>> variables.
>>
>> 2) In models that have certain types of imbalance, parameter
>> estimates can change markedly (even to changing sign), depending
>> on what other terms are in the model.
>>
>> 3) Point 2 argues for choosing the model that is on
>> scientific grounds
>> most reasonable, and sticking with it.  If model parameters are
>> important to the subsequent discussion, be sure that their estimates
>> condition on the 'correct' other set of model variables,
>> i.e., that the
>> other variables that are in the model are the ones that are required
>> to allow this interpretation.
>
> I'm afraid that all too often the reason models are chosen on
> "statistical ground" is the lack of "scientific ground".  Sort of
> a catch 22, I guess...  Even when "scientific ground" exists,
> what exactly constitute one, and how do we know it's not
> another rabbit (or ozone) hole?
>
> Andy

Yes, this is particularly so when studying social systems
or any rapidly evolving system (like the financial markets).
In this situation the statistical picture is often just a
snapshot that should probably be labeled (conditioned)
by the time of observation and the context.

In view of this complexity I'm tempted to view p-values
and hypothesis testing (when used in this context) as
a communication protocol that helps statisticians to
reach a consensus, and not as a tool that reveals
timeless truths.

Dominick

>
>> 4) One may however allow fine tuning that simplifies the model, while
>> changing nothing of consequence (and it really is necessary to check
>> that there are no changes of consequence).  p-values may have a
>> limited use in such fine tuning, but for that purpose the
>> p=0.05 cutoff is
>> not appropriate.
>>
>> John Maindonald             email: john.maindonald at anu.edu.au
>> phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
>> Centre for Mathematics & Its Applications, Room 1194,
>> John Dedman Mathematical Sciences Building (Building 27)
>> Australian National University, Canberra ACT 0200.
>> http://www.maths.anu.edu.au/~johnm
>>
>> On 29/03/2011, at 10:35 PM, Manuel Spínola wrote:
>>
>> > I am not a statistician, but what the p-value is telling me?
>> >
>> > Is not more important the effect size?
>> >
>> > Best,
>> >
>> > Manuel
>> >
>> > On 28/03/2011 04:40 p.m., Ben Bolker wrote:
>> >>
>> >> On 03/28/2011 06:15 PM, John Maindonald wrote:
>> >>
>> >>> Elimination of a term with a p-value greater than say
>> 0.15 or 0.2 is
>> >>> however likely to make little differences to estimates of
>> other terms
>> >>> in the model.  Thus, it may be a reasonable way to proceed.  For
>> >>> this purpose, an anti-conservative (smaller than it should be)
>> >>> p-value will usually serve the purpose.
>> >>   Note that naive likelihood ratio tests of random effects
>> are likely to
>> >> be conservative (in the simplest case, true p-values are twice the
>> >> nominal value) because of boundary issues and those of
>> fixed effects are
>> >> probably anticonservative because of finite-size effects
>> (see PB 2000
>> >> for examples of both cases.)
>> >>
>> >>> John Maindonald             email: john.maindonald at anu.edu.au
>> >>> phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
>> >>> Centre for Mathematics & Its Applications, Room 1194,
>> >>> John Dedman Mathematical Sciences Building (Building 27)
>> >>> Australian National University, Canberra ACT 0200.
>> >>> http://www.maths.anu.edu.au/~johnm
>> >>>
>> >>   Ben
>> >>
>> >> _______________________________________________
>> >> R-sig-mixed-models at r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>> >>
>> >>
>> >
>> >
>> > --
>> > Manuel Spínola, Ph.D.
>> > Instituto Internacional en Conservación y Manejo de Vida Silvestre
>> > Universidad Nacional
>> > Apartado 1350-3000
>> > Heredia
>> > COSTA RICA
>> > mspinola at una.ac.cr
>> > mspinola10 at gmail.com
>> > Teléfono: (506) 2277-3598
>> > Fax: (506) 2237-7036
>> > Personal website: Lobito de río
>> > Institutional website: ICOMVIS
>>
>>
>>       [[alternative HTML version deleted]]
>>
>>
> Notice:  This e-mail message, together with any attachme...{{dropped:11}}
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>