[R-sig-ME] Fwd: same old question - lme4 and p-values

[Reinhold Kliegl]

Here is a section that worked in Kliegl, Risse, & Laubrock (2007, J
Exp Psychol:Human Perception and Performance, 33, 1250-1251).

"Analysis
     Inferential statistics are based on a linear mixed-effects model
(lme) specifying participants and items as crossed random effects.
This analysis takes into account differences between participants and
differences between items in a single sweep and has been shown to
suffer substantially less loss of statistical power in unbalanced
designs than traditional ANOVAs over participants (F1) and items (F2;
see Baayen, in press, Pinheiro & Bates, 2000; Quené & van den Bergh,
2004, for simulations).
    We used the lmer program (lme4 package; Bates & Sarkar, 2006) in
the R system for statistical computing (R Development Core Team, 2006)
and report regression coefficients (b; absolute effect size in ms),
standard errors (SE), and p-values for an upper-bound n of denominator
degrees of freedom computed as n of observations minus n of fixed
effects. As these p-values are potentially anti-conservative, we
generated confidence intervals from the posterior distribution of
parameter estimates with Markov Chain Monte Carlo methods, using the
mcmcsamp program in the lme4 package with default specifications
(e.g., n=1000 samples; locally uniform priors for fixed effects;
locally non-informative priors for random effects). Both procedures
yielded the same results.
    Finally, we also computed post-hoc power statistics for the
preview and lexical status main effects and for the interaction effect
on first fixation durations (with effect sizes similar to those
reported earlier, e.g., Kliegl, 2007), and using lme estimates of
between-participant, between-item, and residual variances (Gelman &
Hill, in press). For the observed proportion of random loss of items,
power estimates based on 1000 simulations each were around .85 for
word n and n+2 and .59 for word n+1 (due to the higher skipping
rate)."  (page 1251)

Power statistics were included in response to a reviewer request. I am
not much in favor of post-hoc power statistics; but note that here
they are restricted to the use of estimates of random effects. For
reviewers, we also included traditional F1- and F2-ANOVA tables; they
are not part of the article. In other articles, it has also been
acceptable to report coefficients, their standard errors, and their
ratio, and to say that coefficients larger than 2 SE are interpreted
as significant (e.g., Kliegl, 2007, J Exp Psychol: General, 136,
530-537), that is, it is possible to leave out p-values completely.

Corrections and improvements of the above sentences are highly welcome
for future articles. In perspective, I think the p-value problem will
simply go away.

Best
Reinhold

PS: Would it be useful to have a site where peer-reviewed articles
using lme4 for statistical inference are listed and, possibly,
retrievable versions are provided?

On Fri, Apr 4, 2008 at 5:48 PM, Hank Stevens <HStevens at muohio.edu> wrote:
> Google:
>  p-values lmer wiki
>
>
>
>  On Apr 4, 2008, at 9:33 AM, Douglas Bates wrote:
>
>  > ---------- Forwarded message ----------
>  > From: Douglas Bates <bates at stat.wisc.edu>
>  > Date: Fri, Apr 4, 2008 at 7:54 AM
>  > Subject: Re: same old question - lme4 and p-values
>  > To: andreas.nord at zooekol.lu.se
>  >
>  >
>  >
>  > On Fri, Apr 4, 2008 at 5:24 AM,  <andreas.nord at zooekol.lu.se> wrote:
>  >> Dear Prof. Bates,
>  >
>  >> I've recently switched to using R for my analyses, and I find the
>  > lme4 package to be extremely helpful. I have read your explanation
>  > (posted on the mailing list) of why you choose not to display
>  > p-values. Unfortunately, most of the journals I publish in require
>  > that I include p-values, which is why I have to find a way of
>  > calculating them from the lmer output. However, not being a trained
>  > statistician I have some difficulties following your recommendations
>  > given in the explanatory text. In other words, after having fitted my
>  > model, I am not at all sure on what to do in order to obtain p-values
>  > (or similar).
>  >
>  >> I am sorry to have to bother you with a question I know you have
>  > already answered many times, but perhaps you would be so kind as to
>  > give me some hints on how to proceed.
>  >
>  > I understand your situation.  Statisticians have created the "every
>  > question of scientific interest must be answered by a p-value" monster
>  > and now it turns on us.  Nevertheless I am reluctant to give advice on
>  > p-values in lme4 because apparently I don't know how to do it
>  > correctly.
>  >
>  > May I send a copy of this reply to the
>  > R-SIG-Mixed-Models at r-project.org mailing list? ("SIG" == "Special
>  > Interest Group")? (I ask your permission to send the copy because I am
>  > quoting your original question.)   Some who subscribe to that mailing
>  > list may have the courage to wade into this swamp and offer their
>  > advice.
>  >
>  > _______________________________________________
>  > R-sig-mixed-models at r-project.org mailing list
>  > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
>
>
>  Dr. Hank Stevens, Associate Professor
>  338 Pearson Hall
>  Botany Department
>  Miami University
>  Oxford, OH 45056
>
>  Office: (513) 529-4206
>  Lab: (513) 529-4262
>  FAX: (513) 529-4243
>  http://www.cas.muohio.edu/~stevenmh/
>  http://www.cas.muohio.edu/ecology
>  http://www.muohio.edu/botany/
>
>  "If the stars should appear one night in a thousand years, how would men
>  believe and adore." -Ralph Waldo Emerson, writer and philosopher
>  (1803-1882)
>
>
>
>  _______________________________________________
>  R-sig-mixed-models at r-project.org mailing list
>  https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>