[R-sig-ME] Fwd: same old question - lme4 and p-values

[Martin Maechler]

>>>>> "Jon" == Jonathan Baron <baron at psych.upenn.edu>
>>>>>     on Sat, 5 Apr 2008 07:21:19 -0400 writes:

    Jon> On 04/05/08 12:10, Reinhold Kliegl wrote:

[...]

    >> In perspective, I think the p-value problem will simply
    >> go away.

    Jon> I'm not sure what you mean here.  If you mean to
    Jon> replace them with confidence intervals, I have no
    Jon> problem with that.  But, as a journal editor, I am
    Jon> afraid that I will continue to insist on some sort of
    Jon> evidence that effects are real.  This can be done in
    Jon> many ways.  But too many authors submit articles in
    Jon> which the claimed effects can result from random
    Jon> variation, either in subjects ("participants*") or
    Jon> items, and they don't correctly reject such alternative
    Jon> explanations of a difference in means.

    Jon> I have noticed a kind of split among those who comment
    Jon> on this issue.  On the one side are those who are
    Jon> familiar with fields such as epidemiology or economics
    Jon> (excluding experimental economics), where the claim is
    Jon> often made that "the null hypothesis is always false
    Jon> anyway, so why bother rejecting it?"  These are the
    Jon> ones interested in effect sizes, variance accounted
    Jon> for, etc.  They are correct for this kind of research,
    Jon> but there are other kinds of research.

    Jon> On the other side, are those from (e.g.) experimental
    Jon> psychology, where the name of the game is to design
    Jon> experiments that are so well controlled that the null
    Jon> hypothesis will be true if the effect of interest is
    Jon> absent.  As a member of this group, when I read people
    Jon> from the first group, I find it very discouraging.  It
    Jon> is almost as if they are saying that what I work so
    Jon> hard to try to do is impossible.

    Jon> To get a little specific, although I found Gelman and
    Jon> Hill's book very helpful on many points (and it does
    Jon> not deny the existence of people like me), it is
    Jon> written largely for members of the first group.  By
    Jon> contrast, Baayen's book is written for people like me,
    Jon> as is the Baayen, Davidson, and Bates article, "Mixed
    Jon> effects modeling with crossed random effects for
    Jon> subjects and items."

    Jon> I'm afraid we do need significance tests, or confidence
    Jon> intervals, or something.

I agree even though I'm very deeply inside the camp of statisticians
who know that all models are wrong but some are useful, and
hence I do not "believe" any P-values (or exact confidence /
credibility intervals).

For those who need ``something like a P-value'' I've heard
yesterday Lorenz Gygax (also subscriber here) proposing
to report the "credibility of 0", possibly "2-sided", as a
pseudo-P value;, i.e. basically that would be
2 * k/n, for an MCMC sample b_1,b_2, ..., b_n 
k := {min k'; b_k' > 0}.
The reasoning would be the following:
Use the 1-to-1 correspondence between confidence intervals and
testing pretending that the credibility intervals are confidence
intervals, and consequently you just need to look at which
confidence level 0 will be at the exact border of the
credibility interval.

Yesterday after the talk, I found that a good idea.
Just now, it seems a bit doubtful, since under the null
hypothesis, I don't think such a pseudo P-value would be uniform
in [0,1].

Martin


    Jon> * On "participants" vs. "subjects" see:
    Jon> http://www.psychologicalscience.org/observer/getArticle.cfm?id=1549

    Jon> -- Jonathan Baron, Professor of Psychology, University
    Jon> of Pennsylvania Home page:
    Jon> http://www.sas.upenn.edu/~baron Editor: Judgment and
    Jon> Decision Making (http://journal.sjdm.org)

    Jon> _______________________________________________
    Jon> R-sig-mixed-models at r-project.org mailing list
    Jon> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models