[R-sig-ME] Fwd: same old question - lme4 and p-values

[Robert Kushler]

For a one-sided situation, the posterior probability that
the parameter is on the wrong side of zero makes a lot of
sense.  (Ironically, the classic mistake of interpreting
a p-value as a probability statement about the parameter
is correct here!)

The two-sided case seems more challenging.  The posterior
probability content of an "indifference zone" around zero
can be computed, but it's easy to play games by carefully
picking the width of the zone.  A graph of the probability
against the width of the zone could be a useful exploratory
tool, but would take up a lot of space if used in reporting
results.

Regards,   Rob Kushler



Martin Maechler wrote:
>>>>>> "Jon" == Jonathan Baron <baron at psych.upenn.edu>
>>>>>>     on Sat, 5 Apr 2008 07:21:19 -0400 writes:
> 
>     Jon> On 04/05/08 12:10, Reinhold Kliegl wrote:
> 
> [...]
> 
>     >> In perspective, I think the p-value problem will simply
>     >> go away.
> 
>     Jon> I'm not sure what you mean here.  If you mean to
>     Jon> replace them with confidence intervals, I have no
>     Jon> problem with that.  But, as a journal editor, I am
>     Jon> afraid that I will continue to insist on some sort of
>     Jon> evidence that effects are real.  This can be done in
>     Jon> many ways.  But too many authors submit articles in
>     Jon> which the claimed effects can result from random
>     Jon> variation, either in subjects ("participants*") or
>     Jon> items, and they don't correctly reject such alternative
>     Jon> explanations of a difference in means.
> 
>     Jon> I have noticed a kind of split among those who comment
>     Jon> on this issue.  On the one side are those who are
>     Jon> familiar with fields such as epidemiology or economics
>     Jon> (excluding experimental economics), where the claim is
>     Jon> often made that "the null hypothesis is always false
>     Jon> anyway, so why bother rejecting it?"  These are the
>     Jon> ones interested in effect sizes, variance accounted
>     Jon> for, etc.  They are correct for this kind of research,
>     Jon> but there are other kinds of research.
> 
>     Jon> On the other side, are those from (e.g.) experimental
>     Jon> psychology, where the name of the game is to design
>     Jon> experiments that are so well controlled that the null
>     Jon> hypothesis will be true if the effect of interest is
>     Jon> absent.  As a member of this group, when I read people
>     Jon> from the first group, I find it very discouraging.  It
>     Jon> is almost as if they are saying that what I work so
>     Jon> hard to try to do is impossible.
> 
>     Jon> To get a little specific, although I found Gelman and
>     Jon> Hill's book very helpful on many points (and it does
>     Jon> not deny the existence of people like me), it is
>     Jon> written largely for members of the first group.  By
>     Jon> contrast, Baayen's book is written for people like me,
>     Jon> as is the Baayen, Davidson, and Bates article, "Mixed
>     Jon> effects modeling with crossed random effects for
>     Jon> subjects and items."
> 
>     Jon> I'm afraid we do need significance tests, or confidence
>     Jon> intervals, or something.
> 
> I agree even though I'm very deeply inside the camp of statisticians
> who know that all models are wrong but some are useful, and
> hence I do not "believe" any P-values (or exact confidence /
> credibility intervals).
> 
> For those who need ``something like a P-value'' I've heard
> yesterday Lorenz Gygax (also subscriber here) proposing
> to report the "credibility of 0", possibly "2-sided", as a
> pseudo-P value;, i.e. basically that would be
> 2 * k/n, for an MCMC sample b_1,b_2, ..., b_n 
> k := {min k'; b_k' > 0}.
> The reasoning would be the following:
> Use the 1-to-1 correspondence between confidence intervals and
> testing pretending that the credibility intervals are confidence
> intervals, and consequently you just need to look at which
> confidence level 0 will be at the exact border of the
> credibility interval.
> 
> Yesterday after the talk, I found that a good idea.
> Just now, it seems a bit doubtful, since under the null
> hypothesis, I don't think such a pseudo P-value would be uniform
> in [0,1].
> 
> Martin
> 
> 
>     Jon> * On "participants" vs. "subjects" see:
>     Jon> http://www.psychologicalscience.org/observer/getArticle.cfm?id=1549
> 
>     Jon> -- Jonathan Baron, Professor of Psychology, University
>     Jon> of Pennsylvania Home page:
>     Jon> http://www.sas.upenn.edu/~baron Editor: Judgment and
>     Jon> Decision Making (http://journal.sjdm.org)
> 
>     Jon> _______________________________________________
>     Jon> R-sig-mixed-models at r-project.org mailing list
>     Jon> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> 
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>