[R-sig-ME] P value value for a large number of degree of freedom in lmer

[Emmanuel Charpentier]

Small Bayesian provocation below...

Le mardi 23 novembre 2010 à 16:51 -0800, Joshua Wiley a écrit :
> On Tue, Nov 23, 2010 at 4:09 PM, Jonathan Baron <baron at psych.upenn.edu> wrote:
> > For the record, I have to register my disagreement.  In the
> > experimental sciences, the name of the game is to design a
> > well-controlled experiment, which means that the null hypothesis will
> > be true if the alternative hypothesis is false.  People who say what
> > is below, which includes almost everyone who responded to this post,
> > have something else in mind.  What they say is true in most
> > disciplines.  But when I hear this sort of thing, it is like someone
> > is telling me that my research career as an EXPERIMENTAL psychologist
> > has been some sort of delusion.
> 
> I would not take it that way.  I agree there is a difference between
> some arbitrary null of no difference and a well designed control, but
> no matter what case, the null is a specific hypothesis.  Given a
> continuous distribution, if you the probability of any constant
> occurring to an infinite decimal place is infinitely small.  With only
> 100,000 observations:
> 
> > dt(.49, df = 10^5) - dt(.5, df = 10^5)
> [1] 0.001747051
> 
> Your career as an experimental psychologist is not a delusion, null
> hypothesis statistical testing is---even with a perfect control, we
> set up an unrealistic hypothesis.  Now if we could set up the null as
> an interval....

Assess (Bayes' theorem, whatever the computational process : conjugacy,
explicit computation, MCMC...) the *distribution* of mu_b-mu_a from
prior knowledge (possibly zilch), compute Pr(mu_b-mu_a \in
YourH0Interval) and Pr(mu_b-mu_a \not\in YourH0Interval), and deduce
(Bayes' theorem again) Pr(y|mu_b-mu_a \in YourH0Interval) if you need
something looking like a p-value ; if you want a simpler-interpretaton
one-number summary, compute Bayes' factor.

Easy.

The hard part is to convince your reviewer (and maybe yourself) that
*this* is a valid probabilistic reasoning, that Karl Popper was not God
and that *all*probabilities are conditional.

A harder part is to convince yourself that your choice of distributional
*shapes* (and, more generally, your modelling  choice) is reasonable.
Some variable transformations might help (e. g. use rank(X) rather than
X, postulate t-shaped distributions, etc...), but entails some
non-negligible computational difficulties.

HTH,

					Emmanuel Charpentier


> > If you have a very large sample and you are doing a correlational
> > study, yes, everything will be significant.  But if you do the kind of
> > experiment we struggle to design, with perfect control conditions, you
> > won't get significant results (except by chance) if your hypothesis is
> > wrong.
> 
> I agree that this is typically a bigger problem for correlational
> studies, but if it became practical to run well-controlled experiments
> on millions of participants, I suspect p-values would be disregarded
> awfully quickly.  Even then, the study was not pointless or a
> delusion, that kind of precision lets you confidently talk about the
> actual effect your treatment had compared to your well-designed
> control, and would give any applied person or practitioner a great
> guide what to expect if they implemented it in the field.
> 
> x <- rnorm(10^6, mean = 0)
> y <- rnorm(10^6, mean = .01)
> t.test(x, y, var.equal = TRUE)
> 
> Best regards,
> 
> Josh (fan of experiments, correlational studies, & psychology...not so
> much of NHST, but you use what you have)
> 
> 
> 
> >
> > Jon
> >
> > On 11/24/10 07:59, Rolf Turner wrote:
> >>
> >> It is well known amongst statisticians that having a large enough data set will
> >> result in the rejection of *any* null hypothesis, i.e. will result in a small
> >> p-value.  There is no ``bias'' involved.
> >
> > --
> > Jonathan Baron, Professor of Psychology, University of Pennsylvania
> > Home page: http://www.sas.upenn.edu/~baron
> > Editor: Judgment and Decision Making (http://journal.sjdm.org)
> >
> > _______________________________________________
> > R-sig-mixed-models at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
> 
> 
>