A confidence interval around the p-value makes no sense because there is 
no parameter being estimated, but the sampling distribution of the p-value 
makes a lot of sense.  The pre-observational P-value is a random variable 
that is a function of the underlying random variable being tested.  That 
is, P_X(t) = Pr(X>t) is itself a random variable with density, 
distribution, and moments.  Thus, one can compute the 95% sampling 
distribution around the expectation of P.

See 

Hung, H. M. J.; O'Neill, R. T.; Bauer, P. & Kohne, K. The behavior of the 
P-value when the alternative hypothesis is true Biometrics, 1997, 53, 1-22

Donahue, R. M. J. A note on information seldom reported via the p value. 
The American Statistician, American Statistical Association, 1999, 53, 
303-306 

 





Greg Snow <Greg.Snow@imail.org> 
Sent by: r-help-bounces@r-project.org
09/09/2010 12:29 PM

To
"ted.harding@manchester.ac.uk" <ted.harding@manchester.ac.uk>, 
"r-help@r-project.org" <r-help@r-project.org>
cc
Fernando Marmolejo Ramos <fernando.marmolejoramos@adelaide.edu.au>
Subject
Re: [R] confidence intervals around p-values






One other case where a confidence interval on a p-value may make sense is 
permutation (or other resampling) tests.  The population parameter p-value 
would be the p-value that would be obtained from the distribution of all 
possible permutations, but in practice we just sample from that population 
and estimate a p-value.  The confidence interval would then be based on 
the number of sample permutations and could give an idea if that number 
was big enough.  If the full confidence interval is less than alpha then 
you can be confident that the "true" p-value would give significance, if 
it is completely above alpha then it is not significant.  The real problem 
comes when the confidence interval includes alpha, that would indicate 
that B (the number of resamples/permutations) was not large enough.  Be 
careful, doing a small number of permutations then deciding to do more 
based on the CI would likely introduce bias (how much is another 
question).

The nice thing is that in this case the p-value is a simple proportion and 
the confidence interval can be computed using binom.test.

But, I fully agree that in most cases the idea of a CI for a p-value is 
not meaningful, you need to have some case where your p-value is an 
estimate of a "population parameter p-value" that has some meaning.

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow@imail.org
801.408.8111


> -----Original Message-----
> From: r-help-bounces@r-project.org [mailto:r-help-bounces@r-
> project.org] On Behalf Of Ted Harding
> Sent: Thursday, September 09, 2010 8:25 AM
> To: r-help@r-project.org
> Cc: Fernando Marmolejo Ramos
> Subject: Re: [R] confidence intervals around p-values
> 
> On 09-Sep-10 13:21:07, Duncan Murdoch wrote:
> >   On 09/09/2010 6:44 AM, Fernando Marmolejo Ramos wrote:
> >> Dear all
> >>
> >> I wonder if anyone has heard of confidence intervals around
> >> p-values...
> >
> > That doesn't really make sense.  p-values are statistics, not
> > parameters. You would compute a confidence interval around a
> > population mean because that's a parameter, but you wouldn't
> > compute a confidence interval around the sample mean: you've
> > observed it exactly.
> >
> > Duncan Murdoch
> 
> Duncan has succinctly stated the essential point in the standard
> interpretation. The P-value is calculated from the sample in
> hand, a definite null hypothesis, and the distribution of the
> test statistic given the null hyptohesis, so (given all of these)
> there is no scope for any other answer.
> 
> However, there are circumstances in which the notion of "confidence
> interval for a P-value" makes some sense. One such might be the
> Mann-Whitney test for identity of distribution of two samples
> of continuous variables, where (because of discretisation of the
> values when they were recorded) there are ties.
> 
> Then you know in theory that the "underlying values" are all
> different, but because you don't know where these lie in the
> discretisation intervals you don't know which way a tie may
> split. So it would make sense to simulate by splitting ties
> at random (e.g. uniformly distribute each "1.5" value over the
> interval (1.5,1.6) or (1.45,1.55)).
> 
> For each such simulated tie-broken sample, calculate the P-value.
> Then you get a distribution of exact P-values calculated from
> samples without ties which are consistent with the recorded data.
> The central 95% of this distribution could be interpreted as a 95%
> coinfidence interval for the true P-value.
> 
> To bring this closer to on-topic, here is an example in R
> (rounding to intervals of 0.2):
> 
>   set.seed(51324)
>   X <- sort(2*round(0.5*rnorm(12),1))
>   Y <- sort(2*round(0.5*rnorm(12)+0.25,1))
>   rbind(X,Y)
> #   [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
> # X -1.8 -1.2 -0.8 -0.6  0.0    0  0.2  0.2  1.2   1.8     2   2.2
> # Y -1.2 -0.4 -0.2  0.4  0.4    1  1.0  1.0  1.2   1.8     2   2.6
> # So several ties (-1.2,1.2,1.8,2.0), as well as 0.0, 0.4, 1.0
> # which don't matter.
> wilcox.test(X,Y,alternative="less",exact=TRUE,correct=FALSE)
> # data:  X and Y   W = 54, p-value = 0.1488
> 
>   Ps <- numeric(1000)
>   for(i in (1:1000)){
>     Xr <- (X-0.1) + 0.2*runif(10)
>     Yr <- (Y-0.1) + 0.2*runif(10)
>     Ps[i] <- wilcox.test(Xr,Yr,alternative="less",
>              exact=TRUE,correct=FALSE)$p.value
>   }
>   hist(Ps)
>   table(round(Ps,4))
>   # 0.1328 0.1457 0.1593 0.1737 0.1888
>   #     81    267    336    226     90
> 
> So this gives you a picture of the uncertainty in the P-value
> (0.1488, calculated from the rounded data) relative to what it
> really should have been (if calculated from unrounded data).
> Since each possible "true" (tie-broken) sample can be viewed
> as a hypothesis about unobserved "truth", it does make a certain
> sense to view these results as a kind of confidence distribution
> for the P-value you should have got. However, this is more of a
> Bayesian argument, since the above calculation has assigned
> equal prior probability to the tie-breaks!
> 
> One could also, I suppose, consider the question of what
> distribution of P-values might arise if the/an alternative
> huypothesis were true, and where in this does the P-value that
> we actually got lie? But these are murkier waters ...
> 
> Ted.
> 
> --------------------------------------------------------------------
> E-Mail: (Ted Harding) <Ted.Harding@manchester.ac.uk>
> Fax-to-email: +44 (0)870 094 0861
> Date: 09-Sep-10                                       Time: 15:24:29
> ------------------------------ XFMail ------------------------------
> 
> ______________________________________________
> R-help@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-
> guide.html
> and provide commented, minimal, self-contained, reproducible code.

______________________________________________
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide 
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.


	[[alternative HTML version deleted]]

