[R] P values

Tue May 11 23:37:02 CEST 2010

Bak,

I think that you are focusing too much on strict definitions rather than on understanding the concepts.  Definitions are important to make sure that we are consistent, but understanding is more important.

The main idea behind p-values and other forms of hypothesis testing is a measure of consistency between a null hypothesis and the observed data.  We don't prove things to be true, we just prove (or gain evidence against) the null hypothesis false.

For example if my null hypothesis is that the coin I am about to flip is 2 headed (no tails) and I flip the coin once and see a head, then the observed data (1 head) and my hypothesis (100% chance of heads) are consistent with each other (p-value=1.00), but does this prove that the coin is 2 headed? No, a single flip resulting in heads is also consistent with the hypothesis that it is a fair coin (50% chance of heads, p-value=0.5).  I have proven that it is not a two-tailed coin (0% chance of heads, p-value=0.0000), but not what it is.  If I flip it 2 more times and get heads each time then my data is still consistent with a hypothesis of 2 heads, but it is still reasonably consistent with the hypothesis of a fair coin (p-value=0.125) and disproves neither.  If I flip a tail then I have disproven the 2 heads hypothesis, but could still be consistent with the fair coin, or a large range of biased coins.  Some people have flipped a coin thousands of times and saw heads close to (but not exactly) 50% of the time.  This does not prove that the coin will come up heads exactly 50% of the times, just shows that the data is consistent with that hypothesis, but there are other values close to 50% that are also consistent with that data.  Small p-values indicate a hypothesis and data that are not very consistent and for small enough p-values we would rather disbelieve the hypothesis (though it may still be theoretically possible).  If I flip a coin 100 times and get 60 heads, then I am unlikely to have been in a situation where heads had a 50% chance (the coin may be biased, or I am not flipping it well, or ...).

For any hypothesis that is consistent with the data, there are a number of other hypotheses that are similar to that hypothesis that we cannot prove inconsistent with the data (the number of these is usually a small infinity).  A linear subset of these forms a confidence interval.  So when I do a test using a specific null hypothesis, I don't necessarily believe that everything matches exactly, just that reality is close enough for me not to care about the differences (the central limit theorem may require infinite sample size to get exact normality, but often it is close enough for much smaller sample sizes).

And small p-values do not measure the goodness of the model, just that the null hypothesis is not consistent with the data.  If you are looking at 2 studies where the 1st has a large sample size and finds a correlation of 0.1 significant with a p-value of 0.000001 and the other finds a correlation of 0.9 with a p-value of 0.01, I would call the 2nd one the stronger relationship even with the higher p-value (though comparing confidence intervals would be even more informative).

-- 
Gregory (Greg) L. Snow Ph.D.
Statistical Data Center
Intermountain Healthcare
greg.snow at imail.org
801.408.8111

> -----Original Message-----
> From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-
> project.org] On Behalf Of Bak Kuss
> Sent: Sunday, May 09, 2010 10:53 AM
> To: murdoch.duncan at gmail.com; jorismeys at gmail.com
> Cc: R-help at r-project.org
> Subject: Re: [R] P values
> 
> Thank you for your replies.
> 
> As I said (wrote) before, 'I am no statistician'.
> But I think I  know what Random Variables are (not).
> 
> Random variables are not random, neither are they variable.
> [It sounds better in french: Une variable alÃ©atoire n'est pas
> variable,
> et n'a rien d'alÃ©atoire.]
> 
> See this definition from: 'Introduction to the mathematical and
> statistical
> foundations of econometrics',
> Herman J. Bierens, Cambridge University Press, 2004, page 21.
> 
> http://docs.google.com/View?id=dct7h449_8748tjc6g9
> 
> 
> Simply put: A random variable is just a mapping (transformation) from a
> set
> to the real line.
> If the mapping is limited to be from a set to the  [0,1] segment of the
> real
> line,
> one calls it a probability. But it is still not variable nor random.
> Just a 'simple transformation'.
> 
> 
> As far as Central Limit Theorems are concerned,
> are they not...  well, far from reality.
> They belong to asymptotics. By definition 'asymptotics' do not belong
> to reality.  'As if...' kind of arguments they are.
> Are they not excuses for our 'misbehavior'? An alibi?
> Just like 'p-values'? They just _indicate_  that  _probably_
> we were wrong in having thought such and such...
> Without ever getting close to whatever the 'real reality' was, is,
> could
> be... probably!
> 
> bak
> 
> 
> 
> 
> 
> That's a common misconception. A p-value expresses no more than the
> chance
> of obtaining the dataset you observe, given that your null hypothesis
> _and
> your assumptions_ are true. Essentially, a p-value is as "real" as your
> assumptions. In that way I can understand what Robert wants to say. But
> with
> lare enough datasets, bootstrapping or permutation tests gives often
> about
> the same p-value as the asymptotic approximation. *At that moment, the
> central limit theorem comes into play*
> 
> On Sat, May 8, 2010 at 9:38 PM, Duncan Murdoch
> <murdoch.duncan at gmail.com>wrote:
> 
> > On 08/05/2010 9:14 PM, Joris Meys wrote:
> >
> >> On Sat, May 8, 2010 at 7:02 PM, Bak Kuss <bakkuss at gmail.com> wrote:
> >>
> >>
> >>
> >>> Just wondering.
> >>>
> >>> The smallest the p-value, the closer  to 'reality'  (the more
> accurate)
> >>> the model is supposed to (not) be (?).
> >>>
> >>> How realistic is it to be that (un-) real?
> >>>
> >>>
> >>>
> >>
> >> That's a common misconception. A p-value expresses no more than the
> chance
> >> of obtaining the dataset you observe, given that your null
> hypothesis _and
> >> your assumptions_ are true.
> >>
> >
> >
> > I'd say it expresses even less than that.  A p-value is simply a
> > transformation of the test statistic to a standard scale.  In the
> nicer
> > situations, if the null hypothesis is true, it'll have a uniform
> > distribution on [0,1].  If H0 is false but the truth lies in the
> direction
> > of the alternative hypothesis, the p-value should have a distribution
> that
> > usually gives smaller values.  So an unusually small value is a sign
> that H0
> > is false:  you don't see values like 1e-6 from a U(0,1) distribution
> very
> > often, but that could be a common outcome under the alternative
> hypothesis.
> >   (The not so nice situations make things a bit more complicated,
> because
> > the p-value might have a discrete distribution, or a distribution
> that tends
> > towards large values, or the U(0,1) null distribution might be a
> limiting
> > approximation.)
> > So to answer Bak, the answer is that yes, a well-designed statistic
> will
> > give p-values that tend to be smaller the further the true model gets
> from
> > the hypothesized one, i.e. smaller p-values are probably associated
> with
> > larger departures from the null.  But the p-value is not a good way
> to
> > estimate that distance.  Use a parameter estimate instead.
> >
> > Duncan Murdoch
> >
> >
> >
> >  Essentially, a p-value is as "real" as your
> >> assumptions. In that way I can understand what Robert wants to say.
> But
> >> with
> >> lare enough datasets, bootstrapping or permutation tests gives often
> about
> >> the same p-value as the asymptotic approximation. At that moment,
> the
> >> central limit theorem comes into play, which says that when the
> sample
> >> size
> >> is big enough, the mean is -close to- normally distributed. In those
> >> cases,
> >> the test statistic also follows the proposed distribution and your
> p-value
> >> is closer to "reality". Mind you, the "sample size" for a specific
> >> statistic
> >> is not always merely the number of observations, especially in more
> >> advanced
> >> methods. Plus, violations of other assumptions, like independence of
> the
> >> observations, changes the picture again.
> >>
> >> The point is : what is reality? As Duncan said, a small p-value
> indicates
> >> that your null hypothesis is not true. That's exactly what you look
> for,
> >> because that is the proof the relation in your dataset you're
> looking at,
> >> did not emerge merely by chance. You're not out to calculate the
> exact
> >> chance. Robert is right, reporting an exact p-value of 1.23 e-7
> doesn't
> >> make
> >> sense at all. But the rejection of your null-hypothesis is as real
> as
> >> life.
> >>
> >> The trick is to test the correct null hypothesis, and that's were it
> most
> >> often goes wrong...
> >>
> >> Cheers
> >> Joris
> >>
> >>
> >>
> >>> bak
> >>>
> >>> p.s. I am no statistician
> >>>
> >>>       [[alternative HTML version deleted]]
> >>>
> >>> ______________________________________________
> >>> R-help at r-project.org mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-help
> >>> PLEASE do read the posting guide
> >>> http://www.R-project.org/posting-guide.html
> >>> and provide commented, minimal, self-contained, reproducible code.
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >
> >
> 
> 	[[alternative HTML version deleted]]