[R-sig-eco] p-values from randomization tests

[Dixon, Philip M [STAT]]

One small revision to Eduard's answer to Korla's question.

When you sample a subset of the possible data configurations under the null hypothesis (what some call a randomization test), the observed data is considered one of the possible samples.  When you generate N (e.g. = 199) random samples, you actually have N+1 (= 200) samples.  So the p-value is (R+1) / (N+1), where R is the number of random samples with test statistics equal to or more extreme than the observed sample.  So in Korla's analysis, when the observed data is more extreme than all the random samples, p = 1/(199+1).  The +1 issue is why you frequently see N=99, N=199, N=999, or even N=9999 random samples.

If you enumerate all possible data configurations under the null (what some call a permutation test), the observed data is automatically included as one sample in the set of possible samples.  Then p = R/N, because the observed value of the test statistic occurs at least once in {R} and that sample is included by definition in {N}.

The spirit of Eduard's answer is spot-on.  The p-values are the same because every pairwise comparison is more extreme than anything in the randomized set.

Best,
Philip Dixon


	[[alternative HTML version deleted]]