[R-sig-eco] p-values from randomization tests

[Kanfra, Xorla]

Dear Philip and Edward, 
Thank you very much for your insight. From the discussion, I guess the P-values are correct and can be used in a publication. If anyone has any suggestions on how to use the many.glm function ex. Selecting alternative adjusted p methods such as FDR, etc., I will gladly appreciate it. 
Kind regards
Xorla

-----Original Message-----
From: R-sig-ecology [mailto:r-sig-ecology-bounces using r-project.org] On Behalf Of Dixon, Philip M [STAT]
Sent: Sunday, June 21, 2020 9:45 PM
To: r-sig-ecology using r-project.org
Subject: [R-sig-eco] p-values from randomization tests

One small revision to Eduard's answer to Korla's question.

When you sample a subset of the possible data configurations under the null hypothesis (what some call a randomization test), the observed data is considered one of the possible samples.  When you generate N (e.g. = 199) random samples, you actually have N+1 (= 200) samples.  So the p-value is (R+1) / (N+1), where R is the number of random samples with test statistics equal to or more extreme than the observed sample.  So in Korla's analysis, when the observed data is more extreme than all the random samples, p = 1/(199+1).  The +1 issue is why you frequently see N=99, N=199, N=999, or even N=9999 random samples.

If you enumerate all possible data configurations under the null (what some call a permutation test), the observed data is automatically included as one sample in the set of possible samples.  Then p = R/N, because the observed value of the test statistic occurs at least once in {R} and that sample is included by definition in {N}.

The spirit of Eduard's answer is spot-on.  The p-values are the same because every pairwise comparison is more extreme than anything in the randomized set.

Best,
Philip Dixon


	[[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
R-sig-ecology using r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology