# [BioC] Minimal group sizes in permutation tests

James W. MacDonald jmacdon at med.umich.edu
Tue Nov 13 15:04:26 CET 2007

```Hi Benjamin,

Benjamin Otto wrote:
> Hey all,
>
> 1. For statistical tests there are usually minimal group sizes recommended
> for appropriate working. For a chi-square test as example the lower level
> was 10 obersvations in each field of the table, if I remember correctly.
> What about permutation tests? Is there some kind of minimal recommendation
> for group sizes? I can't find any hint on that.

The lowest p-value you can get (aside from 0) will be 1/n, where n == #
of combinations. The usual recommendation for any sort of
permutation/sampling test (bootstrap, jackknife, etc) is 500 - 1000
permutations, so you need at least 6 samples per group if you want to

>
> 2. As far as I understand the permutation p-value is given by the quantile
> describing the position of the native p-value in the permutation p-value
> distribution. So for 100 permutations and 5 values smaller than the native
> one the new p-value would be 0.05. What happens when the original p-value is
> the absolut minimum? Is such a thing like p-value equals zero defined?

Sure. If the observed statistic is larger than _all_ the permuted
statistics, then the permuted p-value is 0/n. Of course if your permuted
null is really coarse (say only 20 combinations) then this is less
impressive than if you had much more.

>
> 3. Given a design of 3x3 samples (20 permutations), will the test return
> reasonable values? Doesn't look like it to me.

Well your permuted null will be very coarse. IMO, in this case using
standard normal theory is preferable.

Best,

Jim

>
> Best regards,
>
> Benjamin Otto
>
>
> ======================================
> Benjamin Otto
> University Hospital Hamburg-Eppendorf
> Institute For Clinical Chemistry
> Martinistr. 52
> D-20246 Hamburg
>
> Tel.: +49 40 42803 1908
> Fax.: +49 40 42803 4971
> ======================================
>
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/bioconductor
> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor

--
James W. MacDonald, M.S.
Biostatistician
Affymetrix and cDNA Microarray Core
University of Michigan Cancer Center
1500 E. Medical Center Drive
7410 CCGC
Ann Arbor MI 48109
734-647-5623

```