[BioC] Minimal group sizes in permutation tests

[James W. MacDonald]

Hi Benjamin,

Benjamin Otto wrote:
> Hey all,
> 
> 1. For statistical tests there are usually minimal group sizes recommended
> for appropriate working. For a chi-square test as example the lower level
> was 10 obersvations in each field of the table, if I remember correctly.
> What about permutation tests? Is there some kind of minimal recommendation
> for group sizes? I can't find any hint on that.

The lowest p-value you can get (aside from 0) will be 1/n, where n == # 
of combinations. The usual recommendation for any sort of 
permutation/sampling test (bootstrap, jackknife, etc) is 500 - 1000 
permutations, so you need at least 6 samples per group if you want to 
follow that recommendation.

> 
> 2. As far as I understand the permutation p-value is given by the quantile
> describing the position of the native p-value in the permutation p-value
> distribution. So for 100 permutations and 5 values smaller than the native
> one the new p-value would be 0.05. What happens when the original p-value is
> the absolut minimum? Is such a thing like p-value equals zero defined?

Sure. If the observed statistic is larger than _all_ the permuted 
statistics, then the permuted p-value is 0/n. Of course if your permuted 
null is really coarse (say only 20 combinations) then this is less 
impressive than if you had much more.

> 
> 3. Given a design of 3x3 samples (20 permutations), will the test return
> reasonable values? Doesn't look like it to me.

Well your permuted null will be very coarse. IMO, in this case using 
standard normal theory is preferable.

Best,

Jim


> 
> Best regards,
> 
> Benjamin Otto
> 
> 
> ======================================
> Benjamin Otto
> University Hospital Hamburg-Eppendorf
> Institute For Clinical Chemistry
> Martinistr. 52
> D-20246 Hamburg
> 
> Tel.: +49 40 42803 1908
> Fax.: +49 40 42803 4971
> ======================================
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Bioconductor mailing list
> Bioconductor at stat.math.ethz.ch
> https://stat.ethz.ch/mailman/listinfo/bioconductor
> Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor

-- 
James W. MacDonald, M.S.
Biostatistician
Affymetrix and cDNA Microarray Core
University of Michigan Cancer Center
1500 E. Medical Center Drive
7410 CCGC
Ann Arbor MI 48109
734-647-5623