[R] Partek has Dunn-Sidak Multiple Test Correction. Is this the same/similar to any of R's p.adjust.methods?

[Peter Dalgaard]

"Earl F. Glynn" <efg at stowers-institute.org> writes:

> The Partek package (www.partek.com) allows only two selections for Multiple
> Test Correction:  Bonferroni and Dunn-Sidak.  Can anyone suggest why Partek
> implemented Dunn-Sidak and not the other methods that R has?  Is there any
> particular advantage to the Dunn-Sidak method?
> R knows about these methods (in R 2.1.1):
> 
> > p.adjust.methods
> [1] "holm" "hochberg" "hommel" "bonferroni" "BH" "BY" "fdr"
> [8] "none"
> 
> BH is Benjamini & Hochberg (1995) and is also called "fdr" (I wish R's
> documentation said this clearly).  BY is Benjamini & Yekutieli (2001).
> 
> I found a few hits from Google on Dunn-Sidak, but I'm curious if anyone can
> tell me on a "conservative-liberal" scale, where the Dunn-Sidak method
> falls? My guess is it's less conservative than Bonferroni (but aren't all
> the other methods?), but how does it compare to the other methods?

As far as I gather, D-S is exact for independent tests, conservative
for comparisons of group means, and liberal for mutually exclusive
tests (in which case Bonferroni is exact). It is always less
conservative than Bonferroni, but the difference is small for typical
significance levels: when the Bonferroni level is p, the D-S level is

   1 - (1-p/N)^N

and if you put p=0.05 and vary N you'll find that it varies from 0.05
at N=1 down to 0.04877 at N=100000. (Exercise for the students: what
is the limit as N goes to infinity?)

The three H-methods play a somewhat different game, basically by only
requiring multiple-testing adjustment for non-significant tests.  The
FDR methods play yet differently by allowing the per test level to
increase with the number of significant tests.
 
> A limited numerical experiment suggested this order to me:  bonferroni (most
> conservative), hochberg and holm about the same, BY, BH (also called fdr),
> and then none.
> 
> Thanks for any of  thoughts on this.

I'd expect the differences to be fairly small in scenarios where the
global null hypothesis is true (excluding "none"). The main difference
comes in when some of the nulls are actually false. Also, it depends
on your definitions: With the exception of "BY" and "none" the
p.adjust methods agree on the smallest adjusted p value, so have the
same familywise error rate under the global null. If you count the
total number of rejected tests, then you get a difference due to
"cascading" in the non-bonferroni cases.

-- 
   O__  ---- Peter Dalgaard             Øster Farimagsgade 5, Entr.B
  c/ /'_ --- Dept. of Biostatistics     PO Box 2099, 1014 Cph. K
 (*) \(*) -- University of Copenhagen   Denmark          Ph:  (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)                  FAX: (+45) 35327907