[BioC] limma: decideTests

[Gordon K Smyth]

Dear Al,

No research papers (to my knowledge) have been written yet on multiple 
testing for the hierarchical linear model, so I can't refer you to any 
literature.

Use "separate" if you want to get the same results as topTable().  The 
great advantage of this method is that you'll get the equivalent results 
regardless of which set of contrasts you test together.  This is fine for 
each contrast but does not do any multiple testing adjustment between 
contrasts.  Another disadvantage is that the raw p-value cutoff for any 
adjusted p-value threshold can be very different for different contrasts. 
Use this method if you have only a few contrasts and want to use the 
simplest method.

"global" is the simplest and obvious choice if you want to do multiple 
testing across all the probes and all contrasts simultaneously.  The raw 
p-value cutoff is consistent across all contrasts.  However you have to be 
careful that you do not include spurious contrasts in your test set 
because they will affect the results of the others.  There is no theorem 
which says that "BH" will hold correctly for combinations of negatively 
correlated contrasts.  However simulations suggest that it is pretty safe.

I suggest you avoid "hierarchical" as it is still experimental and its 
properties are not yet well understood with "BH".

Most multiple testing methods tend to underestimate the number of probes 
which are simultaneously significant for two or more contrasts.  In my 
experience, "nestedF" gives good results when you want to focus on probes 
which respond to several different contrasts at once.  However this method 
is virtually undocumentated as I have not found time to write the method 
up properly.  It also provides formal fdr control at the probe level only, 
not at the contrast level.  Do not use it if either of these things 
bothers you.

Hope this helps
Gordon

> Date: Wed, 19 Dec 2007 16:17:47 -0000
> From: "Al Ivens" <alicat at sanger.ac.uk>
> Subject: [BioC] limma: decideTests
> To: "'bioc'" <bioconductor at stat.math.ethz.ch>
> Message-ID: <006501c8425a$b5051c40$881c13ac at internal.sanger.ac.uk>
>
> Hi all,
>
> I am doing some analyses of Affy arrays using limma.  Following the
> manual, I generated the fit2 object using the default eBayes settings,
> and topTable with (almost) default settings.
>
>> dim(eset)
> Features  Samples
>   22625       15
>> fit <- lmFit(eset,designMATRIX)
>> fit2 <- eBayes(contrasts.fit(fit,contrast.matrix))
>> TT <- topTable(fit2,adjust.method="BH",number=length(fit2$genes))
>
> I am now using decideTests, and have read the help that goes with it.
> However, I still can't decide (no pun intended) on what is the best
> method ("separate", "global", "hierarchical", "nestedF") to apply with
> what adjust.method ("none", "BH", "fdr", "BY", "holm"). I have so far
> left adjust.method as "BH", and varied the method.  I can chose one
> (global) that gives me most of the genes I "want", but I'd rather chose
> the method more objectively.  I have 5 contrasts, so am guessing from
> the email archives that nestedF is probably not appropriate for so few
> contrasts?
>
> Are there any general guidelines as to which is the best method to use
> when?
>
> Thanks in anticipation,
>
> a