[BioC] decideTests with nestedF

[Gordon Smyth]

At 08:00 PM 16/06/2006, bioconductor-request at stat.math.ethz.ch wrote:
>Date: Fri, 16 Jun 2006 11:30:18 +0200
>From: Pedro L?pez Romero <plopez at cnic.es>
>Subject: [BioC] decideTests with nestedF
>To: <bioconductor at stat.math.ethz.ch>
>Content-Type: text/plain
>
>Dear list,
>
>  I have an experiment with several contrasts associated to different
>experimental conditions for each gene, so I want to select genes in a first
>step by their moderated-F statistic in order to reduce the number of genes
>for multiple contrasts across genes.
>
>For this purpose, I am using decideTests with nestedF, as it is said in
>limma user guide (chapter 10, p 51). However, when I use decideTests with
>method="nestedF", I get quite a few genes that have an adjusted p value in
>their separate t-tests contrast greater than 0.9. These genes have a really
>small M value as to accept that they are really differentially expressed
>between conditions. Does this make sense or I am doing something wrong?.-
>
>The adjusted pvalue that I am talking about is the one that I see after
>applying the topTable function for the corresponding single contrast between
>two experimental conditions. In this case, the adj.P.Val is the same as the
>ones in decideTests with method ="separate", since we are applying the same
>tests.
>
>My code is:
>
>decideTests(fit2,method="nestedF",adjust.method="fdr",p.value=0,05)
>topTable(fit2,coef=i,number=100 adjust="fdr")
>
>Would be the function p.adjust(fit2$F.p.value,method="BH") (as Gordon
>proposed in a recent email about this topic) more appropriate in order to do
>what I what to do?
>
>Any comment will be of great help to me.
>
>Thanks a lot.-
>p.-

Dear Pedro,

Different adjustment methods do give different results. It is as 
simple as that. The phenomenon that you have observed is perfectly 
possible and quite intentional. (Surely this is not surprising. 
Afterall, if different adjustment methods gave the same result, then 
there wouldn't be different methods.)

Please understand that nestedF does not give any formal control of 
FDR at the contrast level. The only control is at the probe level: by 
the above code, you have controlled the expected proportion of genes 
falsely identified to have *any* significant contrasts to be less than 0.05.

If you are seeking control of FDR at the contrast level, you should 
probably use another adjustment method. The methods "global" and 
"separate" can both give control of FDR at the contrast level. The 
strategy which you seem to want to do, using F-tests first and then 
t-tests for the significant genes, with control of FWER or FDR at 
each level, is called method="heirarchical" in limma.

Please understand also that control of FDR behaves differently from 
control of the family-wise error rate (FWER), which is the 
traditional emphasis of multiple testing. The idea of hierarchical 
testing with F-tests before t-tests is far more effective for 
controlling FWER than it is for controlling FDR. It doesn't make much 
difference for FDR.

The real motivation for nestedF is to discover genes which respond 
simultaneously to several contrasts. You will find that the gene 
which is worrying you so much has several significant contrasts. 
nestedF is designed to become less and less rigorous as it works down 
the contrasts for a given gene. As I said before, this is quite 
intentional. It produceds a method which is better at finding 
simultaneously contrasts but worse at finding genes which respond to 
only one contrast.

Hope this helps.
Best wishes
Gordon