[BioC] Why NONE of any contrast is significant when overall F-test is significant? using "global" method in decideTests() from limma package

[Guiyuan Lei]

Thank you, Jim.

I also tried "separate" and "nestedF". For "separate" method, there
are also genes (around 8% of 397 significantly differential genes by
F-test)  got five '0's. As for "nestedF", every gene got at  least one
non-zero value.

In the post https://stat.ethz.ch/pipermail/bioconductor/2007-April/016698.html

Gordon said
'method="separate" and method="nestedF" do quite different things.
"separate" controls the FDR on a per-contrast basis only.  It does not
control the FDR globally across all contrasts. '

Why "separate" method also generated five "0"s (5 contrasts) for
significantly differential genes by F-test?

Best regards,
Guiyuan

On Thu, Feb 28, 2008 at 3:54 PM, James W. MacDonald
<jmacdon at med.umich.edu> wrote:
> Hi Guiyan,
>
>
>  Guiyuan Lei wrote:
>  > Dear all,
>  >
>  > I am using limma package to identify differential expression. I have 5
>  > contrasts, I used F-statistic to measure significant differential
>  > expression,  the F-test p-value is adjusted by  "fdr" method . I used
>  > the command
>  >
>  > p.adjust(eb$F.p.value, method="fdr") < 0.05
>  >
>  > where eb is the object from eBayes(). I got 397 significant
>  > differential expression.
>  >
>  > Then for those 397 significantly differential expressed genes, I want
>  > to look at each contrast to check which contrast is significant. I
>  > used
>  >
>  > decideTests(eb, method="global")
>  >
>  > to classify each t-statistics as up, down or not significant.
>  >
>  > I found that among genes which are significantly expressed by F-test
>  > p-value, that is, the above 397 genes, some have got five '0's from
>  > decideTests, which means those genes are not significant for any of
>  > its contrast while its F-test is significant. As I understand, if
>  > F-test is significant, should at least one of its contrast (t-test) is
>  > significant. I doubt that the "global" method used in decideTests() is
>  > not properly used for this case? But why? Can any one explain this to
>  > me? Many thanks! I attach the code and results as following.
>
>  Yes. From ?decideTests:
>
>  'method="global"' will treat the entire matrix of t-statistics as
>       a single vector of unrelated tests.
>
>  So when you did the F-test and adjusted for multiplicity, you adjusted
>  for n tests. Then when you did the t-tests, you did the same, but
>  adjusting for 5n tests. Increasing the number of simultaneous tests
>  five-fold was enough to cause your multiplicity-adjusted p-values to all
>  become insignificant at an alpha of 0.05.
>
>  You might try method="separate" or "nestedF".
>
>  Best,
>
>  Jim
>
>
>
>
>
>
>  >
>  > library(limma)
>  > levels= c('h','h','h','hh','hh','mh','mh','s','s','s','y','y','y','y')
>  > dimnames(eset.matrix)[[2]]= levels
>  > TS <- factor(levels, levels= c('h','hh','mh','s','y'))
>  > design <- model.matrix(~0+TS)
>  > colnames(design) <- levels(TS)
>  > fit <- lmFit(eset.gcrma, design)
>  >
>  > #Construct the contrasts
>  > mc <- makeContrasts('s-y','h-y','mh-y','hh-h','mh-hh',levels=design)
>  > fit2 <- contrasts.fit(fit, mc)
>  > eb <- eBayes(fit2)
>  >
>  > #The adjustment methods using "fdr"
>  > modFpvalue <- eb$F.p.value
>  > selectedgenesindx <- p.adjust(eb$F.p.value, method="fdr") < 0.05
>  > Sig<-modFpvalue[selectedgenesindx]
>  > nsiggenes<-length(Sig) #number of significantly differential expression
>  >
>  > #decideTests using "global" method
>  > results1 <- decideTests(eb, method="global")
>  >
>  > #Order the F-statistic
>  > modF <- eb$F
>  > modFordered<-order(modF, decreasing = TRUE)
>  >
>  > #Get the up or down or not significant for nsiggenes significant
>  > (according adjusted p-value of F-test) differential expression
>  > updown<-results1[modFordered[1:nsiggenes],]
>  >
>  > Examples for genes which are significant expression by F-test
>  > F-test           results from decideTests()
>  > eb$F.p.value  contrast1  contrast2 contrast3  contrast4  contrast4
>  >  5.6E-05       0             0                   0             0             0
>  > (this one is not significant for any of its contrast, why five '0's?)
>  > 0.00036          0          -1                   0             0             0
>  >
>  >
>  > Best regards,
>  > Guiyuan
>  >
>  > _______________________________________________
>  > Bioconductor mailing list
>  > Bioconductor at stat.math.ethz.ch
>  > https://stat.ethz.ch/mailman/listinfo/bioconductor
>  > Search the archives: http://news.gmane.org/gmane.science.biology.informatics.conductor
>
>  --
>  James W. MacDonald, M.S.
>  Biostatistician
>  Affymetrix and cDNA Microarray Core
>  University of Michigan Cancer Center
>  1500 E. Medical Center Drive
>  7410 CCGC
>  Ann Arbor MI 48109
>  734-647-5623
>