[BioC] Multiple test question in micrarray- FDR

[Naomi Altman]

I want to reiterate - a q-value is related to FDR.  This is the 
expected percentage of false positives in your gene list if you 
reject at this p-value or less.  It is not the statistical 
significance of any gene on your gene list.
There is however, a Bayesian interpretation that says it is the 
posterior probability that the gene really differentially 
expressed.  If you are Bayesian, you will always update your 
posterior with any new information that arises, such as the results 
of your PCR study.

--Naomi

At 02:38 PM 12/14/2008, Wayne Xu wrote:
>Dear Naomi,
>I may have a silly question. I read a few papers on microarray 
>multiple test, I understood what points they were trying to make. 
>But I still have doubts about it. Since now many journal reviewers 
>require the FDR for microarray differential expresses genes in 
>manuscripts, I really want to clear my doubts.
>
>1). The mathematics model is different from the biology model:
>The typical math model to bring up the multiple test issue is 
>following example: 20 balls in a box with 1 in red and 19 in blue. 
>The possibility of picking up the red ball from the box each time is 
>1/20, i.e 0.05. If draw 20 times, the chance is 0.05 multiplied by 20 is 1.
>Suppose the red represents false positive, if draw one time the FDR 
>is 0.05, if 20 times then FDR is 1. People bring this multiple test 
>issue into microarray data analysis. But in microarray, at least two 
>aspects are different from this math model:
>a). The raw P values are determined by the expression values of 
>samples, not affected by the total number of genes.  So it is 
>different from above example of 1 out of 20 is 0.05.
>b). Pick up a ball and then put it back to the box, you have chance 
>to pick up the exactly same ball twice or more. But in microarray, 
>each genes are tested individually at the same time, and each gene 
>only tested exactly once.
>They are obviously different. If this math model is the only reason 
>that brought up the multiple test issue in microarray, it may be a 
>misleading (I may be silly, since no one else doubts about multiple 
>test in microarray?)
>
>2). Not make biological sense:
>Suppose a gene called XYZ has a raw P value of 0.00001 in two group 
>T test, and it was validated by biological test, e.g. RT-PCR. If the 
>micoarray chip has 40,000 genes, then by whatever adjustment  FDR 
>method, the adj P-value may be 0.4 or lower or higher. If I use FDR 
>cutoff 0.1, this XYZ gene has higher FDR and is not in my interest 
>positive gene list.
>OK, now I play a math game, filter gene by variance or other, shrink 
>the gene list to 5000 (since XYZ gene has low P value, suppose it is 
>within the 5000). Then the XYZ has low FDR and in my interest 
>differential gene list. But this is just a math game!
>The biological reality is XYZ is positive, this positive is 
>determined by, for example 4 control samples and 4 treatment 
>samples, the mean may be big different, and within group variance is 
>very small. and RT-PCR validated. This reality can not be changed by 
>whatever number of genes to be tested. The raw P value is close the 
>biological reality, and it is good to represent the biological 
>reality. The multiple test here just make you feel happier but not a 
>biological sense.
>
>FDR is a very useful term in many biological cases.  But it seems 
>not a good example here for microarray?
>
>Please help to clear it up.
>
>Thank you,
>
>Wayne
>--
>
>
>Naomi Altman wrote:
>>Remember that FDR is a rate - i.e. the expected false discovery rate.
>>If the set of genes is changeds, FDR will change because the 
>>comparison set is different.  This is NOT the same as a p-value, 
>>which depends only on the value of the current test statistic.
>>
>>The same thing happens with FWER, because these methods control the 
>>probability of making at least one mistake, which clearly depends 
>>on which set of tests are performed.
>>
>>--Naomi
>>
>>At 03:11 PM 12/13/2008, Sean Davis wrote:
>>>On Sat, Dec 13, 2008 at 12:36 PM, Wayne Xu <wxu at msi.umn.edu> wrote:
>>> > Hello,
>>> > I am not sure this is a right place to ask this question, but it is about
>>> > micrarray data analysis:
>>> >
>>> > In two group t test, the multiple test Q values are depending 
>>> on the total
>>> > number of genes in the test. If I filter the gene list first, 
>>> for example, I
>>> > only use those genes that have1.2 fold changes for T test and 
>>> multiple test,
>>> > this gene list is much smaller than the total gene list, then 
>>> the multiple
>>> > test q values are much smaller.
>>> >
>>> > Do you think above is a correct way? People who do not do that way may
>>> > consider the statistical power may be lost? But how much power 
>>> lost and how
>>> > to calculate the power in this case?
>>>
>>>No, you cannot filter based on fold change.  However, you can filter
>>>based on variance or some other measure that does not depend on the
>>>two groups being compared.  Anything that filters genes based on
>>>"knowing" the two groups will lead to a biased test.  Remember that
>>>filtering removes genes from consideration from further analysis.
>>>
>>>For further details, there are MANY discussions of this topic in the
>>>mailing list.
>>>
>>> > When people report multiple test Q values, they usually do not 
>>> mention how
>>> > many genes are used in this multiple test. You can get different Q values
>>> > (even use the same method, e.g. Benjamin and Holm adjust 
>>> method) in the same
>>> > dataset. Then how can it make sense if the same genes have different Q
>>> > values?
>>>
>>>A good manuscript should describe in detail the preprocessing and
>>>filtering steps, the statistical tests used, and the methods for
>>>correcting for multiple testing.  You are correct that many papers do
>>>not do so.
>>>
>>>As for different q-values in the same dataset using different methods,
>>>it is important to note that one should not do an analysis, get a
>>>result, and then, based on that result, go back and redo the analysis
>>>with different parameters to get a "better" result.  It is very
>>>important that each step of an analysis (preprocessing, filtering,
>>>testing, multiple-testing correction) be justifiable independent of
>>>the other steps in order for the results to be interpretable.
>>>
>>>Sean
>>>
>>>_______________________________________________
>>>Bioconductor mailing list
>>>Bioconductor at stat.math.ethz.ch
>>>https://stat.ethz.ch/mailman/listinfo/bioconductor
>>>Search the archives: 
>>>http://news.gmane.org/gmane.science.biology.informatics.conductor
>>
>>Naomi S. Altman                                814-865-3791 (voice)
>>Associate Professor
>>Dept. of Statistics                              814-863-7114 (fax)
>>Penn State University                         814-865-1348 (Statistics)
>>University Park, PA 16802-2111
>
>_______________________________________________
>Bioconductor mailing list
>Bioconductor at stat.math.ethz.ch
>https://stat.ethz.ch/mailman/listinfo/bioconductor
>Search the archives: 
>http://news.gmane.org/gmane.science.biology.informatics.conductor

Naomi S. Altman                                814-865-3791 (voice)
Associate Professor
Dept. of Statistics                              814-863-7114 (fax)
Penn State University                         814-865-1348 (Statistics)
University Park, PA 16802-2111