[BioC] Invalid fold-filter

Naomi Altman naomi at stat.psu.edu
Tue Feb 21 17:49:11 CET 2006


I think it is unwise to filter based on observed data.  This biases 
the results.

On the other hand, filtering on a priori considerations, such as lack 
of annotation, should not be a problem.

--Naomi

At 12:19 AM 2/21/2006, Robert Gentleman wrote:


>Bornman, Daniel M wrote:
> > Robert,
> >
> > After reading your response to my initial question, I do not believe you
> > addressed exactly what I attempted to describe.  Please pardon me for
> > not being clear.  I think your response assumed I was filtering on
> > unadjusted p-values then applying a correction such as Benjamini &
> > Hochberg to a reduced set.
> >
> > My question was rather on the validity of first filtering each gene
> > based on fold-change between two sample groups (i.e. controls vs
> > treated) then calculating a test-statistic, raw p-value and corrected
> > p-value on each gene that passed the fold-change filter.  I am worried
> > that using the group phenotype description to filter followed by
> > applying a p-value correction is unfairly reducing my multiple
> > comparison penalty.
>
>   It does, and it does not matter how you get there, using one test
>(fold change) to filter and a different test (t-test) for p-value
>correction does not really change the fact that if both tests make use
>of the same way to define samples, then there are problems with the
>interpretation.
>
> >
> > I propose that a less biased approach to fold-filtering would be to
> > filter probes based on the mean of the lower half versus the mean of the
> > upper half of expression values at each probe regardless of the
> > phenotype (non-specific).  This would surely (except in some instances
> > where a phenotype causes drastic expression changes) cause the
> > fold-filtered set to be larger and thus not unfairly decrease the
> > multiple comparison penalty when computing adjusted p-values.
> >
>
>    Well that is one thing, but really IMHO, you are better off filtering
>on variance than some rather arbitrary division into two groups (why 1/2
>- many of the classification problems I deal with are very unbalanced
>and 1/2 would be a pretty bad choice). And, it is variation that is
>important (whence ANOVA - ANalysis Of VAriance).
>
>   Best wishes,
>     Robert
>
> >
> > Thank You,
> > Daniel
> >
> >
> >
> > -----Original Message-----
> > From: Robert Gentleman [mailto:rgentlem at fhcrc.org]
> > Sent: Saturday, February 18, 2006 12:57 PM
> > To: Bornman, Daniel M
> > Subject: Re: [BioC] Invalid fold-filter
> >
> > Hi Daniel,
> >   I hope not, it is as you have noted a flawed approach.
> >
> > best wishes
> >    Robert
> >
> > Bornman, Daniel M wrote:
> >
> >>I of course agree that filtering on a variable (phenotype) that will
> >>be used later to calculate adjusted p-values is flawed and therefore
> >>it is not a method I would implement; however, it seems that many that
> >
> >
> >>describe fold-filtering are doing just that.
> >>Thank you for your response.
> >>
> >>-----Original Message-----
> >>From: Robert Gentleman [mailto:rgentlem at fhcrc.org]
> >>Sent: Friday, February 17, 2006 2:15 PM
> >>To: Bornman, Daniel M
> >>Cc: bioconductor at stat.math.ethz.ch
> >>Subject: Re: [BioC] Invalid fold-filter
> >>
> >>
> >>
> >>Bornman, Daniel M wrote:
> >>
> >>
> >>>Dear BioC Folks,
> >>>
> >>>As a bioinformatician within a Statistics department I often consult
> >>>with real statisticians about the most appropriate test to apply to
> >>>our microarray experiments.  One issue that is being debated among our
> >>
> >>
> >>>statisticians is whether some types of fold-filtering may be invalid
> >>>or biased in nature.  The types of fold-filtering in question are
> >>>those that tend to NOT be non-specific.
> >>>Some filtering of a 54K probe affy chip is useful prior to making
> >>>decisions on differential expression and there are many examples in
> >>>the Bioconductor documentation (particularly in the {genefilter}
> >>>package) on how to do so.  A popular method of non-specific filtering
> >>>for reducing your probeset prior to applying statistics is to filter
> >>>out low expressed probes followed by filtering out probes that do not
> >>>show a minimum difference between quartiles.  These two steps are
> >>>non-specific in that they do not take into consideration the actual
> >>
> >>samples/arrays.
> >>
> >>
> >>>On the other hand, if we had two groups of samples, say control versus
> >>
> >>
> >>>treated, and we filtered out those probes that do not have a mean
> >>>difference in expression of 2-fold between the control and treated
> >>>groups, this filtering was based on the actual samples.  This is NOT a
> >>
> >>
> >>>non-specific filter.  The problem then comes (or rather the debate
> >>>here
> >>>arises) when a t-test is calculated for each probe that passed the
> >>>sample-specific fold-filtering and the p-values are adjusted for
> >>>multiple comparisons by, for example the Benjamini & Hochberg method.
> >>>Is it valid to fold-filter using the sample identity as a criteria
> >>>followed by correcting for multiple comparisons using just those
> >>>probes that made it through the fold-filter?  When correcting for
> >>>multiple comparisons you take a penalty for the number of comparison
> >>>you are correcting.  The larger the pool of comparisons, the larger
> >>>the penalty, thus the larger the adjusted p-value.  Or more
> >>>importantly, the smaller the set, the less your adjusted p-value is
> >>>adjusted (increased) relative to your raw p-value.  The argument is
> >>>that you used the actual samples themselves you are comparing to
> >>>unfairly reduce the adjusted p-value penalty.
> >>
> >>
> >>  It is not valid to use phenotype to compute t-statistics for a
> >>particular phenotype and filter based on those p-values and to then
> >>use p-value correction methods on the result. I don't think we need
> >>research, it seems pretty obvious that this is not a valid approach.
> >>
> >>   You can do non-specific filtering, but all you are really doing
> >>there is to remove genes that are inherently uninteresting no matter
> >>what the phenotype of the corresponding sample (if there is no
> >
> > variation in
> >
> >>expression for a particular gene across samples then it has   no
> >>information about the phenotype of the sample). Filtering on low
> >>values is probably a bad idea although many do it (and I used to, and
> >>still do sometimes depending on the task at hand).
> >>
> >>
> >>  Best wishes
> >>    Robert
> >>
> >>
> >>
> >>>Has anyone considered this issue or heard of problems of using a
> >>>specific type of filtering rather than a non-specific one?
> >>>Thank You for any responses.
> >>>
> >>>Daniel Bornman
> >>>Research Scientist
> >>>Battelle Memorial Institute
> >>>505 King Ave
> >>>Columbus, OH 43201
> >>>
> >>>_______________________________________________
> >>>Bioconductor mailing list
> >>>Bioconductor at stat.math.ethz.ch
> >>>https://stat.ethz.ch/mailman/listinfo/bioconductor
> >>>
> >>
> >>
> >>--
> >>Robert Gentleman, PhD
> >>Program in Computational Biology
> >>Division of Public Health Sciences
> >>Fred Hutchinson Cancer Research Center 1100 Fairview Ave. N, M2-B876
> >>PO Box 19024 Seattle, Washington 98109-1024 206-667-7700
> >>rgentlem at fhcrc.org
> >>
> >>_______________________________________________
> >>Bioconductor mailing list
> >>Bioconductor at stat.math.ethz.ch
> >>https://stat.ethz.ch/mailman/listinfo/bioconductor
> >>
> >
> >
> > --
> > Robert Gentleman, PhD
> > Program in Computational Biology
> > Division of Public Health Sciences
> > Fred Hutchinson Cancer Research Center
> > 1100 Fairview Ave. N, M2-B876
> > PO Box 19024
> > Seattle, Washington 98109-1024
> > 206-667-7700
> > rgentlem at fhcrc.org
> >
>
>--
>Robert Gentleman, PhD
>Program in Computational Biology
>Division of Public Health Sciences
>Fred Hutchinson Cancer Research Center
>1100 Fairview Ave. N, M2-B876
>PO Box 19024
>Seattle, Washington 98109-1024
>206-667-7700
>rgentlem at fhcrc.org
>
>_______________________________________________
>Bioconductor mailing list
>Bioconductor at stat.math.ethz.ch
>https://stat.ethz.ch/mailman/listinfo/bioconductor

Naomi S. Altman                                814-865-3791 (voice)
Associate Professor
Dept. of Statistics                              814-863-7114 (fax)
Penn State University                         814-865-1348 (Statistics)
University Park, PA 16802-2111



More information about the Bioconductor mailing list