[BioC] ttest or fold change

[Baker, Stephen]

Two POINTS:

First, I think people are missing the point about statistics.  When you
look at tens of thousands of genes, random chance plays a BIG role.
This means that large fold changes will occur and very small p-values
will occur often by chance alone.  The investigator doesn't know which
large fold changes or small p-values are due to chance or which are
"true" results.  

The best we can do is to try to apply statisics (and biology) with these
things in mind, that's what statistics is all about (one of the things
anyway),i.e. applying methods in such a way as to get results that are
meaningful and for which the probabilities of making different types of
errors are known.

With respect to just looking at the rankings, with small samples the
distributions are well known,  very discrete and have very few distinct
probabilities.  The situation that Mike describes, i.e. 10 samples with
5 per group is a situation for which there are 252 possible outcomes (10
choose 5 combinatorically).  If you consider both up and down regulation
as an outcome of interest there are 126 distinct outcomes so that the
probability of this occuring by chance alone is 1 in 126 or .00794.

This seems small but with a microarray with thousands of genes, this
easily produces a bunch of false positives. I looked at 10 chips from a
real control group arbitrarily labeling 5 chips as control and 5 as
experimental.  I would by theory expect 35 false positives and got
exactly 32, that is 32 sitations in which all the low ranks were in one
group and the high ranks in the other.  For a chip with 22000 genes, you
would expect 175 false positive results by this criteria. Standard
statistical methods would give you a specified type I error rate that
you can count on, it would have found NONE of the genes significant
(i.e. bonferroni adjustment)

This same set of control chips produced 50 false positive results using
a 2 fold change criteria. Again, these are ALL false positives.

Second,

With respect to t-Tests a couple of people have mentioned "the problem"
that the t-test denominator may be accidentally "too small" .  This is
because the t-test uses an ESTIMATE of the variance from the sample
itself.  This is what William Sealey Gossett, otherwise known as
"Student" discovered that prompted him to develop the t-distribution and
t-test.  Gossett or Student was a brewmaster for Guinness breweries in
Dublin and was doing experiments with hops and things and discovered
that the well known "normal distribution" was inaccurate when you
estimated the variance from a sample.  He developed the t-distribution
empirically that takes the variability in the variance estimate into
account so that the t-test is ALREADY ADJUSTED to compensate for weird
values in the denominator due to random sampling. 

One thing that I think is too often ignored is that different genes have
different variances, the fact that one gene appears to have a smaller
variance than its neighbors (or a larger one) could be that it ACTUALLY
DOES have a larger or smaller variance OR it may be due to sampling
variability.  The t-test assumes the former but adjusts for the latter
possiblity. It worked then and it works now, it is NOT a problem. 

Student's friend, the genius R.A.Fisher took Student's empirical result
and worked out the theory on which analysis of variance is all based.
This theory has withstood the test of time, it is about 100 years old
and still holds, given the assumptions are correct, t-tests and ANOVA
are still "uniformly most powerful tests".  
-.- -.. .---- .--. ..-.
Stephen P. Baker, MScPH, PhD (ABD)            (508) 856-2625
Sr. Biostatistician- Information Services
Lecturer in Biostatistics                     (775) 254-4885 fax
Graduate School of Biomedical Sciences
University of Massachusetts Medical School, Worcester
55 Lake Avenue North                          stephen.baker at umassmed.edu
Worcester, MA 01655  USA

 


------------------------------

.Message: 3
.Date: Mon, 15 Dec 2003 12:11:43 -0000
.From: "michael watson (IAH-C)" <michael.watson at bbsrc.ac.uk>
.Subject: RE: [BioC] ttest or fold change
.To: bioconductor at stat.math.ethz.ch
.Message-ID:
.
<20B7EB075F2D4542AFFAF813E98ACD93028224D1 at cl-exsrv1.irad.bbsrc.ac.uk>
.Content-Type: text/plain;	charset="utf-8"
.
.
.Why not try the non-parametric t-tests available?
.
.I know all the arguments about a "loss of power" etc, but at the end of
day, as statisticians and bioinformaticians, .sometimes biologists come
to us with small numbers of replicates (for very understandable reasons)
and it is our job to get .some meaning out of that data.  Trying to fit
any kind of statistic involving a p-value to such data is a difficult
and .risky task, and trying to explain those results to the biologist is
often very difficult.
.
.So here's what happens with the non-parametric tests based on ranking.
Those genes with the highest |t| are those where .all the replicates of
one condition are greater than all the replicates of the other
condition.  The next highest |t| is .where all but one of the replicates
of one condition are greater than all the replicates of the other
conddition, etc etc.
.
.OK, so some of these differences could occur by chance, but we're
dealing with often millions of data points and I really .don't think
it's possible to make no mistakes.  And curse me if you like, but if i
have a gene expression measurement, .replicated 5 times in two
conditions, and in one condition all five replicates are higher than the
five replicates of the .other condition, then I believe that that gene
is differentially expressed.  And thats easy to find with non-parametric
t, .and it is easy to explain to a biologist, and at the end of the day,
is it really wrong to do that?