My collaborators have an experimental design in which cells are treated
experimentally with two conditions, and they naturally wish to know the
differences in response between the two.  Moreover, the experiments are
setup in pairs of treatments, with each pair produced from the same "batch"
of cells, inducing a natural pairing that we might want to include in the
limma design.  We would do this to take advantage of expected correlations
in gene expression due to the source of cells in each experiments.

However, when we run the analyses with either a paired or unpaired design,
we find that the unpaired statistics are far more significant (~1000
probesets at FDR < 5%) than with the paired design (~100), which implies
that there is not enough correlation across pairs, at least relative to the
induced treatment effects.   A bit stumped at first, I finally confirmed
for myself that even in the presence of strong correlation, a larger
treatment effect will remain more significant with an unpaired design:

> wt <- c(0.9, 1.0, 1.2)
> mean(wt)
[1] 1.033333

> mu <- c(6.2, 6.1, 5.9)
> mean(mu)
[1] 6.066667

> mean(mu) - mean(wt)
[1] 5.033333

> mean(mu-wt)
[1] 5.033333

In fact, no matter how you pair up mu and wt, you will always get 5.0333 as
the paired fold change.  However, the variance may change, depending on how
correlated mu and wt are (it is this correlation that we are trying to take
advantage of by pairing):

> cor(wt, mu)
[1] -1
> sd(mu-wt)
[1] 0.305505

> mu2 <- sort(mu)
> wt2 <- sort(wt)

> cor(wt2, mu2)
[1] 0.9285714

> sd(mu2-wt2)
[1] 0.05773503

Now let's see how this affects t-test significance:

> t.test(*mu, wt, paired=F*, var.equal=T)

Two Sample t-test

data:  mu and wt
*t = 40.3564, df = 4, p-value = 2.253e-06*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 4.687050 5.379617
sample estimates:
mean of x mean of y
 6.066667  1.033333

> t.test(*mu, wt, paired=T*, var.equal=T)

Paired t-test

data:  mu and wt
*t = 28.5363, df = 2, p-value = 0.001226*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 4.274417 5.792250
sample estimates:
mean of the differences
               5.033333

> t.test(*mu2, wt2, paired=F*, var.equal=T)

Two Sample t-test

data:  mu2 and wt2
*t = 40.3564, df = 4, p-value = 2.253e-06*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 4.687050 5.379617
sample estimates:
mean of x mean of y
 6.066667  1.033333

> t.test(*mu2, wt2, paired=T*, var.equal=T)

Paired t-test

data:  mu2 and wt2
*t = 151, df = 2, p-value = 4.385e-05*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 4.889912 5.176755
sample estimates:
mean of the differences
               5.033333


In the first case, when wt & mu were anti-correlated, the unpaired t-test
gave much better P values; adding the pairing info made the variation in
mu-wt larger, and so the P value got worse (t statistic was smaller; also
the smaller df in the paired test will, for the same t statistics, deflate
the P value).

In the second case, when mt & mu were strongly correlated, the paired
t-test was still very good, and had a much higher t statistics than the
unpaired test, but the P value was still not quite as good as the unpaired
-- this is due to the drop in df.  Much of this has to do with the large
difference between to the two groups; if I make the difference between mu
and wt a bit smaller, without changing the correlation structure:

> *mu3 <- mu2 - 4.5 # cor(wt2, mu3) == cor(wt2, mu2)*
> t.test(*mu3, wt2, paired=F*, var.equal=T)

 Two Sample t-test

data:  mu3 and wt2
*t = 4.2762, df = 4, p-value = 0.01289*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.1870498 0.8796169
sample estimates:
mean of x mean of y
 1.566667  1.033333

> t.test(*mu3, wt2, paired=T*, var.equal=T)

 Paired t-test

data:  mu3 and wt2
*t = 16, df = 2, p-value = 0.003884*
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.3899116 0.6767551
sample estimates:
mean of the differences
              0.5333333

Then, finally, you see a change in the expected direction: the paired test
is more significant than the unpaired test.

So, the question is -- how might you convince yourself (or a savvy and
skeptic reviewer for that matter) that your deliberate removal of pairing
from your design is the statistically valid approach?  My own thoughts were
to show a distribution of observed correlations across pairings, to
demonstrate that the within-pairings variances were much smaller than the
between-treatment variances of interest.

Thanks for your time and attention,
-Aaron

--
Aaron J. Mackey, PhD
Assistant Professor
Center for Public Health Genomics
University of Virginia
amackey@virginia.edu
http://www.cphg.virginia.edu/mackey

	[[alternative HTML version deleted]]

