[R] Lack of independence in anova()

[(Ted Harding)]

On 06-Jul-05 Phillip Good wrote:
> Do you or Lumley have a citation for this conclusion?   Most people go
> forward with the ANOV on the basis that the various tests are
> independent.

This hardly needs a citation -- Thomas Lumley's explanation (excerpted
below from Doug Bates's mail) is justification enough, and it is
succinct and elementary.

However, a place one naturally looks for caveats of this kind is
in Kendall & Stuart, and I duly found it (Vol 3, section 35.46
of my 1983 edition). It is essentially exactly the same explanation:

  "However, the tests in the AV tables which we have considered
   are never independent tests, for although the various SS in a
   table may be independent of each other, all the tests we have
   derived use the Residual SS as denominator of the test statistic,
   and the various tests must therefore be statistically dependent,
   since, e.g., a Residual SS which is (by chance) large will
   depress all the values of the test staistics simultaneously."

(And K&S, thorough as they are with citations, do not cite any
primary reference for this either!)

However, if the "degrees of freedom" for Residual SS is large,
then the amount of random variation in the denominator will be
small and it will be effectively constant. Then, of course,
with independent numerators, the tests will be effectively
independent (and equivalent to chi-squared) and also, therefore,
the p-values.

The fact that "most people go forward with the ANOV on the basis
that the various tests are independent" possibly reflects the
wide-spread act of faith that one has "a large sample", whatever
the value of n may really be. One wonders how often people check
the p-value for their F on (n1:n2) d.f. against the p-value for
(n1:Inf) d.f.? The 5% point decreases quite perceptibly as n2
increases up to about 20, and more slowly thereafter; but still
the difference between F(n1:20) and F(n1:Inf) is substantial
for any n1 (being about 0.5 for n1 up to about 10, increasing
thereafter up to 0.84):

n1<-c(1+2*(0:50),5000);cbind(n1,qf(0.95,n1,20) - qf(0.95,n1,Inf))

F(Inf,Inf) = 1 ; F(20:Inf) = qf(0.95,Inf,20) = 1.841

Conversely (e.g.):

  > 1-pf(qf(0.95,20,20),20,20)
  [1] 0.05
  > 1-pf(qf(0.95,20,20),20,Inf)
  [1] 0.002391189

Such differences are related to the degree of non-independence of
several tests on the same data.

> [Douglas Bates]:
> Thomas Lumley came to my rescue with an explanation.  There is no
> reason why the results of the F tests should be independent.  The
> numerators are independent but the denominator is the same for both
> tests.  When, due to random variation, the denominator is small, then
> the p-values for both tests will tend to be small.  If, instead of
> F-tests you use chi-square tests then you do see independence.

But surely this amounts to assuming n2 = Inf? If that's an adequate
approximation, then fine; but if not (see e.g. above) then not!

Best wishes to all,
Ted.


--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.Harding at nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 06-Jul-05                                       Time: 19:29:18
------------------------------ XFMail ------------------------------