[R] Anova - adjusted or sequential sums of squares?

[Douglas Bates]

Michael Watson (IAH-C) wrote:
> I guess the real problem is this:
> 
> As I have a different number of observations in each of the groups, the
> results *change* depending on which order I specify the factors in the
> model.  This unnerves me.  With a completely balanced design, this
> doesn't happen - the results are the same no matter which order I
> specify the factors.  
> 
> It's this reason that I have been given for using the so-called type III
> adjusted sums of squares...
> 

Yes, but how do you know that the test represented by the type III sums 
of squares makes sense in the context of your data?

The point that Bill is trying to make is that hypothesis tests always 
involve comparing the fits of two nested models to some data.  If you 
can't describe what the models being compared are, how can you interpret 
the results of the test?

There are many concepts introduced in statistics that apply to certain, 
specific cases but have managed to outgrow the original context so that 
people think they have general application.  The area of linear models 
and the analysis of variance is overrun with such concepts.  The list 
includes "significance of main effects", "overall mean", "R^2", 
"expected mean square", ...   These concepts do not stand on their own - 
they apply to specific models.

You are looking for *the answer* to the question "Is this main effect 
significant?"   What Bill is saying is that the question doesn't make 
sense.  You can ask "Does the model that incorporates this term and all 
other first-order terms provide a significantly better fit than same 
model without this one term?"  That's a well-phrased question.  You 
could even ask the same question about a model with first-order and 
higher-order terms versus the same model without this one term and you 
can get an answer to that question.  Whether or not that answer makes 
sense depends on whether or not the model with all the terms except the 
one being considered makes sense.  In most cases it doesn't so why say 
that you must get a p-value for a nonsensical test.  That number does 
*not* characterize a test of the "significance of the main effect".

How does R come in to this?  Well, with R you can fit models of great 
complexity to reasonably large data sets quickly and easily so, if you 
can formulate the hypothesis of interest to you by comparing the fit of 
an inner model to an outer model, then you simply fit them and compare 
them, usually with anova(fm1, fm2).  That's it.  That's all that the 
analysis of variance is about.  The complicated formulas and convoluted 
reasoning that we were all taught are there solely for the purpose of 
trying to "simplify" the calculations for this comparison.   They're 
unnecessary.  With a tool like R you simple fit model 1 then fit model 2 
  and compare the fits.  The only kicker is that you have to be able to 
describe your hypothesis in terms of the difference between two models.

With tools like R we have the potential to change statistics is viewed 
as a discipline and especially the way that it is taught.  Statistics is 
not about formulas - statistics is about models.  R allows you to think 
about the models and not grubby details of the calculations.