As always, thanks Gordon for keeping the conversation focused. I guess I
was responding to Ryan's statement that the only hypothesis to be tested
was a difference between the two main groups, so the additional modeling of
subgroups seems only to reduce the overall residual variance, possibly
leading to inflated significance between groups.

re: "carefully interpreted", would you suggest in such situations that we
F-test the full model (four subgroups) vs. the nested model (two groups),
and only perform group-wise comparisons for cases where the nested model
could not be rejected (i.e. no evidence of interaction effects?)  That
would seem to fit the ANOVA interpretation, and the usual concerns about
testing for main effects when interactions may be present.

Thanks again,
-Aaron


On Fri, Jan 24, 2014 at 1:33 AM, Gordon K Smyth <smyth@wehi.edu.au> wrote:

> On Fri, 24 Jan 2014, Aaron Mackey wrote:
>
>  On Thu, Jan 23, 2014 at 6:48 PM, Gordon K Smyth <smyth@wehi.edu.au>
>> wrote:
>>
>>  My worry is that with this contrast, I'm effectively just testing two
>>>
>>>> groups against each other, and by having 4 groups in the design I will
>>>> be
>>>> estimating dispersions that are not appropriate for the test that I'm
>>>> doing, and hence I will overstate my confidence.
>>>>
>>>>
>>> The dispersions remain unchanged regardless of the contrast you test. The
>>> dispersions have been estimated after removing all differences between
>>> the
>>> four groups, i.e., without bias.
>>>
>>>
>> But had he more simply coded the samples as belonging to two groups, AB
>> and CD, then the dispersions could be larger, and the AB vs. CD mean
>> differences could be less significant than in his four-subgroup design. I
>> think that was the intent of Ryan's question.
>>
>
> Coding that doesn't reflect the true experimental design is likely to
> perform badly, and give less significance.  That doesn't make it more
> correct.
>
>
>  Is it fair to stratify along a priori expected subgroupings to minimize
>> variance and then ask group-level questions?
>>
>
> You are not asking a well posed question.  For one thing, "fair" and
> "unfair" are unhelpful concepts.  The only consideration is whether the
> statistical test that is done answers the scientific question being
> answered.  You haven't explained what scientific question you want to
> answer, so there is no basis for choosing a scientific test.
>
> Fitting a model that matches the experimental conditions and then making
> comparisons between groups has been the anova method since anova was first
> invented.  It answers what it answers, as I explained in my response to
> Ryan.
>
> Gordon
>
>  As you later say, if there is "lots of DE" between
>> A vs. B and/or C vs. D (read: larger group-wise variance) then the test
>> may
>> need to be "carefully interpreted".
>>
>> Thanks for your insight,
>> -Aaron
>>
>>
> ______________________________________________________________________
> The information in this email is confidential and inte...{{dropped:10}}

