[R] ok to use glht() when interaction is NOT significant?

[Bert Gunter]

I basically agree -- an additional $.01 inline below. -- Bert

On Tue, Mar 8, 2011 at 7:44 AM, Liaw, Andy <andy_liaw at merck.com> wrote:
> Just to add my ever depreciating $0.02 USD:
>
> Keep in mind that the significance testing paradigm puts a constraint on
> false positive rate, and let false negative rate float.  What you should
> consider is whether that makes sense in your situation.  All too often
> this is not carefully considered, and sometimes people will do
> not-very-kosher things to compensate for the conservativism of the
> significance testing.
>
> If you want to stay with the formality of the "protected tests", you
> should first check the overall F-test of the entire model and make sure
> that's significant before you look at the individual terms in the model.

The caveat here is that without (reasonably near) balance, individual
terms don't necessarily mean what you would like them to; in
particular "main effects" and "interactions" no longer have clear
interpretations.

>
> It's not sufficient for A1 and A2 to be significantly different at B2
> and not at B1 to say that there's significant interaction, but that the
> difference between A1 and A2 at B1 has to be significantly different
> that that at B2.  That's the definition of the interaction in the 2x2
> case.  If you have a priori interest in the comparison of A1 vs. A2 at
> B2, then you can test it as a pre-planned contrast and not worry too
> much about "protection" or multiplicity.

Yes. The key is "pre-planned." This does not mean something you found
in the course of the analysis (including graphics) and then" post"
pre-planned . ;-)


Cheers,
Bert

>
> HTH,
> Andy
>
>
>> -----Original Message-----
>> From: r-help-bounces at r-project.org
>> [mailto:r-help-bounces at r-project.org] On Behalf Of array chip
>> Sent: Tuesday, March 08, 2011 1:31 AM
>> To: Bert Gunter
>> Cc: r-help at stat.math.ethz.ch
>> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
>>
>> Hi Bert, thank you for your thoughtful and humorous comments, :-)
>>
>> It is scientifically meaningful to do those comparisons, and
>> the results of
>> these comparisons actually make sense to our hypothesis, i.e.
>> one is significant
>> at B2 level while the other is not at B1 level. Just
>> unfortunately, the overall
>> F test for interaction is not significant. I understand
>> "formally" one should
>> not do these post-hoc comparisons under non-significant
>> interaction term. But
>> should I really stop comparing under this situation,
>> especially when these
>> comparisons conform to our hypothesis? I am encouraged to see
>> that you said "For
>> exploratory purposes, such post hoc comparisons might lead to
>> great science".
>> However, my concern is these results may not pass reviewers
>> when sent out for
>> publication.
>>
>> BTW, I am non-US reader, so I did google "never inhaled". :-)
>>
>> John
>>
>>
>>
>>
>> ________________________________
>> From: Bert Gunter <gunter.berton at gene.com>
>>
>> Cc: r-help at stat.math.ethz.ch
>> Sent: Mon, March 7, 2011 9:20:11 PM
>> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
>>
>> Inline below
>>
>>
>> > Hi, let's say I have a simple ANOVA model with 2 factors A
>> (level A1 and A2)
>> >and
>> > B (level B1 and B2) and their interaction:
>> >
>> > aov(y~A*B, data=dat)
>> >
>> > It turns out that the interaction term is not significant
>> (e.g. P value =
>> 0.2),
>> > but if I used glht() to compare A1 vs. A2 within each level
>> of B, I found that
>> > the comparison is not significant when B=B1, but is very
>> significant (P<0.01)
>> > when B=B2.
>> >
>> > My question is whether it's legal to do this post-hoc
>> comparison when the
>> > interaction is NOT significant? Can I still make the claim
>> that there is a
>> > significant difference between A1 and A2 when B=B2?
>>
>> (I am serious here). Don't know what "legal" means. Why do you want to
>> make the claim? When does it **ever** mean anything scientifically
>> meaningful to make it? What is the **scientific** question of
>> interest? Are the data unbalanced? Have you plotted the data to tell
>> you what's going on?
>>
>> Warning: I come from the school (maybe I'm the only student...) that
>> believes all such formal post hoc comparisons are pointless, silly,
>> wastes of effort.  Note the word: "formal" -- that is pretending the P
>> values mean anything, For exploratory purposes, which can certainly
>> include producing P values as well as graphs, such post hoc
>> comparisons might lead to great science. It's the "formal" part that I
>> reject and that you seem to be hung up on.
>>
>> Note also: If you're a Bayesian and can put priors on everything, you
>> can spit out posteriors and Bayes factors to your heart's content.
>> Really! -- no need to sweat multiplicity even. Of course, I speak here
>> only as an observer, having never actually inhaled myself.*
>>
>> Cheers,
>> Bert
>>
>> *Apologies to all non-US and younger readers. This is a smart-aleck
>> reference to an infamous dumb remark from a recent famous, smart
>> former U.S. president. Google "never inhaled" for details.
>>
>> >
>> > Thanks
>> >
>> > John
>> >
>> >
>> >
>> >
>> >        [[alternative HTML version deleted]]
>> >
>> > ______________________________________________
>> > R-help at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-help
>> > PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> > and provide commented, minimal, self-contained, reproducible code.
>> >
>>
>>
>>
>> --
>> Bert Gunter
>> Genentech Nonclinical Biostatistics
>> 467-7374
>> http://devo.gene.com/groups/devo/depts/ncb/home.shtml
>>
>>
>>
>>
>>       [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
> Notice:  This e-mail message, together with any attachments, contains
> information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
> New Jersey, USA 08889), and/or its affiliates Direct contact information
> for affiliates is available at
> http://www.merck.com/contact/contacts.html) that may be confidential,
> proprietary copyrighted and/or legally privileged. It is intended solely
> for the use of the individual or entity named on this message. If you are
> not the intended recipient, and have received this message in error,
> please notify us immediately by reply e-mail and then delete it from
> your system.
>
>



-- 
Bert Gunter
Genentech Nonclinical Biostatistics
467-7374
http://devo.gene.com/groups/devo/depts/ncb/home.shtml