[R] ok to use glht() when interaction is NOT significant?

[Liaw, Andy]

Just to add my ever depreciating $0.02 USD:

Keep in mind that the significance testing paradigm puts a constraint on
false positive rate, and let false negative rate float.  What you should
consider is whether that makes sense in your situation.  All too often
this is not carefully considered, and sometimes people will do
not-very-kosher things to compensate for the conservativism of the
significance testing.

If you want to stay with the formality of the "protected tests", you
should first check the overall F-test of the entire model and make sure
that's significant before you look at the individual terms in the model.

It's not sufficient for A1 and A2 to be significantly different at B2
and not at B1 to say that there's significant interaction, but that the
difference between A1 and A2 at B1 has to be significantly different
that that at B2.  That's the definition of the interaction in the 2x2
case.  If you have a priori interest in the comparison of A1 vs. A2 at
B2, then you can test it as a pre-planned contrast and not worry too
much about "protection" or multiplicity.

HTH,
Andy
 

> -----Original Message-----
> From: r-help-bounces at r-project.org 
> [mailto:r-help-bounces at r-project.org] On Behalf Of array chip
> Sent: Tuesday, March 08, 2011 1:31 AM
> To: Bert Gunter
> Cc: r-help at stat.math.ethz.ch
> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
> 
> Hi Bert, thank you for your thoughtful and humorous comments, :-)
> 
> It is scientifically meaningful to do those comparisons, and 
> the results of 
> these comparisons actually make sense to our hypothesis, i.e. 
> one is significant 
> at B2 level while the other is not at B1 level. Just 
> unfortunately, the overall 
> F test for interaction is not significant. I understand 
> "formally" one should 
> not do these post-hoc comparisons under non-significant 
> interaction term. But 
> should I really stop comparing under this situation, 
> especially when these 
> comparisons conform to our hypothesis? I am encouraged to see 
> that you said "For 
> exploratory purposes, such post hoc comparisons might lead to 
> great science". 
> However, my concern is these results may not pass reviewers 
> when sent out for 
> publication.
> 
> BTW, I am non-US reader, so I did google "never inhaled". :-)
> 
> John
> 
> 
> 
> 
> ________________________________
> From: Bert Gunter <gunter.berton at gene.com>
> 
> Cc: r-help at stat.math.ethz.ch
> Sent: Mon, March 7, 2011 9:20:11 PM
> Subject: Re: [R] ok to use glht() when interaction is NOT significant?
> 
> Inline below
> 
> 
> > Hi, let's say I have a simple ANOVA model with 2 factors A 
> (level A1 and A2) 
> >and
> > B (level B1 and B2) and their interaction:
> >
> > aov(y~A*B, data=dat)
> >
> > It turns out that the interaction term is not significant 
> (e.g. P value = 
> 0.2),
> > but if I used glht() to compare A1 vs. A2 within each level 
> of B, I found that
> > the comparison is not significant when B=B1, but is very 
> significant (P<0.01)
> > when B=B2.
> >
> > My question is whether it's legal to do this post-hoc 
> comparison when the
> > interaction is NOT significant? Can I still make the claim 
> that there is a
> > significant difference between A1 and A2 when B=B2?
> 
> (I am serious here). Don't know what "legal" means. Why do you want to
> make the claim? When does it **ever** mean anything scientifically
> meaningful to make it? What is the **scientific** question of
> interest? Are the data unbalanced? Have you plotted the data to tell
> you what's going on?
> 
> Warning: I come from the school (maybe I'm the only student...) that
> believes all such formal post hoc comparisons are pointless, silly,
> wastes of effort.  Note the word: "formal" -- that is pretending the P
> values mean anything, For exploratory purposes, which can certainly
> include producing P values as well as graphs, such post hoc
> comparisons might lead to great science. It's the "formal" part that I
> reject and that you seem to be hung up on.
> 
> Note also: If you're a Bayesian and can put priors on everything, you
> can spit out posteriors and Bayes factors to your heart's content.
> Really! -- no need to sweat multiplicity even. Of course, I speak here
> only as an observer, having never actually inhaled myself.*
> 
> Cheers,
> Bert
> 
> *Apologies to all non-US and younger readers. This is a smart-aleck
> reference to an infamous dumb remark from a recent famous, smart
> former U.S. president. Google "never inhaled" for details.
> 
> >
> > Thanks
> >
> > John
> >
> >
> >
> >
> >        [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-help at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
> 
> 
> 
> -- 
> Bert Gunter
> Genentech Nonclinical Biostatistics
> 467-7374
> http://devo.gene.com/groups/devo/depts/ncb/home.shtml
> 
> 
> 
>       
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 
Notice:  This e-mail message, together with any attachme...{{dropped:11}}