[R-sig-ME] Fwd: Question about non-significant interactions
Fox, John
j|ox @end|ng |rom mcm@@ter@c@
Thu Jul 11 11:32:59 CEST 2019
Dear Francesco,
> -----Original Message-----
> From: Francesco Romano [mailto:fbromano77 using gmail.com]
> Sent: Thursday, July 11, 2019 9:50 AM
> To: Fox, John <jfox using mcmaster.ca>
> Cc: Thierry Onkelinx <thierry.onkelinx using inbo.be>; r-sig-mixed-models using r-
> project.org
> Subject: Re: [R-sig-ME] Fwd: Question about non-significant interactions
>
> Dear John (and others),
>
> I think I have solved this issue, if my understanding of the stats is correct.
>
> First, wIth regards to (2), yes, I have taken that into account. I recoded my
> factors as contr.sum, ran the Anova again (with and without the bayesian
> wrapper), to find that results did not change.
The tests for the interaction should be identical but it would be odd if the type-III tests for the main effects didn't change at least slightly when you changed the contrast coding from contr.treatment() to contr.sum().
> I also tried the other ways,
> namely via the mixed function, which allows you to use mixed-effect models,
> automatically recodes your factors to contr.sum, and uses LRT, and the
> traditional anova function with lihelihood ratio tests comparing two models
> at a time. In all cases, I never get a significant interaction but always get a
> main effect for both fixed effects. By the way, regarding your (3), our original
> contact a few years back was a query of mine trying to find a shortcut to
> traditional likelihood ratio testing comparing models that would also allow
> random effects, hence your suggestion to use your car::Anova, which for the
> record, I love.
Thank you.
>
> As far as my understanding of the results goes, but feel free to chime in if you
> disagree, there is no main interaction because the simple effects that I find
> (i.e. the pairwise comparisons) are the same across the two tests. To simplify
> things a bit, in my GJT test, the first level of the task factor, I find the following
> relationships between groups HL = L2, L1 > HL, and L1 > L2, where HL, L1, and
> L2 are the three levels of the group factor, and = is just shorthand for no
> significant difference. The relationships are exactly the same at the other level
> of the task factor, namely priming, Same goes for the relationship between
> tasks at each level of the group variable: every group is significantly more
> likely to score correctly on the GJT than the priming task. I take these to
> explain why the main interaction turns out not to be statistically significant.
If the simple effects of group are identical across the levels of task, then of course there is no interaction between group and task. If the simple effects are similar, then you probably won't detect an interaction unless, e.g., you have a lot of data. But to treat "no significant difference" as indicative of equality (even heuristically) could certainly lead to apparently paradoxical results, although that hasn't happened in your case.
Best,
John
>
> Best,
>
>
> Frank
>
>
> On Wed, Jul 10, 2019 at 3:43 PM Fox, John <jfox using mcmaster.ca
> <mailto:jfox using mcmaster.ca> > wrote:
>
>
> Dear Francesco and Thierry,
>
> To elaborate slightly on Thierry's suggestion and also to address some
> other points:
>
> (1) You (Francesco) could use the linearHypothesis() function in the
> car package to test more specific hypotheses. It would probably be easier to
> formulate such hypotheses if you fit an equivalent cell-mean model, defining a
> factor with levels for the 9 = 3*3 combinations of levels of your two factors.
>
> (2) The type-III tests produced by Anova() still aren't sensible. Did you
> look at the material from ?Anova that I included in my previous response?
>
> (3) You say that Anova() uses "a form of shortcut to the traditional
> way of model-fitting/ reduction via the function anova() comparing models."
> That's not quite true in general, because what Anova() does depends on the
> model class and the type of test statistic selected. It's true that for a mixed-
> effects model, Anova() produces Wald tests rather than likelihood-ratio tests.
> Anova() doesn't have a "bglmerMod" method, and so I assume that the
> "merMod" method is inherited or that the default method is being used.
>
> Best,
> John
>
> -----------------------------
> John Fox, Professor Emeritus
> McMaster University
> Hamilton, Ontario
> Canada L8S 4M4
> web: socserv.mcmaster.ca/jfox <http://socserv.mcmaster.ca/jfox>
>
>
> ________________________________________
> From: R-sig-mixed-models [r-sig-mixed-models-bounces using r-
> project.org <mailto:r-sig-mixed-models-bounces using r-project.org> ] on behalf of
> Thierry Onkelinx via R-sig-mixed-models [r-sig-mixed-models using r-project.org
> <mailto:r-sig-mixed-models using r-project.org> ]
> Sent: July 10, 2019 2:51 AM
> To: Francesco Romano
> Cc: r-sig-mixed-models using r-project.org <mailto:r-sig-mixed-models using r-
> project.org>
> Subject: Re: [R-sig-ME] Fwd: Question about non-significant
> interactions
>
> Dear Francesco,
>
> To answer your question, you should convert your hypothesis in a set
> of
> linear contrasts and test those.
>
> Best regards,
>
> ir. Thierry Onkelinx
> Statisticus / Statistician
>
> Vlaamse Overheid / Government of Flanders
> INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH
> INSTITUTE FOR NATURE AND
> FOREST
> Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality
> Assurance
> thierry.onkelinx using inbo.be <mailto:thierry.onkelinx using inbo.be>
> Havenlaan 88 bus 73, 1000 Brussel
> www.inbo.be <http://www.inbo.be>
>
> ///////////////////////////////////////////////////////////////////////////
> ////////////////
> To call in the statistician after the experiment is done may be no more
> than asking him to perform a post-mortem examination: he may be
> able to say
> what the experiment died of. ~ Sir Ronald Aylmer Fisher
> The plural of anecdote is not data. ~ Roger Brinner
> The combination of some data and an aching desire for an answer
> does not
> ensure that a reasonable answer can be extracted from a given body
> of data.
> ~ John Tukey
> ///////////////////////////////////////////////////////////////////////////
> ////////////////
>
> <https://www.inbo.be>
>
>
> Op di 9 jul. 2019 om 23:25 schreef Francesco Romano
> <fbromano77 using gmail.com <mailto:fbromano77 using gmail.com> >:
>
> > ---------- Forwarded message ---------
> > From: Francesco Romano <fbromano77 using gmail.com
> <mailto:fbromano77 using gmail.com> >
> > Date: Tue, Jul 9, 2019 at 11:24 PM
> > Subject: Re: [R-sig-ME] Question about non-significant interactions
> > To: Fox, John <jfox using mcmaster.ca <mailto:jfox using mcmaster.ca> >
> >
> >
> > Dear John,
> >
> > Thanks for the reply. One of my research entails examining the
> relationship
> > between 3 groups of speakers, the 3 levels of the group categorical
> > variable previously mentioned, and two tasks. One prediction is that
> one
> > group will perform better than other groups on one test but not the
> other.
> >
> > I fit a maximal model using the bglmr function as shown previously,
> then
> > used car::Anova to determine main effects. My understanding from
> previous
> > interaction with you precisely here on r-sig-me is that the function
> works
> > as a form of shortcut to the traditional way of model-fitting/
> reduction
> > via the function anova() comparing models, eliminating terms one at
> a time.
> >
> > I hope this is clearer now and yes, the question is more of a
> statistical
> > one than an R one, even though I suspect the mixed-effect aspect of
> the
> > regression may be relevant to answering it.
> >
> > Frank
> >
> > Tue, Jul 9, 2019 at 11:03 PM Fox, John <jfox using mcmaster.ca
> <mailto:jfox using mcmaster.ca> > wrote:
> >
> > > Dear Francesco,
> > >
> > > I didn't entirely follow your question and I expect that to answer it,
> it
> > > would be necessary to know more about what your research
> entails. As you
> > > imply, this seems to be more a statistics question than an R
> question.
> > It's
> > > also not clear to me what function you used to fit the mixed-
> effects
> > > logistic regression.
> > >
> > > But I did notice that you're apparently using Anova() for type-III
> tests
> > > with the default contr.treatment() coding for factors. The main-
> effect
> > > tests that result are not sensible. As it says in ?Anova:
> > >
> > > "Warning
> > > Be careful of type-III tests: For a traditional multifactor ANOVA
> model
> > > with interactions, for example, these tests will normally only be
> > sensible
> > > when using contrasts that, for different terms, are orthogonal in
> the
> > > row-basis of the model, such as those produced by contr.sum,
> contr.poly,
> > or
> > > contr.helmert, but not by the default contr.treatment. In a model
> that
> > > contains factors, numeric covariates, and interactions, main-effect
> tests
> > > for factors will be for differences over the origin. In contrast (pun
> > > intended), type-II tests are invariant with respect to (full-rank)
> > contrast
> > > coding. If you don't understand this issue, then you probably
> shouldn't
> > use
> > > Anova for type-III tests."
> > >
> > > I hope that this is of some help,
> > > John
> > > -----------------------------
> > > John Fox, Professor Emeritus
> > > McMaster University
> > > Hamilton, Ontario
> > > Canada L8S 4M4
> > > web: socserv.mcmaster.ca/jfox <http://socserv.mcmaster.ca/jfox>
> > >
> > >
> > > ________________________________________
> > > From: R-sig-mixed-models [r-sig-mixed-models-bounces using r-
> project.org <mailto:r-sig-mixed-models-bounces using r-project.org> ] on
> > > behalf of Francesco Romano [fbromano77 using gmail.com
> <mailto:fbromano77 using gmail.com> ]
> > > Sent: July 9, 2019 9:49 AM
> > > To: r-sig-mixed-models using r-project.org <mailto:r-sig-mixed-
> models using r-project.org>
> > > Subject: [R-sig-ME] Question about non-significant interactions
> > >
> > > Dear all,
> > >
> > >
> > > I have more of a theoretical than practical question for you. The
> model I
> > > am using has two IVs, group (3 levels) and task (2 levels), and a
> > > categorical DV (correct versus incorrect), hence logistic regression.
> > > Random effects for subjects and items, as well as slopes for group
> by
> > item
> > > and task by subject.
> > >
> > > I am interested in the effect of belonging any of three groups, the
> > levels
> > > of the group IV, in order to test some a priori predictions. The
> bayesian
> > > wrapper is to help the model converge.
> > >
> > > Here is the output:
> > >
> > > > summary(paper2analysis1)
> > > Cov prior : item ~ wishart(df = 5.5, scale = Inf, posterior.scale =
> cov,
> > > common.scale = TRUE)
> > > : Participant ~ wishart(df = 4.5, scale = Inf,
> > posterior.scale =
> > > cov, common.scale = TRUE)
> > > Prior dev : 6.9466
> > >
> > > Generalized linear mixed model fit by maximum likelihood
> (Laplace
> > > Approximation) ['bglmerMod']
> > > Family: binomial ( logit )
> > > Formula: correctness ~ task * group + (1 + task | Participant) + (1 +
> > > group | item)
> > > Data: data
> > > Control: glmerControl(optimizer = "bobyqa")
> > >
> > > AIC BIC logLik deviance df.resid
> > > 3857.8 3957.2 -1913.9 3827.8 5570
> > >
> > > Scaled residuals:
> > > Min 1Q Median 3Q Max
> > > -2.0196 -0.3744 -0.2312 -0.1368 6.9534
> > >
> > > Random effects:
> > > Groups Name Variance Std.Dev. Corr
> > > item (Intercept) 1.1266 1.0614
> > > groupL2 0.1311 0.3620 -0.12
> > > groupNS 0.2029 0.4504 -0.31 0.17
> > > Participant (Intercept) 0.7582 0.8708
> > > taskpriming 1.2163 1.1029 -0.77
> > > Number of obs: 5585, groups: item, 219; Participant, 46
> > >
> > > Fixed effects:
> > > Estimate Std. Error z value Pr(>|z|)
> > > (Intercept) -2.49187 0.28318 -8.800 < 2e-16 ***
> > > taskpriming 1.30911 0.37367 3.503 0.000459 ***
> > > groupL2 -0.04042 0.38322 -0.105 0.916005
> > > groupNS -1.01144 0.36607 -2.763 0.005727 **
> > > taskpriming:groupL2 0.04305 0.48693 0.088 0.929544
> > > taskpriming:groupNS -0.04942 0.46034 -0.107 0.914506
> > > ---
> > > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> > >
> > > Correlation of Fixed Effects:
> > > (Intr) tskprm gropL2 gropNS tsk:L2
> > > taskpriming -0.733
> > > groupL2 -0.660 0.482
> > > groupNS -0.693 0.507 0.509
> > > tskprmng:L2 0.499 -0.632 -0.755 -0.386
> > > tskprmng:NS 0.530 -0.676 -0.390 -0.750 0.508
> > >
> > > The model was then subjected to car::Anova for ANOVA type III
> analysis
> > with
> > > the following output:
> > >
> > > > car::Anova(paper2analysis1, type = "III")
> > > Analysis of Deviance Table (Type III Wald chisquare tests)
> > >
> > > Response: correctness
> > > Chisq Df Pr(>Chisq)
> > > (Intercept) 77.4344 1 < 2.2e-16 ***
> > > task 12.2737 1 0.0004594 ***
> > > group 9.9237 2 0.0070000 **
> > > task:group 0.0391 2 0.9806462
> > > ---
> > > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> > >
> > > I am not sure how to interpret the non-significant interaction in
> this
> > > case. Does this mean that, although simple effects exist at group
> level
> > > within one particular task or at task level within one particular
> group,
> > I
> > > lack sufficient power to conclude those effects are real? If I look at
> > the
> > > simple effects, I do indeed find such effects but am not sure how to
> > > interpret them against the lack of a main interaction. At a practical
> > > level, the interaction, rather than the main effects, is the most
> > important
> > > part of the analysis.
> > >
> > > Thank you in advance for any advice.
> > >
> > > Francesco
> > >
> > >
> > >
> > >
> > >
> > > Best,
> > >
> > > Frank
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-mixed-models using r-project.org <mailto:R-sig-mixed-
> models using r-project.org> mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> > >
> > --
> > Inviato da Gmail Mobile
> > --
> > Inviato da Gmail Mobile
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org <mailto:R-sig-mixed-models using r-
> project.org> mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org <mailto:R-sig-mixed-models using r-
> project.org> mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
More information about the R-sig-mixed-models
mailing list