[R-sig-ME] Fwd: Question about non-significant interactions
Francesco Romano
|brom@no77 @end|ng |rom gm@||@com
Thu Jul 11 09:50:09 CEST 2019
Dear John (and others),
I think I have solved this issue, if my understanding of the stats is
correct.
First, wIth regards to (2), yes, I have taken that into account. I recoded
my factors as contr.sum, ran the Anova again (with and without the bayesian
wrapper), to find that results did not change. I also tried the other ways,
namely via the mixed function, which allows you to use mixed-effect models,
automatically recodes your factors to contr.sum, and uses LRT, and the
traditional anova function with lihelihood ratio tests comparing two models
at a time. In all cases, I never get a significant interaction but always
get a main effect for both fixed effects. By the way, regarding your (3),
our original contact a few years back was a query of mine trying to find a
shortcut to traditional likelihood ratio testing comparing models that
would also allow random effects, hence your suggestion to use your
car::Anova, which for the record, I love.
As far as my understanding of the results goes, but feel free to chime in
if you disagree, there is no main interaction because the simple effects
that I find (i.e. the pairwise comparisons) are the same across the two
tests. To simplify things a bit, in my GJT test, the first level of the
task factor, I find the following relationships between groups HL = L2, L1
> HL, and L1 > L2, where HL, L1, and L2 are the three levels of the group
factor, and = is just shorthand for no significant difference. The
relationships are exactly the same at the other level of the task factor,
namely priming, Same goes for the relationship between tasks at each level
of the group variable: every group is significantly more likely to score
correctly on the GJT than the priming task. I take these to explain why the
main interaction turns out not to be statistically significant.
Best,
Frank
On Wed, Jul 10, 2019 at 3:43 PM Fox, John <jfox using mcmaster.ca> wrote:
> Dear Francesco and Thierry,
>
> To elaborate slightly on Thierry's suggestion and also to address some
> other points:
>
> (1) You (Francesco) could use the linearHypothesis() function in the car
> package to test more specific hypotheses. It would probably be easier to
> formulate such hypotheses if you fit an equivalent cell-mean model,
> defining a factor with levels for the 9 = 3*3 combinations of levels of
> your two factors.
>
> (2) The type-III tests produced by Anova() still aren't sensible. Did you
> look at the material from ?Anova that I included in my previous response?
>
> (3) You say that Anova() uses "a form of shortcut to the traditional way
> of model-fitting/ reduction via the function anova() comparing models."
> That's not quite true in general, because what Anova() does depends on the
> model class and the type of test statistic selected. It's true that for a
> mixed-effects model, Anova() produces Wald tests rather than
> likelihood-ratio tests. Anova() doesn't have a "bglmerMod" method, and so I
> assume that the "merMod" method is inherited or that the default method is
> being used.
>
> Best,
> John
>
> -----------------------------
> John Fox, Professor Emeritus
> McMaster University
> Hamilton, Ontario
> Canada L8S 4M4
> web: socserv.mcmaster.ca/jfox
>
>
> ________________________________________
> From: R-sig-mixed-models [r-sig-mixed-models-bounces using r-project.org] on
> behalf of Thierry Onkelinx via R-sig-mixed-models [
> r-sig-mixed-models using r-project.org]
> Sent: July 10, 2019 2:51 AM
> To: Francesco Romano
> Cc: r-sig-mixed-models using r-project.org
> Subject: Re: [R-sig-ME] Fwd: Question about non-significant interactions
>
> Dear Francesco,
>
> To answer your question, you should convert your hypothesis in a set of
> linear contrasts and test those.
>
> Best regards,
>
> ir. Thierry Onkelinx
> Statisticus / Statistician
>
> Vlaamse Overheid / Government of Flanders
> INSTITUUT VOOR NATUUR- EN BOSONDERZOEK / RESEARCH INSTITUTE FOR NATURE AND
> FOREST
> Team Biometrie & Kwaliteitszorg / Team Biometrics & Quality Assurance
> thierry.onkelinx using inbo.be
> Havenlaan 88 bus 73, 1000 Brussel
> www.inbo.be
>
>
> ///////////////////////////////////////////////////////////////////////////////////////////
> To call in the statistician after the experiment is done may be no more
> than asking him to perform a post-mortem examination: he may be able to say
> what the experiment died of. ~ Sir Ronald Aylmer Fisher
> The plural of anecdote is not data. ~ Roger Brinner
> The combination of some data and an aching desire for an answer does not
> ensure that a reasonable answer can be extracted from a given body of data.
> ~ John Tukey
>
> ///////////////////////////////////////////////////////////////////////////////////////////
>
> <https://www.inbo.be>
>
>
> Op di 9 jul. 2019 om 23:25 schreef Francesco Romano <fbromano77 using gmail.com
> >:
>
> > ---------- Forwarded message ---------
> > From: Francesco Romano <fbromano77 using gmail.com>
> > Date: Tue, Jul 9, 2019 at 11:24 PM
> > Subject: Re: [R-sig-ME] Question about non-significant interactions
> > To: Fox, John <jfox using mcmaster.ca>
> >
> >
> > Dear John,
> >
> > Thanks for the reply. One of my research entails examining the
> relationship
> > between 3 groups of speakers, the 3 levels of the group categorical
> > variable previously mentioned, and two tasks. One prediction is that one
> > group will perform better than other groups on one test but not the
> other.
> >
> > I fit a maximal model using the bglmr function as shown previously, then
> > used car::Anova to determine main effects. My understanding from previous
> > interaction with you precisely here on r-sig-me is that the function
> works
> > as a form of shortcut to the traditional way of model-fitting/ reduction
> > via the function anova() comparing models, eliminating terms one at a
> time.
> >
> > I hope this is clearer now and yes, the question is more of a statistical
> > one than an R one, even though I suspect the mixed-effect aspect of the
> > regression may be relevant to answering it.
> >
> > Frank
> >
> > Tue, Jul 9, 2019 at 11:03 PM Fox, John <jfox using mcmaster.ca> wrote:
> >
> > > Dear Francesco,
> > >
> > > I didn't entirely follow your question and I expect that to answer it,
> it
> > > would be necessary to know more about what your research entails. As
> you
> > > imply, this seems to be more a statistics question than an R question.
> > It's
> > > also not clear to me what function you used to fit the mixed-effects
> > > logistic regression.
> > >
> > > But I did notice that you're apparently using Anova() for type-III
> tests
> > > with the default contr.treatment() coding for factors. The main-effect
> > > tests that result are not sensible. As it says in ?Anova:
> > >
> > > "Warning
> > > Be careful of type-III tests: For a traditional multifactor ANOVA model
> > > with interactions, for example, these tests will normally only be
> > sensible
> > > when using contrasts that, for different terms, are orthogonal in the
> > > row-basis of the model, such as those produced by contr.sum,
> contr.poly,
> > or
> > > contr.helmert, but not by the default contr.treatment. In a model that
> > > contains factors, numeric covariates, and interactions, main-effect
> tests
> > > for factors will be for differences over the origin. In contrast (pun
> > > intended), type-II tests are invariant with respect to (full-rank)
> > contrast
> > > coding. If you don't understand this issue, then you probably shouldn't
> > use
> > > Anova for type-III tests."
> > >
> > > I hope that this is of some help,
> > > John
> > > -----------------------------
> > > John Fox, Professor Emeritus
> > > McMaster University
> > > Hamilton, Ontario
> > > Canada L8S 4M4
> > > web: socserv.mcmaster.ca/jfox
> > >
> > >
> > > ________________________________________
> > > From: R-sig-mixed-models [r-sig-mixed-models-bounces using r-project.org] on
> > > behalf of Francesco Romano [fbromano77 using gmail.com]
> > > Sent: July 9, 2019 9:49 AM
> > > To: r-sig-mixed-models using r-project.org
> > > Subject: [R-sig-ME] Question about non-significant interactions
> > >
> > > Dear all,
> > >
> > >
> > > I have more of a theoretical than practical question for you. The
> model I
> > > am using has two IVs, group (3 levels) and task (2 levels), and a
> > > categorical DV (correct versus incorrect), hence logistic regression.
> > > Random effects for subjects and items, as well as slopes for group by
> > item
> > > and task by subject.
> > >
> > > I am interested in the effect of belonging any of three groups, the
> > levels
> > > of the group IV, in order to test some a priori predictions. The
> bayesian
> > > wrapper is to help the model converge.
> > >
> > > Here is the output:
> > >
> > > > summary(paper2analysis1)
> > > Cov prior : item ~ wishart(df = 5.5, scale = Inf, posterior.scale =
> cov,
> > > common.scale = TRUE)
> > > : Participant ~ wishart(df = 4.5, scale = Inf,
> > posterior.scale =
> > > cov, common.scale = TRUE)
> > > Prior dev : 6.9466
> > >
> > > Generalized linear mixed model fit by maximum likelihood (Laplace
> > > Approximation) ['bglmerMod']
> > > Family: binomial ( logit )
> > > Formula: correctness ~ task * group + (1 + task | Participant) + (1 +
> > > group | item)
> > > Data: data
> > > Control: glmerControl(optimizer = "bobyqa")
> > >
> > > AIC BIC logLik deviance df.resid
> > > 3857.8 3957.2 -1913.9 3827.8 5570
> > >
> > > Scaled residuals:
> > > Min 1Q Median 3Q Max
> > > -2.0196 -0.3744 -0.2312 -0.1368 6.9534
> > >
> > > Random effects:
> > > Groups Name Variance Std.Dev. Corr
> > > item (Intercept) 1.1266 1.0614
> > > groupL2 0.1311 0.3620 -0.12
> > > groupNS 0.2029 0.4504 -0.31 0.17
> > > Participant (Intercept) 0.7582 0.8708
> > > taskpriming 1.2163 1.1029 -0.77
> > > Number of obs: 5585, groups: item, 219; Participant, 46
> > >
> > > Fixed effects:
> > > Estimate Std. Error z value Pr(>|z|)
> > > (Intercept) -2.49187 0.28318 -8.800 < 2e-16 ***
> > > taskpriming 1.30911 0.37367 3.503 0.000459 ***
> > > groupL2 -0.04042 0.38322 -0.105 0.916005
> > > groupNS -1.01144 0.36607 -2.763 0.005727 **
> > > taskpriming:groupL2 0.04305 0.48693 0.088 0.929544
> > > taskpriming:groupNS -0.04942 0.46034 -0.107 0.914506
> > > ---
> > > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> > >
> > > Correlation of Fixed Effects:
> > > (Intr) tskprm gropL2 gropNS tsk:L2
> > > taskpriming -0.733
> > > groupL2 -0.660 0.482
> > > groupNS -0.693 0.507 0.509
> > > tskprmng:L2 0.499 -0.632 -0.755 -0.386
> > > tskprmng:NS 0.530 -0.676 -0.390 -0.750 0.508
> > >
> > > The model was then subjected to car::Anova for ANOVA type III analysis
> > with
> > > the following output:
> > >
> > > > car::Anova(paper2analysis1, type = "III")
> > > Analysis of Deviance Table (Type III Wald chisquare tests)
> > >
> > > Response: correctness
> > > Chisq Df Pr(>Chisq)
> > > (Intercept) 77.4344 1 < 2.2e-16 ***
> > > task 12.2737 1 0.0004594 ***
> > > group 9.9237 2 0.0070000 **
> > > task:group 0.0391 2 0.9806462
> > > ---
> > > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> > >
> > > I am not sure how to interpret the non-significant interaction in this
> > > case. Does this mean that, although simple effects exist at group level
> > > within one particular task or at task level within one particular
> group,
> > I
> > > lack sufficient power to conclude those effects are real? If I look at
> > the
> > > simple effects, I do indeed find such effects but am not sure how to
> > > interpret them against the lack of a main interaction. At a practical
> > > level, the interaction, rather than the main effects, is the most
> > important
> > > part of the analysis.
> > >
> > > Thank you in advance for any advice.
> > >
> > > Francesco
> > >
> > >
> > >
> > >
> > >
> > > Best,
> > >
> > > Frank
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-mixed-models using r-project.org mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> > >
> > --
> > Inviato da Gmail Mobile
> > --
> > Inviato da Gmail Mobile
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
[[alternative HTML version deleted]]
More information about the R-sig-mixed-models
mailing list