[R-meta] [EXT] Re: Interpreting meta-regression results for dummy-coded variables

Mon Jun 20 16:58:29 CEST 2022

Hi Selcuk,

I think the key issue behind Michael's response was how to draw an
inference about whether the moderator explains variation in the
outcome, not about whether this was done in the context of a univariate
analysis or a meta-regression analysis involving multiple predictors.

Testing a categorical predictor with more than two levels can be done as
part of a meta-regression involving other predictors as well. It's just
that the results of such a test aren't automatically displayed in the
meta-regression output---instead, you have to ask for them. There's a few
ways to do so. I'll demonstrate with a meta-regression on the bcg data,
following the post that Wolfgang linked to in his reply.:
library(metafor)
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg,
data=dat.bcg)
res <- rma(yi, vi, mods = ~ year + ablat + factor(alloc), data=dat)

One is to use metafor::anova() to perform a Q test, as demonstrated in
Wolfgang's post:
anova(res, btt = 4:5)
anova(res, btt = "alloc") # equivalent to the above, but using a regular
expression

Alternately, you can use the btt argument in rma.uni() or rma.mv() to
specify a set of coefficients to test (using the same type of Q-test). If
you specify a btt argument, then the results will be reported in the
metafor output under the Q-M test. See ?rma.uni in the section titled
"Omnibus Tests of Moderators."
rma(yi, vi, mods = ~ year + ablat + factor(alloc), data=dat, btt = "alloc")

A further alternative, if you're using robust variance estimation, is to
use the Wald_test() function from clubSandwich. Here' you'll need to
specify a constraint matrix that describes which coefficients you want to
test. The constrain_zero() helper function makes this easier, with a syntax
very similar to the btt argument in metafor:
Wald_test(res, vcov = "CR2", cluster = dat$author, constraints =
constrain_zero(4:5))
Wald_test(res, vcov = "CR2", cluster = dat$author, constraints =
constrain_zero("alloc", reg_ex = TRUE))

One thing to watch out for here is that the coding of the constraint matrix
depends on whether you're using a reference level for the categorical
moderator. That's how the above example was coded (the reference level is
'alternate'). If you've instead specified a model with separate intercepts,
then you can use the constrain_equal() helper function to construct a
constraint matrix:
res_intercepts <- rma(yi, vi, mods = ~ 0 + factor(alloc) + year + ablat,
data=dat)
Wald_test(res_intercepts, vcov = "CR2", cluster = dat$author, constraints =
constrain_equal(1:3))
Wald_test(res_intercepts, vcov = "CR2", cluster = dat$author, constraints =
constrain_equal("alloc", reg_ex = TRUE))

The difference in coding is purely a matter of preference/aesthetics. It
will not matter for the test results. (If it does make a difference in the
p-values, then you know some part of the code must be wrong.)

James

> > Michael,
> >
> > Thanks for your response--I am glad to have it corrected.
> >
> > I do not do univariate analyses when I do meta-regression, but with this
> > perspective, they could help me figure if a moderator is significant or
> > not rather than a specific pair of categories is significantly from each
> > other. I do not do it because meta-regression provides more stringent
> > evidence than univariate analyses and when they conflict, I would still
> > go with the meta-regression results for interpretation. So, I question
> > the usefulness of such univariate analyses beyond checking if a
> > moderator is significant whose sub-categories turned out significant in
> > meta-regression.
> >
> > Am I correct in my thinking/understanding of it?
> >
> > Selcuk Acar, Ph.D.
> > Associate Professor
> > Department of Educational Psychology
> > University of North Texas
> > ------------------------------------------------------------------------
> > *From:* Michael Dewey <lists using dewey.myzen.co.uk>
> > *Sent:* Sunday, June 19, 2022 7:30 AM
> > *To:* Acar, Selcuk <Selcuk.Acar using unt.edu>;
> > r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> > *Subject:* [EXT] Re: [R-meta] Interpreting meta-regression results for
> > dummy-coded variables
> > Coments in-line
> >
> > On 18/06/2022 23:14, Acar, Selcuk wrote:
> >> Hi,
> >>
> >> I ran a meta-regression in metafor package with both continuous and
> dummy-coded moderators. In some of the moderators, when we had only one
> dummy-code significant, we interpreted this as this moderator with several
> categories being significant. For example,  we had "participant group"
> moderator consisting of  "elementary"
> > "middle" "high" and "undergraduate" categories, and used
> > "undergraduates" as the reference group. We thought this moderator would
> > be significant even when only one of dummy codes "undergraduates vs
> > elementary" is significant without a separate test (linear hypothesis
> > testing).
> >>
> >> One of the reviewers provided the following feedback:
> >>
> >> "Because of the dummy-coding these coefficients are differences in
> Fisher-z-transformed correlations between the coded category and the
> reference category. Hence, this reporting could be more accurately reflect
> this. In addition, these tests of coefficients  are not a substitute for an
> overall test of the moderator. In other
> > words, the fact that one coefficient related to a moderator is
> > significant, does not imply that the moderator is significant. For
> > example, an overall test for Index of Creativity can be non-significant
> > even when a single coefficient such as the one for flexibility vs.
> > fluency is significant. Overall, moderator tests could be done by means
> > of linear hypothesis testing, for example."
> >>
> >
> > That is correct. an overall test of the moderator is needed.
> >
> >> In my opinion, running separate tests for each moderator kills the
> point of a meta-regression, and meta-regression should be the basis of the
> interpretation including these dummy-coded variables. I thought a
> categorical moderator would be significant even  when one of the dummy
> codes turn out significant.
> >>
> >
> > I am afraid your thought is a mis-conception, albeit a common one.
> >
> > Michael
> >
> >> Who is correct here? Is there a good source that I could cite\?
> >>
> >> I would appreciate input on this.
> >>
> >> Selcuk Acar, Ph.D.
> >> Associate Professor
> >> Department of Educational Psychology
> >> University of North Texas
> >>
> >>        [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> >> To manage your subscription to this mailing list, go to:
> >>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UPv%2B7wy%2Bpfu5%2B2pGC0iF8Dn6lvi5qDSRE1EBxN7iNpU%3D&reserved=0
> > <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UPv%2B7wy%2Bpfu5%2B2pGC0iF8Dn6lvi5qDSRE1EBxN7iNpU%3D&reserved=0
> >
> >>
> >
> > --
> > Michael
> >
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gYh189T0Z1gkDf%2FnovOR3FDro5dGysGQr5M5qXk68fc%3D&reserved=0
> > <
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gYh189T0Z1gkDf%2FnovOR3FDro5dGysGQr5M5qXk68fc%3D&reserved=0
> >
> >
> >
> > ------------------------------------------------------------------------
> > AVG logo <https://www.avg.com/internet-security>
> >
> > This email has been checked for viruses by AVG antivirus software.
> > www.avg.com <https://www.avg.com/internet-security>
> >
> >
> > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> --
> Michael
> http://www.dewey.myzen.co.uk/home.html
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]