[R-meta] [EXT] Re: Interpreting meta-regression results for dummy-coded variables

Tue Jun 21 08:10:02 CEST 2022

Wolfgang,

Thank you for sharing the link on the discrepancies. It is quite helpful.

James,

Indeed, I am using RVE, so this part of it is especially useful to have. I appreciate your time and explanation on it.

Selcuk Acar, Ph.D.
Associate Professor
Department of Educational Psychology
University of North Texas
________________________________
From: James Pustejovsky <jepusto using gmail.com>
Sent: Monday, June 20, 2022 9:58 AM
To: Michael Dewey <lists using dewey.myzen.co.uk>
Cc: Acar, Selcuk <Selcuk.Acar using unt.edu>; r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] [EXT] Re: Interpreting meta-regression results for dummy-coded variables

Hi Selcuk,

I think the key issue behind Michael's response was how to draw an inference about whether the moderator explains variation in the outcome, not about whether this was done in the context of a univariate analysis or a meta-regression analysis involving multiple predictors.

Testing a categorical predictor with more than two levels can be done as part of a meta-regression involving other predictors as well. It's just that the results of such a test aren't automatically displayed in the meta-regression output---instead, you have to ask for them. There's a few ways to do so. I'll demonstrate with a meta-regression on the bcg data, following the post that Wolfgang linked to in his reply.:
library(metafor)
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
res <- rma(yi, vi, mods = ~ year + ablat + factor(alloc), data=dat)

One is to use metafor::anova() to perform a Q test, as demonstrated in Wolfgang's post:
anova(res, btt = 4:5)
anova(res, btt = "alloc") # equivalent to the above, but using a regular expression

Alternately, you can use the btt argument in rma.uni() or rma.mv<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Frma.mv%2F&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8BOX0z1Y0Zg2rmAUB1iziH80BLYItex30JyH3Ag4YyE%3D&reserved=0>() to specify a set of coefficients to test (using the same type of Q-test). If you specify a btt argument, then the results will be reported in the metafor output under the Q-M test. See ?rma.uni in the section titled "Omnibus Tests of Moderators."
rma(yi, vi, mods = ~ year + ablat + factor(alloc), data=dat, btt = "alloc")

A further alternative, if you're using robust variance estimation, is to use the Wald_test() function from clubSandwich. Here' you'll need to specify a constraint matrix that describes which coefficients you want to test. The constrain_zero() helper function makes this easier, with a syntax very similar to the btt argument in metafor:
Wald_test(res, vcov = "CR2", cluster = dat$author, constraints = constrain_zero(4:5))
Wald_test(res, vcov = "CR2", cluster = dat$author, constraints = constrain_zero("alloc", reg_ex = TRUE))

One thing to watch out for here is that the coding of the constraint matrix depends on whether you're using a reference level for the categorical moderator. That's how the above example was coded (the reference level is 'alternate'). If you've instead specified a model with separate intercepts, then you can use the constrain_equal() helper function to construct a constraint matrix:
res_intercepts <- rma(yi, vi, mods = ~ 0 + factor(alloc) + year + ablat, data=dat)
Wald_test(res_intercepts, vcov = "CR2", cluster = dat$author, constraints = constrain_equal(1:3))
Wald_test(res_intercepts, vcov = "CR2", cluster = dat$author, constraints = constrain_equal("alloc", reg_ex = TRUE))

The difference in coding is purely a matter of preference/aesthetics. It will not matter for the test results. (If it does make a difference in the p-values, then you know some part of the code must be wrong.)

James

> Michael,
>
> Thanks for your response--I am glad to have it corrected.
>
> I do not do univariate analyses when I do meta-regression, but with this
> perspective, they could help me figure if a moderator is significant or
> not rather than a specific pair of categories is significantly from each
> other. I do not do it because meta-regression provides more stringent
> evidence than univariate analyses and when they conflict, I would still
> go with the meta-regression results for interpretation. So, I question
> the usefulness of such univariate analyses beyond checking if a
> moderator is significant whose sub-categories turned out significant in
> meta-regression.
>
> Am I correct in my thinking/understanding of it?
>
> Selcuk Acar, Ph.D.
> Associate Professor
> Department of Educational Psychology
> University of North Texas
> ------------------------------------------------------------------------
> *From:* Michael Dewey <lists using dewey.myzen.co.uk<mailto:lists using dewey.myzen.co.uk>>
> *Sent:* Sunday, June 19, 2022 7:30 AM
> *To:* Acar, Selcuk <Selcuk.Acar using unt.edu<mailto:Selcuk.Acar using unt.edu>>;
> r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org> <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>>
> *Subject:* [EXT] Re: [R-meta] Interpreting meta-regression results for
> dummy-coded variables
> Coments in-line
>
> On 18/06/2022 23:14, Acar, Selcuk wrote:
>> Hi,
>>
>> I ran a meta-regression in metafor package with both continuous and dummy-coded moderators. In some of the moderators, when we had only one dummy-code significant, we interpreted this as this moderator with several categories being significant. For example,  we had "participant group" moderator consisting of  "elementary"
> "middle" "high" and "undergraduate" categories, and used
> "undergraduates" as the reference group. We thought this moderator would
> be significant even when only one of dummy codes "undergraduates vs
> elementary" is significant without a separate test (linear hypothesis
> testing).
>>
>> One of the reviewers provided the following feedback:
>>
>> "Because of the dummy-coding these coefficients are differences in Fisher-z-transformed correlations between the coded category and the reference category. Hence, this reporting could be more accurately reflect this. In addition, these tests of coefficients  are not a substitute for an overall test of the moderator. In other
> words, the fact that one coefficient related to a moderator is
> significant, does not imply that the moderator is significant. For
> example, an overall test for Index of Creativity can be non-significant
> even when a single coefficient such as the one for flexibility vs.
> fluency is significant. Overall, moderator tests could be done by means
> of linear hypothesis testing, for example."
>>
>
> That is correct. an overall test of the moderator is needed.
>
>> In my opinion, running separate tests for each moderator kills the point of a meta-regression, and meta-regression should be the basis of the interpretation including these dummy-coded variables. I thought a categorical moderator would be significant even  when one of the dummy codes turn out significant.
>>
>
> I am afraid your thought is a mis-conception, albeit a common one.
>
> Michael
>
>> Who is correct here? Is there a good source that I could cite\?
>>
>> I would appreciate input on this.
>>
>> Selcuk Acar, Ph.D.
>> Associate Professor
>> Department of Educational Psychology
>> University of North Texas
>>
>>        [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org<mailto:R-sig-meta-analysis using r-project.org>
>> To manage your subscription to this mailing list, go to:
>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UPv%2B7wy%2Bpfu5%2B2pGC0iF8Dn6lvi5qDSRE1EBxN7iNpU%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IF9eBhNCxluERH7vd3XN0uREQdktJ7Lp1Hv238Fnxsw%3D&reserved=0>
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UPv%2B7wy%2Bpfu5%2B2pGC0iF8Dn6lvi5qDSRE1EBxN7iNpU%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IF9eBhNCxluERH7vd3XN0uREQdktJ7Lp1Hv238Fnxsw%3D&reserved=0>>
>>
>
> --
> Michael
> https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gYh189T0Z1gkDf%2FnovOR3FDro5dGysGQr5M5qXk68fc%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8vR2GThLuxJRXkOLwSQymm4K4DuQ%2Bw3AXw9pPaX7P3M%3D&reserved=0>
> <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7C322f70b684c046ff8ae108da51ef8f4a%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637912386722870865%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=gYh189T0Z1gkDf%2FnovOR3FDro5dGysGQr5M5qXk68fc%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8vR2GThLuxJRXkOLwSQymm4K4DuQ%2Bw3AXw9pPaX7P3M%3D&reserved=0>>
>
>
> ------------------------------------------------------------------------
> AVG logo <https://www.avg.com/internet-security<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.avg.com%2Finternet-security&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3%2BHCKUCLYouk9wSXMMCphredC7270uDMiPj4RKBYHig%3D&reserved=0>>
>
> This email has been checked for viruses by AVG antivirus software.
> www.avg.com<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com%2F&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bzFeiErz0dsMA2%2FaNprBmz9zBCQShUYFQ2JD09681cE%3D&reserved=0> <https://www.avg.com/internet-security<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.avg.com%2Finternet-security&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3%2BHCKUCLYouk9wSXMMCphredC7270uDMiPj4RKBYHig%3D&reserved=0>>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

--
Michael
http://www.dewey.myzen.co.uk/home.html<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.dewey.myzen.co.uk%2Fhome.html&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8vR2GThLuxJRXkOLwSQymm4K4DuQ%2Bw3AXw9pPaX7P3M%3D&reserved=0>

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org<mailto:R-sig-meta-analysis using r-project.org>
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-sig-meta-analysis&data=05%7C01%7CSelcuk.Acar%40unt.edu%7Cea9bdddabd444c443a6708da52cd5d29%7C70de199207c6480fa318a1afcba03983%7C0%7C0%7C637913339245641069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=IF9eBhNCxluERH7vd3XN0uREQdktJ7Lp1Hv238Fnxsw%3D&reserved=0>

	[[alternative HTML version deleted]]