[R-meta] Comparing heterogeneity explained from different meta-regression models with the same moderator

[Viechtbauer, Wolfgang (NP)]

Dear Zhenyu,

Making inferences about the (pseudo) R^2 statistic is a bit tricky, since it is computed by essentially comparing the tau^2 estimate from the random-effects model and the mixed-effects meta-regression model, so two different models. Hence, computing some kind of standard error for R^2 would be quite tricky. To get a CI for R^2, one could use bootstrapping. For example:

library(metafor)
library(boot)

dat <- dat.bangertdrowns2004
dat <- dat[!is.na(dat$minutes),]

rma(yi, vi, mods = ~ minutes, data=dat)  

boot.func <- function(dat, indices) {
   res <- try(suppressWarnings(rma(yi, vi, mods = ~ minutes, data=dat[indices,])), silent=TRUE)
   ifelse(inherits(res, "try-error"), NA, res$R2)
}

set.seed(1234)
res.boot <- boot(dat, boot.func, R=10000)
res.boot

hist(res.boot$t, breaks=50)

boot.ci(res.boot)

You will see that the bootstrap distribution of R^2 is quite weird, so I would recommend the percentile CI (which also avoids the possibility of getting bounds below 0% or above 100%). And the bootstrap CIs are super wide. That's to be expected when k is 'only' 24. Precise estimates of R^2 require quite large datasets.

The same applies if one would want to compare R^2 from two different models. The only way I could think of doing this is to use bootstrapping. How to set this up requires some thought. Are the two outcomes coming from the same set of studies? In that case, one might want to sample studies, not estimates. Then calculate R^2 for the two different models, take the difference, and repeat. If the data are in wide format (so the two outcomes are different columns and so you have one row per study), then this should work:

boot.func <- function(dat, indices) {
   res1 <- try(suppressWarnings(rma(y1i, v1i, mods = ~ x, data=dat[indices,])), silent=TRUE)
   res2 <- try(suppressWarnings(rma(y2i, v2i, mods = ~ x, data=dat[indices,])), silent=TRUE)
   ifelse(inherits(res1, "try-error") || inherits(res2, "try-error"), NA, res1$R2-res2$R2)
}

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Yuan, Zhenyu
>Sent: Tuesday, 20 September, 2022 2:32
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Comparing heterogeneity explained from different meta-
>regression models with the same moderator
>
>Greetings,
>
>I ran into a question regarding the heterogeneity explained by a meta-regression
>model and I would appreciate any insights or guidance you all can provide.
>
>Specifically, I have two meta-regression models (analyzed via the "metafor"
>package) whereby the outcome variables are different but the moderator is the
>same. I reported the heterogeneity explained in these two models in my research.
>I was asked whether there would be any way to formally test the difference in the
>amount of heterogeneity explained between these two models. My understanding is
>that this would be difficult given that the outcome variables are different.
>However, I wanted to reach out to the listserv and see if my understanding is
>correct.
>
>I'd greatly appreciate any guidance in this regard. Thank you for your time.
>
>Best,
>
>Zhenyu Yuan
>Assistant Professor
>Department of Managerial Studies
>University of Illinois Chicago