[R-meta] I2 interpretation for Multilevel meta-analysis with moderators

[Ivan Jukic]

Dear Wolfgang,

thank you for taking the time to explain this so nicely! I really appreciate it. 

Indeed, everything is defined and illustrated at this point, and I'll probably opt for I2 and R2 combination to avoid potential confusion among the readers. 

Cheers,
Ivan

From: Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
Sent: Thursday, 14 October 2021 10:45 PM
To: Ivan Jukic <ivan.jukic using aut.ac.nz>; r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
Subject: RE: I2 interpretation for Multilevel meta-analysis with moderators 
 
I can't answer your question (Is this correct?) at the moment. Instead, let me just explain again what I^2, R^2, and I^2_R (the latter denoting the I^2 from a model with moderators) represent.

Let's just go back to the case of a 'standard' random/mixed-effects models with no multilevel structure to keep things simple.

Draw a circle and then cut it up into three pieces of various sizes (i.e., draw a venn diagram). The three areas are:

(1) variability due to sampling variance
(2) heterogeneity accounted for by the moderators included in the model
(3) residual heterogeneity (i.e., heterogeneity that is not accounted for by the moderators)

One can think of tau^2 from the random-effects model (without any moderators) as the sum of (2) and (3) (i.e., the 'total' amount of heterogeneity). tau^2 from the mixed-effects meta-regression model (with the moderators) is (3). Therefore, tau^2 from the random-effects model minus tau^2 from the mixed-effects model is (2).

Then:

I^2   = 100 * ((2) + (3)) / ((1) + (2) + (3))
R^2   = 100 * (2) / ((2) + (3))
I^2_R = 100 * (3) / ((1) + (3))

So, I^2 estimates how much of the total amount of variability (the sum of (1), (2), and (3)) is due to heterogeneity.

R^2 estimates how much of the total amount of heterogeneity is accounted for by the moderators.

I^2_R estimates how much of the unaccounted for variability (the sum of (1) and (3)) is due to residual heterogeneity.

It is indeed the case that if (3) gets smaller, then R^2 increases and I^2_R decreases, but since the denominators of these statistic involve different quantities (beside (3)), they are only indirectly related to each other. And of all these, I still find I^2_R to be the least useful quantity to report.

And finally, an example to illustrate all of the above. Note that (1) is actually a bit of a tricky thing to quantify, since each study has a different sampling variance. So we compute an average/typical sampling variance for (1) and then use that for the computations. For the definitions above to be exactly true in the example below, I pretend that each study has the same sampling variance given by this somewhat odd-looking definition of the 'typical' sampling variance (vt).

dat <- dat.bcg
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat)

k <- nrow(dat)
wi <- 1/dat$vi
vt <- (k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
dat$vi <- vt

res1 <- rma(yi, vi, data=dat)
res1

res2 <- rma(yi, vi, mods = ~ ablat, data=dat)
res2

round(100 * res1$tau2 / (res1$tau2 + vt), 2)        # I^2
round(100 * (res1$tau2 - res2$tau2) / res1$tau2, 2) # R^2
round(100 * res2$tau2 / (vt + res2$tau2), 2)        # I^2_R

I will drop out of the discussion after this post. Everything is defined and illustrated at this point. Nothing wrong with reporting I^2_R, but I just don't find it very informative and I suspect most people will struggle to make sense of it. Heck, many people even misinterpret I^2 (as some kind measure of absolute heterogeneity), so chances that I^2_R gets correctly interpreted are quite slim.

Best,
Wolfgang