[R-meta] Recommendations on quantifying heterogeneity in three-level meta-regression

[Viechtbauer Wolfgang (SP)]

Hi Jens,

Maybe Mike can also chime in here, but I don't think metaSEM and metafor do anything fundamentally different here. Consider this example:

library(metafor)
library(metaSEM)

dat <- get(data(dat.konstantopoulos2011))
dat$year <- dat$year - 1976

res0 <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat, method="ML")
res0

res1 <- rma.mv(yi, vi, mods = ~ year, random = ~ 1 | district/school, data=dat, method="ML")
res1

### pseudo R^2 values
round(max(0, (res0$sigma2[1] - res1$sigma2[1]) / res0$sigma2[1]), 4) ### level 3
round(max(0, (res0$sigma2[2] - res1$sigma2[2]) / res0$sigma2[2]), 4) ### level 2

res2 <- meta3(yi, vi, x=year, cluster=district, data=dat)
summary(res2)

It takes a bit of time to match things up, but you will find that the results are pretty much the same.

I agree with Mike that one needs to be cautious about computing 'I^2-like' measures for such models. The code on the metafor website (i.e., http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate#multilevel_models) is pretty much a logical extension of one way of how I^2 can be computed for standard RE models and it is there because I have received many questions on how to compute I^2 for more complex models, but there is no consensus yet as to how this should be done. Heck, there isn't even consensus on how to compute I^2 for standard RE models!

Also, the interpretation of I^2-like measures computed based on models that include moderators/predictors is somewhat nebulous (to the point that one could question whether it is even useful to compute such things).

Best,
Wolfgang

-- 
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and 
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD 
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com 

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>project.org] On Behalf Of Jens Schüler
>Sent: Thursday, 16 November, 2017 14:26
>To: r-sig-meta-analysis at r-project.org
>Subject: [R-meta] Recommendations on quantifying heterogeneity in three-
>level meta-regression
>
>Dear all,
>
>I am currently working on a three-level meta-regression and I am a bit
>unsure about how to properly/adequately report heterogeneity in such
>mixed-effect models (whereas it is pretty clear cut for regular "two-
>level" meta-regressions e.g. the recent best practice recommendations of
>Gonzalez-Mule & Aguinis 2017
>(http://journals.sagepub.com/doi/abs/10.1177/0149206317710723)
>
>Currently the metafor rma.mv function provides us with variance estimates
>for both levels and a Q-test on residual heterogeneity. Moreover, we are
>provided with code on how to calculate I2 in multilevel models, which I
>did (http://www.metafor-
>project.org/doku.php/tips:i2_multilevel_multivariate#multilevel_models).
>
>Turning towards the metaSEM package of Cheung, he reports variance
>estimates (no predictors vs. predictors) for both levels and R2
>(https://courses.nus.edu.sg/course/psycwlm/internet/metasem/3level.html#m
>odel-1-type-as-a-covariate). However, not I2. In his book (Cheung 2015)
>reports, on page 186, some formulas on how to calculate I2 and ICC in
>three-level models but ends with a cautionary note that we lack further
>insight in order to make reliable statements on that matter.
>
>Hence, I am a bit left in the dark in terms of what is currently
>appropriate (or how to defend such choices) and would like to pick your
>brains on that issue.
>
>Best regards
>Jens