[R-meta] Recommendations on quantifying heterogeneity in three-level meta-regression

[Jens Schüler]

Hi Wolfgang,

thank you very much for the honest clarification of the current situation on
this matter.
Spending a weekend pondering over these issues, I was really left unsure
about what is appropriate/applicable.
Now I have to see what reviewers expect or how they feel about this.


Best
Jens


-----Ursprüngliche Nachricht-----
Von: Viechtbauer Wolfgang (SP)
[mailto:wolfgang.viechtbauer at maastrichtuniversity.nl] 
Gesendet: Donnerstag, 16. November 2017 21:38
An: Jens Schüler <jens.schueler at wiwi.uni-kl.de>;
r-sig-meta-analysis at r-project.org
Betreff: RE: Recommendations on quantifying heterogeneity in three-level
meta-regression

Hi Jens,

Maybe Mike can also chime in here, but I don't think metaSEM and metafor do
anything fundamentally different here. Consider this example:

library(metafor)
library(metaSEM)

dat <- get(data(dat.konstantopoulos2011))
dat$year <- dat$year - 1976

res0 <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat,
method="ML")
res0

res1 <- rma.mv(yi, vi, mods = ~ year, random = ~ 1 | district/school,
data=dat, method="ML")
res1

### pseudo R^2 values
round(max(0, (res0$sigma2[1] - res1$sigma2[1]) / res0$sigma2[1]), 4) ###
level 3 round(max(0, (res0$sigma2[2] - res1$sigma2[2]) / res0$sigma2[2]), 4)
### level 2

res2 <- meta3(yi, vi, x=year, cluster=district, data=dat)
summary(res2)

It takes a bit of time to match things up, but you will find that the
results are pretty much the same.

I agree with Mike that one needs to be cautious about computing 'I^2-like'
measures for such models. The code on the metafor website (i.e.,
http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate#mult
ilevel_models) is pretty much a logical extension of one way of how I^2 can
be computed for standard RE models and it is there because I have received
many questions on how to compute I^2 for more complex models, but there is
no consensus yet as to how this should be done. Heck, there isn't even
consensus on how to compute I^2 for standard RE models!

Also, the interpretation of I^2-like measures computed based on models that
include moderators/predictors is somewhat nebulous (to the point that one
could question whether it is even useful to compute such things).

Best,
Wolfgang

--
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com 

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>project.org] On Behalf Of Jens Schüler
>Sent: Thursday, 16 November, 2017 14:26
>To: r-sig-meta-analysis at r-project.org
>Subject: [R-meta] Recommendations on quantifying heterogeneity in 
>three- level meta-regression
>
>Dear all,
>
>I am currently working on a three-level meta-regression and I am a bit 
>unsure about how to properly/adequately report heterogeneity in such 
>mixed-effect models (whereas it is pretty clear cut for regular "two- 
>level" meta-regressions e.g. the recent best practice recommendations 
>of Gonzalez-Mule & Aguinis 2017
>(http://journals.sagepub.com/doi/abs/10.1177/0149206317710723)
>
>Currently the metafor rma.mv function provides us with variance 
>estimates for both levels and a Q-test on residual heterogeneity. 
>Moreover, we are provided with code on how to calculate I2 in 
>multilevel models, which I did (http://www.metafor- 
>project.org/doku.php/tips:i2_multilevel_multivariate#multilevel_models).
>
>Turning towards the metaSEM package of Cheung, he reports variance 
>estimates (no predictors vs. predictors) for both levels and R2 
>(https://courses.nus.edu.sg/course/psycwlm/internet/metasem/3level.html
>#m odel-1-type-as-a-covariate). However, not I2. In his book (Cheung 
>2015) reports, on page 186, some formulas on how to calculate I2 and 
>ICC in three-level models but ends with a cautionary note that we lack 
>further insight in order to make reliable statements on that matter.
>
>Hence, I am a bit left in the dark in terms of what is currently 
>appropriate (or how to defend such choices) and would like to pick your 
>brains on that issue.
>
>Best regards
>Jens
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7675 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20171121/e5307def/attachment.p7s>