[R-meta] confidence interval for I^2 in multilevel model

[Viechtbauer Wolfgang (SP)]

Hi Kyle,

With confint(), you can get CIs for variance components. You can plug the CI bounds for a variance component into the equation for I^2 to obtain a CI for I^2. Here is an example:

dat <- get(data(dat.konstantopoulos2011))

res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat)
res

sav <- confint(res)
sav

W <- diag(1/dat$vi)
X <- model.matrix(res)
P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
100 * res$sigma2 / (res$sigma2 + (res$k-res$p)/sum(diag(P))) ### district and school level I^2
100 * sav[[1]]$random[1,2:3] / (sav[[1]]$random[1,2:3] + (res$k-res$p)/sum(diag(P))) ### CI for district-level I^2
100 * sav[[2]]$random[1,2:3] / (sav[[2]]$random[1,2:3] + (res$k-res$p)/sum(diag(P))) ### CI for the school-level I^2

100 * sum(res$sigma2) / (sum(res$sigma2) + (res$k-res$p)/sum(diag(P))) ### total I^2

But you cannot use this approach to get a CI for the total I^2. For this, you have to use the 'multivariate parameterization' of this model (see: http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011). So:

res <- rma.mv(yi, vi, random = ~ factor(study) | district, data=dat)
res

sav <- confint(res)
sav

W <- diag(1/dat$vi)
X <- model.matrix(res)
P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
100 * res$tau2 / (res$tau2 + (res$k-res$p)/sum(diag(P))) ### total I^2
100 * sav$random[1,2:3] / (sav$random[1,2:3] + (res$k-res$p)/sum(diag(P))) ### CI for the total I^2

But you are using 'random = ~ 1 | ID', which does not give you a multilevel model. Are you maybe making this mistake?

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#a_common_mistake_in_the_three-level_model

Also, 'struct' is only relevant when you have something like a 'random = ~ inner | outer' structure, so struct="DIAG" has no effect.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Porter, Kyle
Sent: Wednesday, August 23, 2017 22:09
To: 'r-sig-meta-analysis at r-project.org'
Subject: [R-meta] confidence interval for I^2 in multilevel model

Hello all,

I am using metafor to calculate an I^2 analogue for a multilevel random effects meta-regression as described at http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate

I would like help in calculating a 95% confidence interval for this derivation of I^2.

Specifically, I am using the code taken below from the website to generalize the concept of I^2 to a clustered/multilevel model.

(the text below is taken from http://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate):
Based on the discussion above, it is now very easy to generalize the concept of I2 to such a model (see also Nakagawa & Santos, 2012). That is, we can first compute:

W <- diag(1/dat$vi)
X <- model.matrix(res)
P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
100 * sum(res$sigma2) / (sum(res$sigma2) + (res$k-res$p)/sum(diag(P)))

My specific application is as follows (the outcome is a single proportion):

PPV_prevdat <- escalc(measure="PAS", data=PPV_prev, xi=TP, ni=posTest)

mvregAll <- rma.mv(data=PPV_prevdat, yi, vi, method="REML", random = ~ 1 | ID, struct="DIAG", mods = ~ factor(Bethesda) + factor(Blinded) + prev +factor(NRAS12_13) + factor(HRAS12_13) + factor(KRAS61))

w <- diag(1/PPV_prevdat$vi)
x <- model.matrix(mvregAll)
P <- w - w %*% x %*% solve(t(x) %*% w %*% x) %*% t(x) %*% w
100 * sum(mvregAll$sigma2) / (sum(mvregAll$sigma2) + (mvregAll$k-mvregAll$p)/sum(diag(P)))

Any help/suggestion is appreciated.

Thank you,
Kyle Porter