[R-meta] I^2 results with small sampling variance

[Viechtbauer Wolfgang (SP)]

For a model without moderators, I^2 estimates how much of the total variance (which in this case is composed of study level variance, effect size level variance, and sampling variance -- where for latter, we plug in a sort of average/typical value) is due to study and effect size level variance. Or we can compute separate I^2 values for study level variance / total variance and effect size level variance / total variance.

When including moderators, the interpretation of I^2 becomes more tricky. In essence, any accounted for variance is removed from the computation, so we are looking at unaccounted for study and effect size level variance / total unaccounted for variance. That value might actually be larger than the I^2 from a model without moderators.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Martineau, Roger
Sent: Friday, August 18, 2017 16:08
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] I^2 results with small sampling variance

Dear metafor users,

I downloaded the data set (i.e., p154-dataset.csv) from  Assinka and Wibbelink (2016) available from http://www.tqmp.org/RegularArticles/vol12-3/p154/p154.pdf

The data set contains 100 studies and I compared I^2 statistics from the model with no moderator (overall) and the one with 2 significant moderators (multiplemoderator).

1)      Model overall (Listing 5 in the publication) and I^2 statistics:

overall <- rma.mv(y, v, random = list(~ 1 | effectsizeID, ~ 1 | studyID), tdist=TRUE, data=dataset)

> # Calculation of I^2 statistic
> W <- diag(1/overall$vi)
> X <- model.matrix(overall)
> P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> 100 * sum(overall$sigma2) / # I^2 statistic
+   (sum(overall$sigma2) + (overall$k-overall$p)/sum(diag(P)))
[1] 93.058
> # Separation of total variance by cluster
> 100 * overall$sigma2 / (sum(overall$sigma2) + (overall$k-overall$p)/sum(diag(P)))
[1] 34.754 58.304

2)      Model multiplemoderator (Listing 5 in the publication) and I^2 statistics:

multiplemoderator <- rma.mv(y, v, mods = ~ pyear + typeovert + typecovert,
                            random =list(~ 1 | effectsizeID, ~ 1 | studyID),
                            tdist=TRUE, data=dataset)

> # Calculation of I^2 statistic
> W <- diag(1/multiplemoderator$vi)
> X <- model.matrix(multiplemoderator)
> P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> 100 * sum(multiplemoderator$sigma2) / # I^2 statistic
+   (sum(multiplemoderator$sigma2) + (multiplemoderator$k-multiplemoderator$p)/sum(diag(P)))
[1] 90.993
> # Separation of total variance by cluster
> 100 * multiplemoderator$sigma2 / (sum(multiplemoderator$sigma2) + (multiplemoderator$k-multiplemoderator$p)/sum(diag(P)))
[1] 33.079 57.913

Questions:

The sampling variance is small at 7% to 11% in these models.
What is the correct interpretation of I^2 results? And for the slight reduction in sampling variance despite the addition of significant moderators?
I get similar results from my own data set with a large number of studies despite the addition of several significant moderators in the model.

Thanks in advance,

Roger :)