[R-meta] Differences in I^2 estimation between meta and metafor packages
Viechtbauer, Wolfgang (SP)
wo||g@ng@v|echtb@uer @end|ng |rom m@@@tr|chtun|ver@|ty@n|
Wed May 15 10:31:42 CEST 2019
Interesting, thanks for mentioning this, Mike. I compared this with metafor:
library(metaSEM)
library(metafor)
summary(meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"), intervals.type="LB"))
res <- rma(yi, vi, data=Hox02, method="ML")
(res$QE - (res$k - 1)) / res$QE
This would be the 'usual' definition, which yields 0.6168211. But that's not what is given by "I2 (Q statistic)". Instead, that is the same as what rma() uses, namely:
res$I2 / 100
which we can also compute manually with:
wi <- 1/res$vi
s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
res$tau2 / (res$tau2 + s2)
The harmonic and arithmetic mean definitions are:
res$tau2 / (res$tau2 + res$k/sum(wi))
res$tau2 / (res$tau2 + mean(res$vi))
Note that R_b is not the same as the arithmetic mean definition:
1/res$k * sum(res$tau2 / (res$vi + res$tau2))
So, yay, we are up to 4 definitions now :)
One other thing I noticed: The 95% likelihood-based CIs (I assume these are profile likelihood CIs) are different from what I obtain with metafor. For example:
Heterogeneity indices (I2) and their 95% likelihood-based CIs:
lbound Estimate ubound
Intercept1: I2 (Q statistic) 0.27625 0.60780 0.6749
Comparing this with metafor (where I switch to rma.mv() so I also get profile likelihood CIs; rma() computes the CIs for tau^2 and I^2 using the Q-profile method):
res <- rma.mv(yi, vi, random = ~ 1 | study, data=Hox02, method="ML")
ci <- confint(res)
wi <- 1/res$vi
s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
res$sigma2 / (res$sigma2 + s2)
ci$random[1,] / (ci$random[1,] + s2)
This yields:
estimate ci.lb ci.ub
0.6078023 0.2763288 0.8110635
So again the same estimate, but the lower bound is slightly different and the upper bound is totally different.
Looking at the CI for the variance component itself:
Estimate Std.Error lbound ubound z value Pr(>|z|)
Intercept1 0.579035 NA 0.365413 0.800250 NA NA
Tau2_1_1 0.131520 NA 0.034863 0.364294 NA NA
While confint(res) shows:
estimate ci.lb ci.ub
sigma^2 0.1315 0.0324 0.3643
sigma 0.3627 0.1800 0.6036
Now the lower bound is slightly different, but that might just be a numerical issue.
Any idea why the that upper bound for I^2 is so different above?
Best,
Wolfgang
-----Original Message-----
From: Mike Cheung [mailto:mikewlcheung using gmail.com]
Sent: Wednesday, 15 May, 2019 2:48
To: Viechtbauer, Wolfgang (SP)
Cc: Guido Schwarzer; Rushkin, Megan C; r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Differences in I^2 estimation between meta and metafor packages
As a follow-up of the discussion, the metaSEM package implements these definitions of the I^2 with the likelihood-based CIs on them.
library(metaSEM)
summary( meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"), intervals.type="LB") )
Heterogeneity indices (I2) and their 95% likelihood-based CIs:
lbound Estimate ubound
Intercept1: I2 (Q statistic) 0.27625 0.60780 0.6749
Intercept1: I2 (harmonic mean) 0.27713 0.60885 0.6766
Intercept1: I2 (arithmetic mean) 0.26209 0.59052 0.6431
A brief introduction is available here: https://books.google.com.sg/books?id=Pw7QBwAAQBAJ&pg=PA92&lpg=PA92&dq=%224.3.3+Quantifying+the+degree+of+the+heterogeneity+of+effect+sizes%22&source=bl&ots=zDjRbpiGL5&sig=ACfU3U10nYLvLi9idJ_QnXCKcasi8sDJ3g&hl=en&sa=X&ved=2ahUKEwjk3t6lppziAhXZbSsKHcEmA70Q6AEwAHoECAUQAQ#v=onepage&q=%224.3.3%20Quantifying%20the%20degree%20of%20the%20heterogeneity%20of%20effect%20sizes%22&f=false
--
---------------------------------------------------------------------
Mike W.L. Cheung Phone: (65) 6516-3702
Department of Psychology Fax: (65) 6773-1843
National University of Singapore
http://mikewlcheung.github.io/
---------------------------------------------------------------------
On Tue, May 14, 2019 at 6:26 PM Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Good points. I wrote up that FAQ because this question comes up quite a bit, since (by default, that is, with REML estimation) metafor does deviate a bit from how I^2 is typically computed (i.e., with 100*(Q - (k-1))/Q), so users are sometimes surprised by this.
And I fully agree that there isn't a *right* way of computing I^2, just different definitions. I actually find the way the 'typical' sampling variance is computed (which is implicitly used by both definitions) rather weird. Other definitions have been proposed (in particular, by Takkouche/Spiegelman/Crippa), such as:
I^2 = tau^2 / (tau^2 + s^2), where s^2 = k/sum(wi), where wi=1/vi (i.e., the haromic mean of the sampling variances)
or
I^2 = 1/k sum(tau^2 / (vi + tau^2)).
Those actually seem a bit more intuitive to me. The 'hetmeta' package provides those versions (called R_I and R_b).
Best,
Wolfgang
-----Original Message-----
From: Guido Schwarzer [mailto:sc using imbi.uni-freiburg.de]
Sent: Tuesday, 14 May, 2019 11:59
To: Viechtbauer, Wolfgang (SP); Rushkin, Megan C; r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Differences in I^2 estimation between meta and metafor packages
Am 13.05.19 um 10:45 schrieb Viechtbauer, Wolfgang (SP):
> Hi Megan,
>
> This is answered here:
>
> http://www.metafor-project.org/doku.php/faq#how_are_i_2_and_h_2_computed_i
This website nicely describes the differences in the calculation of I2
in metafor and meta. As this is written from the metafor perspective, I
would like to describe why meta uses the "other" definition. ;-)
Wolfgang mentions two advantages of the metafor implementation of I2:
1) "more general definition"
2) "values of I2 and H2 will be consistent with ... tau2"
I agree that the metafor implementation is more general as one gets a
different I2 value for each estimation method of tau2. However, the meta
implementation is also based on a generalization (to the situation in
which precisions differ between studies) - see section 3.3 in Higgins &
Thompons (2002).
On the other hand, while the metafor implementation guarantees
consistent estimates for I2 and tau2, the meta implementation guarantees
consistency of the I2 estimate and the test for heterogeneity (which -
like I2 - is based on Q and the number of studies).
In summary, one should know the differences in the estimation of I2
between metafor and meta, however, there is (in my opinion) no clear
"winner".
Best wishes, Guido
More information about the R-sig-meta-analysis
mailing list