[R-meta] Differences in I^2 estimation between meta and metafor packages
Mike Cheung
m|kew|cheung @end|ng |rom gm@||@com
Thu May 16 11:41:07 CEST 2019
Hi, Wolfgang.
Thanks for bringing these issues to my attention.
For the first one, you are correct that I do not use the Q statistic to
calculate the I^2. I use the "typical" v defined in Higgins and Thompson
(2002, Equation 6). There are two reasons. First, it is more consistent
with the other definitions by changing the calculations of the "typical" v
(the harmonic and arithmetic mean). Second, CIs can be calculated from the
estimated tau^2. The limitation is that it depends on the estimation method
because tau^2 is involved.
Regarding the second question, I am also surprised that the upper bounds
are different. The metaSEM does not use the inversion approach to calculate
the CIs, i.e., calculating the CIs on tau^2 first and then converting them
to the CIs on I^2. It calculates the likelihood-based CI (Neale & Miller,
1997, Pek & Wu, 2015) on I^2 directly. I need to carefully check them
again. I will get back to you soon.
Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a
meta‐analysis. *Statistics in Medicine*, *21*(11), 1539–1558.
https://doi.org/10.1002/sim.1186
Neale, M. C., & Miller, M. B. (1997). The use of likelihood-based
confidence intervals in genetic models. *Behavior Genetics*, *27*(2),
113–120. https://doi.org/10.1023/A:1025681223921
Pek, J., & Wu, H. (2015). Profile likelihood-based confidence intervals and
regions for structural equation models. *Psychometrika*, *80*(4),
1123–1145. https://doi.org/10.1007/s11336-015-9461-1
<https://doi.org/10.1023/A:1025681223921>
Best,
Mike
On Wed, May 15, 2019 at 4:32 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> Interesting, thanks for mentioning this, Mike. I compared this with
> metafor:
>
> library(metaSEM)
> library(metafor)
>
> summary(meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"),
> intervals.type="LB"))
>
> res <- rma(yi, vi, data=Hox02, method="ML")
> (res$QE - (res$k - 1)) / res$QE
>
> This would be the 'usual' definition, which yields 0.6168211. But that's
> not what is given by "I2 (Q statistic)". Instead, that is the same as what
> rma() uses, namely:
>
> res$I2 / 100
>
> which we can also compute manually with:
>
> wi <- 1/res$vi
> s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
> res$tau2 / (res$tau2 + s2)
>
> The harmonic and arithmetic mean definitions are:
>
> res$tau2 / (res$tau2 + res$k/sum(wi))
> res$tau2 / (res$tau2 + mean(res$vi))
>
> Note that R_b is not the same as the arithmetic mean definition:
>
> 1/res$k * sum(res$tau2 / (res$vi + res$tau2))
>
> So, yay, we are up to 4 definitions now :)
>
> One other thing I noticed: The 95% likelihood-based CIs (I assume these
> are profile likelihood CIs) are different from what I obtain with metafor.
> For example:
>
> Heterogeneity indices (I2) and their 95% likelihood-based CIs:
> lbound Estimate ubound
> Intercept1: I2 (Q statistic) 0.27625 0.60780 0.6749
>
> Comparing this with metafor (where I switch to rma.mv() so I also get
> profile likelihood CIs; rma() computes the CIs for tau^2 and I^2 using the
> Q-profile method):
>
> res <- rma.mv(yi, vi, random = ~ 1 | study, data=Hox02, method="ML")
> ci <- confint(res)
> wi <- 1/res$vi
> s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
> res$sigma2 / (res$sigma2 + s2)
> ci$random[1,] / (ci$random[1,] + s2)
>
> This yields:
>
> estimate ci.lb ci.ub
> 0.6078023 0.2763288 0.8110635
>
> So again the same estimate, but the lower bound is slightly different and
> the upper bound is totally different.
>
> Looking at the CI for the variance component itself:
>
> Estimate Std.Error lbound ubound z value Pr(>|z|)
> Intercept1 0.579035 NA 0.365413 0.800250 NA NA
> Tau2_1_1 0.131520 NA 0.034863 0.364294 NA NA
>
> While confint(res) shows:
>
> estimate ci.lb ci.ub
> sigma^2 0.1315 0.0324 0.3643
> sigma 0.3627 0.1800 0.6036
>
> Now the lower bound is slightly different, but that might just be a
> numerical issue.
>
> Any idea why the that upper bound for I^2 is so different above?
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Mike Cheung [mailto:mikewlcheung using gmail.com]
> Sent: Wednesday, 15 May, 2019 2:48
> To: Viechtbauer, Wolfgang (SP)
> Cc: Guido Schwarzer; Rushkin, Megan C; r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Differences in I^2 estimation between meta and
> metafor packages
>
> As a follow-up of the discussion, the metaSEM package implements these
> definitions of the I^2 with the likelihood-based CIs on them.
>
> library(metaSEM)
>
> summary( meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"),
> intervals.type="LB") )
>
> Heterogeneity indices (I2) and their 95% likelihood-based CIs:
> lbound Estimate ubound
> Intercept1: I2 (Q statistic) 0.27625 0.60780 0.6749
> Intercept1: I2 (harmonic mean) 0.27713 0.60885 0.6766
> Intercept1: I2 (arithmetic mean) 0.26209 0.59052 0.6431
>
> A brief introduction is available here:
> https://books.google.com.sg/books?id=Pw7QBwAAQBAJ&pg=PA92&lpg=PA92&dq=%224.3.3+Quantifying+the+degree+of+the+heterogeneity+of+effect+sizes%22&source=bl&ots=zDjRbpiGL5&sig=ACfU3U10nYLvLi9idJ_QnXCKcasi8sDJ3g&hl=en&sa=X&ved=2ahUKEwjk3t6lppziAhXZbSsKHcEmA70Q6AEwAHoECAUQAQ#v=onepage&q=%224.3.3%20Quantifying%20the%20degree%20of%20the%20heterogeneity%20of%20effect%20sizes%22&f=false
>
> --
> ---------------------------------------------------------------------
> Mike W.L. Cheung Phone: (65) 6516-3702
> Department of Psychology Fax: (65) 6773-1843
> National University of Singapore
> http://mikewlcheung.github.io/
> ---------------------------------------------------------------------
>
> On Tue, May 14, 2019 at 6:26 PM Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> Good points. I wrote up that FAQ because this question comes up quite a
> bit, since (by default, that is, with REML estimation) metafor does deviate
> a bit from how I^2 is typically computed (i.e., with 100*(Q - (k-1))/Q), so
> users are sometimes surprised by this.
>
> And I fully agree that there isn't a *right* way of computing I^2, just
> different definitions. I actually find the way the 'typical' sampling
> variance is computed (which is implicitly used by both definitions) rather
> weird. Other definitions have been proposed (in particular, by
> Takkouche/Spiegelman/Crippa), such as:
>
> I^2 = tau^2 / (tau^2 + s^2), where s^2 = k/sum(wi), where wi=1/vi (i.e.,
> the haromic mean of the sampling variances)
>
> or
>
> I^2 = 1/k sum(tau^2 / (vi + tau^2)).
>
> Those actually seem a bit more intuitive to me. The 'hetmeta' package
> provides those versions (called R_I and R_b).
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Guido Schwarzer [mailto:sc using imbi.uni-freiburg.de]
> Sent: Tuesday, 14 May, 2019 11:59
> To: Viechtbauer, Wolfgang (SP); Rushkin, Megan C;
> r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Differences in I^2 estimation between meta and
> metafor packages
>
> Am 13.05.19 um 10:45 schrieb Viechtbauer, Wolfgang (SP):
>
> > Hi Megan,
> >
> > This is answered here:
> >
> >
> http://www.metafor-project.org/doku.php/faq#how_are_i_2_and_h_2_computed_i
>
> This website nicely describes the differences in the calculation of I2
> in metafor and meta. As this is written from the metafor perspective, I
> would like to describe why meta uses the "other" definition. ;-)
>
> Wolfgang mentions two advantages of the metafor implementation of I2:
> 1) "more general definition"
> 2) "values of I2 and H2 will be consistent with ... tau2"
>
> I agree that the metafor implementation is more general as one gets a
> different I2 value for each estimation method of tau2. However, the meta
> implementation is also based on a generalization (to the situation in
> which precisions differ between studies) - see section 3.3 in Higgins &
> Thompons (2002).
>
> On the other hand, while the metafor implementation guarantees
> consistent estimates for I2 and tau2, the meta implementation guarantees
> consistency of the I2 estimate and the test for heterogeneity (which -
> like I2 - is based on Q and the number of studies).
>
> In summary, one should know the differences in the estimation of I2
> between metafor and meta, however, there is (in my opinion) no clear
> "winner".
>
> Best wishes, Guido
>
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list