[R-meta] Differences in I^2 estimation between meta and metafor packages

[Viechtbauer, Wolfgang (SP)]

Interesting, thanks for mentioning this, Mike. I compared this with metafor:

library(metaSEM)
library(metafor)

summary(meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"), intervals.type="LB")) 

res <- rma(yi, vi, data=Hox02, method="ML")
(res$QE - (res$k - 1)) / res$QE

This would be the 'usual' definition, which yields 0.6168211. But that's not what is given by "I2 (Q statistic)". Instead, that is the same as what rma() uses, namely:

res$I2 / 100

which we can also compute manually with:

wi <- 1/res$vi
s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
res$tau2 / (res$tau2 + s2)

The harmonic and arithmetic mean definitions are:

res$tau2 / (res$tau2 + res$k/sum(wi))
res$tau2 / (res$tau2 + mean(res$vi))

Note that R_b is not the same as the arithmetic mean definition:

1/res$k * sum(res$tau2 / (res$vi + res$tau2))

So, yay, we are up to 4 definitions now :)

One other thing I noticed: The 95% likelihood-based CIs (I assume these are profile likelihood CIs) are different from what I obtain with metafor. For example:

Heterogeneity indices (I2) and their 95% likelihood-based CIs:
                                  lbound Estimate ubound
Intercept1: I2 (Q statistic)     0.27625  0.60780 0.6749

Comparing this with metafor (where I switch to rma.mv() so I also get profile likelihood CIs; rma() computes the CIs for tau^2 and I^2 using the Q-profile method):

res <- rma.mv(yi, vi, random = ~ 1 | study, data=Hox02, method="ML")
ci <- confint(res)
wi <- 1/res$vi
s2 <- (res$k-1) * sum(wi) / (sum(wi)^2 - sum(wi^2))
res$sigma2 / (res$sigma2 + s2)
ci$random[1,] / (ci$random[1,] + s2)

This yields:

estimate     ci.lb     ci.ub 
0.6078023 0.2763288 0.8110635

So again the same estimate, but the lower bound is slightly different and the upper bound is totally different.

Looking at the CI for the variance component itself:

           Estimate Std.Error   lbound   ubound z value Pr(>|z|)
Intercept1 0.579035        NA 0.365413 0.800250      NA       NA
Tau2_1_1   0.131520        NA 0.034863 0.364294      NA       NA

While confint(res) shows:

        estimate  ci.lb  ci.ub 
sigma^2   0.1315 0.0324 0.3643 
sigma     0.3627 0.1800 0.6036

Now the lower bound is slightly different, but that might just be a numerical issue.

Any idea why the that upper bound for I^2 is so different above?

Best,
Wolfgang

-----Original Message-----
From: Mike Cheung [mailto:mikewlcheung using gmail.com] 
Sent: Wednesday, 15 May, 2019 2:48
To: Viechtbauer, Wolfgang (SP)
Cc: Guido Schwarzer; Rushkin, Megan C; r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Differences in I^2 estimation between meta and metafor packages

As a follow-up of the discussion, the metaSEM package implements these definitions of the I^2 with the likelihood-based CIs on them.

library(metaSEM)

summary( meta(y=yi, v=vi, data=Hox02, I2=c("I2q", "I2hm", "I2am"), intervals.type="LB") ) 

Heterogeneity indices (I2) and their 95% likelihood-based CIs:
                                  lbound Estimate ubound
Intercept1: I2 (Q statistic)     0.27625  0.60780 0.6749
Intercept1: I2 (harmonic mean)   0.27713  0.60885 0.6766
Intercept1: I2 (arithmetic mean) 0.26209  0.59052 0.6431

A brief introduction is available here: https://books.google.com.sg/books?id=Pw7QBwAAQBAJ&pg=PA92&lpg=PA92&dq=%224.3.3+Quantifying+the+degree+of+the+heterogeneity+of+effect+sizes%22&source=bl&ots=zDjRbpiGL5&sig=ACfU3U10nYLvLi9idJ_QnXCKcasi8sDJ3g&hl=en&sa=X&ved=2ahUKEwjk3t6lppziAhXZbSsKHcEmA70Q6AEwAHoECAUQAQ#v=onepage&q=%224.3.3%20Quantifying%20the%20degree%20of%20the%20heterogeneity%20of%20effect%20sizes%22&f=false

-- 
---------------------------------------------------------------------
 Mike W.L. Cheung               Phone: (65) 6516-3702
 Department of Psychology       Fax:   (65) 6773-1843
 National University of Singapore
 http://mikewlcheung.github.io/
---------------------------------------------------------------------

On Tue, May 14, 2019 at 6:26 PM Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Good points. I wrote up that FAQ because this question comes up quite a bit, since (by default, that is, with REML estimation) metafor does deviate a bit from how I^2 is typically computed (i.e., with 100*(Q - (k-1))/Q), so users are sometimes surprised by this.

And I fully agree that there isn't a *right* way of computing I^2, just different definitions. I actually find the way the 'typical' sampling variance is computed (which is implicitly used by both definitions) rather weird. Other definitions have been proposed (in particular, by Takkouche/Spiegelman/Crippa), such as:

I^2 = tau^2 / (tau^2 + s^2), where s^2 = k/sum(wi), where wi=1/vi (i.e., the haromic mean of the sampling variances)

or

I^2 = 1/k sum(tau^2 / (vi + tau^2)).

Those actually seem a bit more intuitive to me. The 'hetmeta' package provides those versions (called R_I and R_b).

Best,
Wolfgang

-----Original Message-----
From: Guido Schwarzer [mailto:sc using imbi.uni-freiburg.de] 
Sent: Tuesday, 14 May, 2019 11:59
To: Viechtbauer, Wolfgang (SP); Rushkin, Megan C; r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Differences in I^2 estimation between meta and metafor packages

Am 13.05.19 um 10:45 schrieb Viechtbauer, Wolfgang (SP):

> Hi Megan,
>
> This is answered here:
>
> http://www.metafor-project.org/doku.php/faq#how_are_i_2_and_h_2_computed_i

This website nicely describes the differences in the calculation of I2 
in metafor and meta. As this is written from the metafor perspective, I 
would like to describe why meta uses the "other" definition. ;-)

Wolfgang mentions two advantages of the metafor implementation of I2:
1) "more general definition"
2) "values of I2 and H2 will be consistent with ... tau2"

I agree that the metafor implementation is more general as one gets a 
different I2 value for each estimation method of tau2. However, the meta 
implementation is also based on a generalization (to the situation in 
which precisions differ between studies) - see section 3.3 in Higgins & 
Thompons (2002).

On the other hand, while the metafor implementation guarantees 
consistent estimates for I2 and tau2, the meta implementation guarantees 
consistency of the I2 estimate and the test for heterogeneity (which - 
like I2 - is based on Q and the number of studies).

In summary, one should know the differences in the estimation of I2 
between metafor and meta, however, there is (in my opinion) no clear 
"winner".

Best wishes, Guido