[R-meta] Different results for tests of funnel plot asymmetry using R package meta and metafor (was: Publication Bias)

[Viechtbauer Wolfgang (SP)]

Hi Guido,

metareg() (with the default model="rma") *is* the same as "AS-Thompson" (additive model). As you say, the choice of outcome measure is a secondary issue here. The difference you are observing is due to the fact that metabias(..., method="mm") computes the p-value based on a t-distribution with k-2 dfs, while metareg() computes the p-values by default using a standard normal distribution (but this can be changed, depending on what was used in rma()). Let's abstract this to a simpler example to see what is going on:

#######################################

library(metafor)
library(meta)

set.seed(1234)

k <- 5

sei <- runif(k, .01, 1)
yi <- rnorm(k, 0, sqrt(sei^2 + .5))

### fit RE model with DL estimator (exact same results)

res1 <- rma(yi, sei=sei, method="DL")
res1
res2 <- metagen(yi, sei)
summary(res2)

### regression (Egger) test using multiplicative model (exact same results)

regtest(res1, model="lm", ret.fit=TRUE)
metabias(res2, k.min=5)

### regression test using additive model (same results except for p-value)
### metabias() uses t(df=k-2) while regtest() uses standard normal distibution

regtest(res1, model="rma", ret.fit=TRUE)
metabias(res2, method="mm", k.min=5)

### force use of t(df=k-2) with regtest() (exact same results as metabias() now)

res1 <- rma(yi, sei=sei, method="DL", test="t")
regtest(res1, model="rma")

#######################################

The test statistic (under the null) for the additive model does not follow a t-distribution with k-2 degrees of freedom. It also does not follow a standard normal distribution. Probably the better thing to do would be to use the Knapp and Hartung method. That would be:

res1 <- rma(yi, sei=sei, method="DL", test="knha")
regtest(res1, model="rma")

Then the distribution of the test statistic is very well approximated by a t-distribution with k-2 dfs.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Guido Schwarzer
Sent: Wednesday, 29 November, 2017 10:32
To: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Different results for tests of funnel plot asymmetry using R package meta and metafor (was: Publication Bias)

ATTACHMENT(S) REMOVED: metaprop03.txt | metaprop03.R 

Am 10.11.17 um 11:17 schrieb Viechtbauer Wolfgang (SP):
[...]
Note that "AS-Thompson" refers to using the 'rma' model.

Wolfgang, I fear this is not the case.

The "AS-Thompson" test refers to using (i) the arcsine difference as effect measure which is unimportant for the following discussion and (ii) method 3a in Thompson & Sharp (1999) which implements an additive between-study variance component. This method is implemented in metabias() of R package meta (argument method = "mm").

I had a look at results of regtest() from metafor and metabias() from meta using two (small) examples which are part of the examples on the help page of metaprop(). The results are summarized in the attached text file and show that p-values from regtest() with argument model = "rma" (default) and metabias() with argument method = "mm" do not agree. On the other side, results from regtest() with argument model = "lm" and metabias() with argument method = "linreg" (default) are identical. Actually, in the second example, we see a similar pattern for regtest() as observed by Laura: non-significant results for model = "lm" and (highly) significant results for model = "rma". Clearly, it is not possible to deduce any general patterns from two examples.

I only had a quick glance at the R code of regtest(), however, I assume that argument model = "rma" uses a multiplicative overdispersion factor (see equation (2) in Thompson & Sharp, 1999).

Main reason to implement an additive variance component in metabias() is the following statement by Thompson and Higgins (2002):

"There is little to motivate the use of a multiplicative variance adjustment factor in meta-regression, since the within-study variances are known, although this is what is achieved by the conventional use of weighted regression programs in most statistical software. An additive component for the residual variance is more reasonable in both meta-regression [9] [...]".

See also Harbord et al. (2006) - including Matthias Egger as co-author:

"The alternative ‘weighted’ version of the test also suggested by Egger et al. [7], denoted by ‘EW’ in Reference [14], is seldom used and lacks a theoretical justification [24]."

Furthermore, the test by Thompson and Sharp (1999), method 3a, is the only test considering between-study heterogeneity mentioned in Sterne et al. (2011), albeit in the setting of a binary outcome with two groups.

Best wishes,
Guido

References:

Harbord RM, Egger M, Sterne JA, 2006, A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints, Statistics in Medicine, 25(20), pp. 3443-57

Sterne JAC et al., 2011, Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials, BMJ (Clinical research ed.), 343, p. d4002

Thompson SG, Higgins JP, 2002, How should meta-regression analyses be undertaken and interpreted? Statistics in Medicine, 21(11), pp. 1559-73

Thompson SG, Sharp SJ, 1999, Explaining heterogeneity in meta-analysis: A comparison of methods. Statistics in Medicine, 18(20): pp. 2693-708