[R-meta] Different results for tests of funnel plot asymmetry using R package meta and metafor (was: Publication Bias)
Guido Schwarzer
sc at imbi.uni-freiburg.de
Wed Nov 29 10:31:51 CET 2017
Am 10.11.17 um 11:17 schrieb Viechtbauer Wolfgang (SP):
> [...]
> Note that "AS-Thompson" refers to using the 'rma' model.
Wolfgang, I fear this is not the case.
The "AS-Thompson" test refers to using (i) the arcsine difference as
effect measure which is unimportant for the following discussion and
(ii) method 3a in Thompson & Sharp (1999) which implements an additive
between-study variance component. This method is implemented in
metabias() of R package *meta* (argument /method = "mm"/).
I had a look at results of regtest() from *metafor* and metabias() from
*meta* using two (small) examples which are part of the examples on the
help page of metaprop(). The results are summarized in the attached text
file and show that p-values from regtest() with argument /model = "rma"/
(default) and metabias() with argument /method = "mm"/ do not agree. On
the other side, results from regtest() with argument /model = "lm"/ and
metabias() with argument /method = "linreg"/ (default) are identical.
Actually, in the second example, we see a similar pattern for regtest()
as observed by Laura: non-significant results for /model = "lm"/ and
(highly) significant results for /model = "rma"/. Clearly, it is not
possible to deduce any general patterns from two examples.
I only had a quick glance at the R code of regtest(), however, I assume
that argument /model = "rma"/ uses a multiplicative overdispersion
factor (see equation (2) in Thompson & Sharp, 1999).
Main reason to implement an additive variance component in metabias() is
the following statement by Thompson and Higgins (2002):
"There is little to motivate the use of a multiplicative variance
adjustment factor in meta-regression, since the within-study variances
are known, although this is what is achieved by the conventional use of
weighted regression programs in most statistical software. An additive
component for the residual variance is more reasonable in both
meta-regression [9] [...]".
See also Harbord et al. (2006) - including Matthias Egger as co-author:
"The alternative ‘weighted’ version of the test also suggested by Egger
et al. [7], denoted by ‘EW’ in Reference [14], is seldom used and lacks
a theoretical justification [24]."
Furthermore, the test by Thompson and Sharp (1999), method 3a, is the
only test considering between-study heterogeneity mentioned in Sterne et
al. (2011), albeit in the setting of a binary outcome with two groups.
Best wishes,
Guido
References:
Harbord RM, Egger M, Sterne JA, 2006, A modified test for small-study
effects in meta-analyses of controlled trials with binary endpoints,
Statistics in Medicine, 25(20), pp. 3443-57
Sterne JAC et al., 2011, Recommendations for examining and interpreting
funnel plot asymmetry in meta-analyses of randomised controlled trials,
BMJ (Clinical research ed.), 343, p. d4002
Thompson SG, Higgins JP, 2002, How should meta-regression analyses be
undertaken and interpreted? Statistics in Medicine, 21(11), pp. 1559-73
Thompson SG, Sharp SJ, 1999, Explaining heterogeneity in meta-analysis:
A comparison of methods. Statistics in Medicine, 18(20): pp. 2693-708
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20171129/9ec69fd3/attachment.html>
-------------- next part --------------
library(meta)
##
## Example from Miller (1978):
##
death <- c(3, 6, 10, 1)
animals <- c(11, 17, 21, 6)
##
m1.pft <- metaprop(death, animals, sm = "PFT")
m1.pas <- update(m1.pft, sm = "PAS")
##
m1.pft.m4.fe <- metafor::rma(yi = m1.pft$TE, sei = m1.pft$seTE, method = "FE")
m1.pas.m4.fe <- metafor::rma(yi = m1.pas$TE, sei = m1.pas$seTE, method = "FE")
m1.pft.m4.dl <- metafor::rma(yi = m1.pft$TE, sei = m1.pft$seTE, method = "DL")
m1.pas.m4.dl <- metafor::rma(yi = m1.pas$TE, sei = m1.pas$seTE, method = "DL")
##
round(m1.pft$TE.fixed - m1.pft.m4.fe$b, 12)
round(m1.pft$TE.random - m1.pft.m4.dl$b, 12)
##
round(m1.pas$TE.fixed - m1.pas.m4.fe$b, 12)
round(m1.pas$TE.random - m1.pas.m4.dl$b, 12)
d1 <- data.frame(sm = rep(c("PFT", "PAS"), 4),
func = rep(c("metabias()", "regtest()"), rep(4, 2)),
argument = c(rep("*default*", 2),
rep("method = \"mm\"", 2),
rep("*default*", 2),
rep("model = \"lm\"", 2)),
p.value = NA)
##
## Linear regression test using standard error, i.e., classic Egger test
##
d1$p.value[1] <- metabias(m1.pft, k.min = 4)$p.value
d1$p.value[2] <- metabias(m1.pas, k.min = 4)$p.value
##
## Linear regression test with additive between-study variance
##
d1$p.value[3] <- metabias(m1.pft, method = "mm", k.min = 4)$p.value
d1$p.value[4] <- metabias(m1.pas, method = "mm", k.min = 4)$p.value
##
## Linear regression test using R package metafor
##
d1$p.value[5] <- metafor::regtest(m1.pft.m4.dl)$pval
d1$p.value[6] <- metafor::regtest(m1.pas.m4.dl)$pval
##
d1$p.value[7] <- metafor::regtest(m1.pft.m4.dl, model = "lm")$pval
d1$p.value[8] <- metafor::regtest(m1.pas.m4.dl, model = "lm")$pval
d1$p.value <- meta:::format.p(d1$p.value)
##
sink("metaprop03.txt")
cat("*** Example from Miller (1978) ***\n\n")
##
d1[order(d1$sm), ]
sink()
##
## Example from Newcombe (1998)
##
event <- c(81, 15, 0, 1)
n <- c(263, 148, 20, 29)
##
m2.pft <- metaprop(event, n, sm = "PFT")
m2.pas <- update(m2.pft, sm = "PAS")
##
m2.pft.m4.fe <- metafor::rma(yi = m2.pft$TE, sei = m2.pft$seTE, method = "FE")
m2.pas.m4.fe <- metafor::rma(yi = m2.pas$TE, sei = m2.pas$seTE, method = "FE")
m2.pft.m4.dl <- metafor::rma(yi = m2.pft$TE, sei = m2.pft$seTE, method = "DL")
m2.pas.m4.dl <- metafor::rma(yi = m2.pas$TE, sei = m2.pas$seTE, method = "DL")
##
round(m2.pft$TE.fixed - m2.pft.m4.fe$b, 12)
round(m2.pft$TE.random - m2.pft.m4.dl$b, 12)
##
round(m2.pas$TE.fixed - m2.pas.m4.fe$b, 12)
round(m2.pas$TE.random - m2.pas.m4.dl$b, 12)
d2 <- data.frame(sm = rep(c("PFT", "PAS"), 4),
func = rep(c("metabias()", "regtest()"), rep(4, 2)),
argument = c(rep("*default*", 2),
rep("method = \"mm\"", 2),
rep("*default*", 2),
rep("model = \"lm\"", 2)),
p.value = NA)
##
## Linear regression test using standard error, i.e., classic Egger test
##
d2$p.value[1] <- metabias(m2.pft, k.min = 4)$p.value
d2$p.value[2] <- metabias(m2.pas, k.min = 4)$p.value
##
## Linear regression test with additive between-study variance
##
d2$p.value[3] <- metabias(m2.pft, method = "mm", k.min = 4)$p.value
d2$p.value[4] <- metabias(m2.pas, method = "mm", k.min = 4)$p.value
##
## Linear regression test using R package metafor
##
d2$p.value[5] <- metafor::regtest(m2.pft.m4.dl)$pval
d2$p.value[6] <- metafor::regtest(m2.pas.m4.dl)$pval
##
d2$p.value[7] <- metafor::regtest(m2.pft.m4.dl, model = "lm")$pval
d2$p.value[8] <- metafor::regtest(m2.pas.m4.dl, model = "lm")$pval
d2$p.value <- meta:::format.p(d2$p.value)
##
sink("metaprop03.txt", append = TRUE)
cat("\n\n*** Example from Newcombe (1998) ***\n\n")
d2[order(d2$sm), ]
sink()
-------------- next part --------------
*** Example from Miller (1978) ***
sm func argument p.value
2 PAS metabias() *default* 0.0585
4 PAS metabias() method = "mm" 0.2584
6 PAS regtest() *default* 0.1180
8 PAS regtest() model = "lm" 0.0585
1 PFT metabias() *default* 0.0761
3 PFT metabias() method = "mm" 0.3061
5 PFT regtest() *default* 0.1730
7 PFT regtest() model = "lm" 0.0761
*** Example from Newcombe (1998) ***
sm func argument p.value
2 PAS metabias() *default* 0.1538
4 PAS metabias() method = "mm" 0.1177
6 PAS regtest() *default* 0.0080
8 PAS regtest() model = "lm" 0.1538
1 PFT metabias() *default* 0.1996
3 PFT metabias() method = "mm" 0.1686
5 PFT regtest() *default* 0.0343
7 PFT regtest() model = "lm" 0.1996
More information about the R-sig-meta-analysis
mailing list