[R-meta] Meta-analytical test of mediation model including dependent tests - looking to resolve metafor issue or find alternative approach

[Lukas Wallrich]

Dear list members,

I am trying to run a meta-analysis that includes dependent effect sizes
(correlations), nested into studies and then into measures, for which I
ultimately want to estimate a mediation model. For that, I am trying to
follow Wilson et al. (2016,
https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1199) who propose to
collate a long dataset of each correlation value with a factor that
indicates which correlation it is testing, and then using that as a
moderator.

However, the results differ from separate estimates of the correlation
coefficients and, more importantly, the confidence intervals shrink quite a
lot. I am not sure why this would happen and whether I can trust the
results.

Below the code and results - I would very much appreciate any pointers
regarding what is going wrong, or suggestions for an alternative approach.
Please note that I am cross-posting this from Stackexchange where I have
not received a response within a week:
https://stats.stackexchange.com/q/499690/240420

Many thanks,

Lukas

library(metafor)
#> Loading required package: Matrix
#> Loading 'metafor' package (version 2.4-0). For an overview
#> and introduction to the package please type: help(metafor).

dat <- tibble::tribble(
  ~study, ~measure, ~pair, ~r, ~inv_N,
  "UK_mediation", "affective", "pos_att", -0.38, 0.00446428571428571,
  "UK_mediation", "cognitive", "pos_att", -0.2, 0.00446428571428571,
  "UK_mediation", "affective", "neg_att", 0.18, 0.00446428571428571,
  "UK_mediation", "cognitive", "neg_att", 0.21, 0.00446428571428571,
  "DE_mediation", "only", "pos_att", -0.43, 0.000381970970206264,
  "DE_mediation", "only", "neg_att", 0.26, 0.000381970970206264,
  "longit", "T2_prej", "pos_att", -0.221742469419445, 0.004739336492891,
  "longit", "T2_prej", "neg_att", 0.136975390214378, 0.004739336492891,
  "longit", "T1_therm", "neg_att", 0.148356343157473, 0.004739336492891,
  "longit", "T1_therm", "pos_att", -0.325301851215349, 0.004739336492891
)

#Approach per Wilson et al. - correlation pair as moderator
summary(rma.mv(r,
               inv_N,
               random = ~ 1 | measure/study,
               data = dat,
               method = "REML",
               mods = ~factor(pair)-1))
#>
#> Multivariate Meta-Analysis Model (k = 10; method: REML)
#>
#>   logLik  Deviance       AIC       BIC      AICc
#>   2.9182   -5.8363    2.1637    2.4814   15.4970
#>
#> Variance Components:
#>
#>             estim    sqrt  nlvls  fixed         factor
#> sigma^2.1  0.0000  0.0055      5     no        measure
#> sigma^2.2  0.0000  0.0055      5     no  measure/study
#>
#> Test for Residual Heterogeneity:
#> QE(df = 8) = 25.1654, p-val = 0.0015
#>
#> Test of Moderators (coefficients 1:2):
#> QM(df = 2) = 726.4284, p-val < .0001
#>
#> Model Results:
#>
#>                      estimate      se      zval    pval    ci.lb
 ci.ub
#> factor(pair)neg_att    0.2389  0.0179   13.3561  <.0001   0.2038
0.2739  ***
#> factor(pair)pos_att   -0.3917  0.0179  -21.8970  <.0001  -0.4267
 -0.3566  ***
#>
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

#Basic approach for single correlation
summary(rma.mv(r,
       inv_N,
       random = ~ 1 | measure/study,
       data = dat[dat$pair == "pos_att",],
       method = "REML"))
#>
#> Multivariate Meta-Analysis Model (k = 5; method: REML)
#>
#>   logLik  Deviance       AIC       BIC      AICc
#>   3.4054   -6.8107   -0.8107   -2.6519   23.1893
#>
#> Variance Components:
#>
#>             estim    sqrt  nlvls  fixed         factor
#> sigma^2.1  0.0040  0.0631      5     no        measure
#> sigma^2.2  0.0040  0.0631      5     no  measure/study
#>
#> Test for Heterogeneity:
#> Q(df = 4) = 19.1211, p-val = 0.0007
#>
#> Model Results:
#>
#> estimate      se     zval    pval    ci.lb    ci.ub
#>  -0.3224  0.0478  -6.7481  <.0001  -0.4160  -0.2287  ***
#>
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

*Lukas Wallrich* // *Goldsmiths*, University of London
PhD candidate in Social Psychology
Department of Psychology // 216 Whitehead Building
New Cross // London SE14 6NW
l.wallrich using gold.ac.uk // +44 7591 975294

	[[alternative HTML version deleted]]