[R-meta] Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs

[Röhl, Sebastian]

Hi,

I have the following problem:
I am conducting a multilevel meta-analysis using metafor with cluster robust variance estimation and want to test the moderating effect of different kinds of outcomes. Additionally I want to test whether the several outcomes differ significantly from each other.
Here is an example:
out_3 <- rma.mv(zr, V=var, random = ~ 1| Sample_ID / number, mods = ~ -1 + out_intg + out_acad + out_socem,
                       data = data_int)
out_3_rob <- robust(out_3, Sample_ID, clubSandwich = T)
anova(out_3_rob, X=rbind(c(-1,1,0),c(-1,0,1), c(0,-1,1)))

The robust model result shows C.I. that overlap.
Model Results:

           estimate     se¹   tval¹    df¹   pval¹  ci.lb¹  ci.ub¹     ​
out_intg     0.2302  0.0231  9.9484  30.84  <.0001  0.1830  0.2773  ***
out_acad     0.1646  0.0220  7.4677  17.36  <.0001  0.1182  0.2111  ***
out_socem    0.2458  0.0278  8.8510  22.27  <.0001  0.1882  0.3034  ***

BUT the anova results show significant differences between 2 outcomes:

Hypotheses:

1:  -out_intg + out_acad = 0

2: -out_intg + out_socem = 0

3: -out_acad + out_socem = 0



Results:

   estimate     se    tval    df   pval

1:  -0.0655 0.0322 -2.0349 10.92 0.0669

2:   0.0157 0.0330  0.4742 13.21 0.6431

3:   0.0812 0.0188  4.3264  9.60 0.0016

Do I have a thinking error here about the ANOVA or is this pairwise testing not possible with the CRVE-results?
Perhaps I am also interpreting the C.I.s incorrectly? If I calculate a pairwise comparison with the non-robust model, I also get significant difference although also the non-robust C.I. overlap.

Thank you very much for your help!
Best,
Sebastian

****************************
Dr. Sebastian Röhl
Eberhard Karls Universität Tübingen
Institute for Educational Science
Tübingen School of Education (TüSE)
Wilhelmstraße 31 / Room 302
D-72074 Tübingen
Germany

Phone: +49 7071 29-75527
Fax: +49 7071 29-35309
Email: sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de>
Twitter: @sebastian_roehl  @ResTeacherEdu



	[[alternative HTML version deleted]]