[R-meta] Additional Info: Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs

[Röhl, Sebastian]

Hi all,

Just an addition to my question from yesterday: 
Additionally to using the robust() and anova() function, I also tried out Wald_test() from the clubSandwich packacke.
The results are the same (with F instead of T statistics):
> Wald_test(out_3, constraints = constrain_pairwise(1:3), vcov="CR2")
$`out_acad - out_intg`
 test Fstat df_num df_denom  p_val sig
  HTZ  4.14      1     10.9 0.0669   .

$`out_socem - out_intg`
 test Fstat df_num df_denom p_val sig
  HTZ 0.225      1     13.2 0.643    

$`out_socem - out_acad`
 test Fstat df_num df_denom   p_val sig
  HTZ  18.7      1      9.6 0.00165  **

Can anybody help me?

Thank you.

All the best,
Sebastian

-----Ursprüngliche Nachricht-----
Von: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> Im Auftrag von Röhl, Sebastian via R-sig-meta-analysis
Gesendet: Donnerstag, 9. Februar 2023 12:30
An: r-sig-meta-analysis using r-project.org
Cc: Röhl, Sebastian <sebastian.roehl using uni-tuebingen.de>
Betreff: [R-meta] Pairwise moderator testing in multilevel meta-analysis with CRVE / CIs

Hi,

I have the following problem:
I am conducting a multilevel meta-analysis using metafor with cluster robust variance estimation and want to test the moderating effect of different kinds of outcomes. Additionally I want to test whether the several outcomes differ significantly from each other.
Here is an example:
out_3 <- rma.mv(zr, V=var, random = ~ 1| Sample_ID / number, mods = ~ -1 + out_intg + out_acad + out_socem,
                       data = data_int)
out_3_rob <- robust(out_3, Sample_ID, clubSandwich = T) anova(out_3_rob, X=rbind(c(-1,1,0),c(-1,0,1), c(0,-1,1)))

The robust model result shows C.I. that overlap.
Model Results:

           estimate     se¹   tval¹    df¹   pval¹  ci.lb¹  ci.ub¹     ​
out_intg     0.2302  0.0231  9.9484  30.84  <.0001  0.1830  0.2773  ***
out_acad     0.1646  0.0220  7.4677  17.36  <.0001  0.1182  0.2111  ***
out_socem    0.2458  0.0278  8.8510  22.27  <.0001  0.1882  0.3034  ***

BUT the anova results show significant differences between 2 outcomes:

Hypotheses:

1:  -out_intg + out_acad = 0

2: -out_intg + out_socem = 0

3: -out_acad + out_socem = 0



Results:

   estimate     se    tval    df   pval

1:  -0.0655 0.0322 -2.0349 10.92 0.0669

2:   0.0157 0.0330  0.4742 13.21 0.6431

3:   0.0812 0.0188  4.3264  9.60 0.0016

Do I have a thinking error here about the ANOVA or is this pairwise testing not possible with the CRVE-results?
Perhaps I am also interpreting the C.I.s incorrectly? If I calculate a pairwise comparison with the non-robust model, I also get significant difference although also the non-robust C.I. overlap.

Thank you very much for your help!
Best,
Sebastian

****************************
Dr. Sebastian Röhl
Eberhard Karls Universität Tübingen
Institute for Educational Science
Tübingen School of Education (TüSE)
Wilhelmstraße 31 / Room 302
D-72074 Tübingen
Germany

Phone: +49 7071 29-75527
Fax: +49 7071 29-35309
Email: sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de>
Twitter: @sebastian_roehl  @ResTeacherEdu



	[[alternative HTML version deleted]]

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis