[R-meta] Assessing selection bias / multivariate meta-analysis

[Viechtbauer, Wolfgang (NP)]

Dear Pia,

Generally, I don't think there really is any method that is going to be a great choice here. The 'Egger sandwich' (i.e., an Egger type regression model using cluster-robust inference methods) is a decent option, since it logically generalizes the standard Egger regression method to this context, but it is unclear what kind of bias/selection effect this may pick up (missing studies, missing estimates within studies, a combination thereof).

Yes, for the 3PSM, you would have to either ignore the dependencies or select one estimate per study (and maybe repeat the latter a large number of times for different subsets).

I assume you are familiar with these papers. If not, they are directly relevant:

Rodgers, M. A., & Pustejovsky, J. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26(2), 141-160. https://doi.org/10.1037/met0000300

Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P., & Van den Noortgate, W. (2021). Detecting selection bias in meta-analyses with multiple outcomes: A simulation study. The Journal of Experimental Education, 89(1), 125-144. https://doi.org/10.1080/00220973.2019.1582470

Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez-Tójar, A., Yang, Y., & O'Dea, R. E. (2022). Methods for testing publication bias in ecological and evolutionary meta-analyses. Methods in Ecology and Evolution, 13(1), 4-21. https://doi.org/10.1111/2041-210X.13724

I think James is working on some methods related to this topic:

https://jepusto.com/posts/cluster-bootstrap-selection-model/

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Pia-Magdalena Schmidt via R-sig-meta-analysis
> Sent: Wednesday, November 20, 2024 21:58
> To: r-sig-meta-analysis using r-project.org
> Cc: Pia-Magdalena Schmidt <pia-magdalena.schmidt using uni-bonn.de>
> Subject: [R-meta] Assessing selection bias / multivariate meta-analysis
>
> Dear all,
> Although this topic has been discussed several times and I read the archives
> and referenced papers, I’m still not sure how to assess and possibly correct
> for selection bias in multivariate meta-analyses.
>
> I used the metafor package and ran meta-analyses with SMCC as effect size
> (all studies used within-designs) and fitted rma.mv models as several
> studies report more than one effect size. Furthermore, I used cluster-robust
> methods to examine the robustness of the models.
> For a subset of my data, I used meta-regressions with one continuous
> moderator.
> All effect sizes are from published journal articles. The range of included
> studies is between 30 and 6 with a number of effect sizes between 45 and 10.
>
> Since I want to take the dependencies into account, I would not use funnel
> plots or trim and fill. I wonder if using Egger's regression test adjusted
> for rma.mv as well as PET-PEESE and perhaps the sensitivity analysis
> suggested by Mathur & Vanderweele (2020) as well as 3PSM would be a
> reasonable way to go? Although the latter would only use one effect size per
> study or an aggregated effect size, right?
>
> I would be very grateful for any recommendations!
> Best,
> Pia
>
> Below is an excerpt from my code:
> ES_all <- escalc(measure="SMCC", m1i= m1i, sd1i= sd1i, ni = ni, m2i= m2i,
> sd2i= sd2i, pi= pi, ri = ri, data= dat)
> V <- vcalc(vi=ES_all$vi, cluster=id_database, obs = effect_id, rho =0.605,
> data=dat)
> res <- rma.mv(yi=ES_all$yi, V, random = ~ 1 | id_database/effect_id, data =
> dat)
> res.robust <- robust(res, cluster = id_database, clubSandwich = TRUE)
>
> # subset
> res_LOR <- rma.mv(yi=ES_LOR$yi, V, random = ~ 1 | id_database/effect_id,
> mods = ~ dose, data = dat)