[R-meta] RVE or not RVE in meta-regressions with small number of studies?

[Röhl, Sebastian]

Dear all,

I came across an article in RER that argues that one could or should forgo RVE for analysis of categorical moderators in case of smaller study numbers:

Cao, Y., Grace Kim, Y.‑S., & Cho, M. (2022). Are Observed Classroom Practices Related to Student Language/Literacy Achievement? Review of Educational Research, 003465432211306. https://doi.org/10.3102/00346543221130687
Page 10: “We acknowledge the superiority of robust variance estimation (RVE) for handling dependent effect sizes. However, it has a few important limitations. First, it neither
models heterogeneity at multiple levels nor provides corresponding hypothesis tests. Second, the power of the categorical moderator highly depends on the number of studies and features of the covariate (Tanner-Smith, Tipton, & Polanin, 2016). When the number of studies is small, the test statistics and confidence intervals based on RVE can have inflated Type I error (Hedges et al., 2010; Tipton & Pustejovsky, 2015). Relating to our cases, many of our moderators had imbalanced distributions […]. Consequently, tests of particular moderators may be severely underpowered.”
Of course, the first argument can be invalidated by the use of correlated hierarchical effects models with RVE. However, I find the second argument very relevant from my experience.

How is this viewed here on the mailing list?

In the social sciences, after all, we more often conduct meta-analyses with relatively small study corpus (n<100 or n<50). In high-ranked journals in this research field (e.g., Psychological Bulletin, Review of Educational Research, Educational Research Review…) I very rarely find the use of RVE / CRVE.

In mentioned types of moderator analyses with small number of studies in one category, I also often face the same problem that effects become non-significant when using CRVE as soon as moderator levels are populated with less than 10-15 studies. Joshi et al (2022) also talk about RVE being (too) highly conversative in these cases. I have also used cluster wild bootstrapping for significance testing of individual effects in this case. However, the problem of missing SEs and C.I.s as well as the high computation time arises here.

Right now, I am again facing the problem of model selection for a meta-analysis with about 50 studies and 500 ES (correlations). Since we are dealing with ES within studies, I would choose a correlated hierarchical effects model with CRVE, which also works very well for the main effects, but again leads to said very large SEs for the moderators. As a pure CHE model (which in my opinion still fits better than the pure HE model in the above mentioned article by Cao et al) the SEs are of course somewhat more moderate.
Do you have any tips or hints for an alternative?

Thank you for your help and comments!
Kind regards,
Sebastian

****************************
Dr. Sebastian Röhl
Eberhard Karls Universität Tübingen
Institut für Erziehungswissenschaft
Tübingen School of Education (TüSE)
Wilhelmstraße 31 / Raum 302
72074 Tübingen

Telefon: +49 7071 29-75527
Fax: +49 7071 29-35309
E-Mail: sebastian.roehl using uni-tuebingen.de<mailto:sebastian.roehl using uni-tuebingen.de>
Twitter: @sebastian_roehl  @ResTeacherEdu


	[[alternative HTML version deleted]]