[R-meta] Seeking advice on multimoderator meta-regression in multilevel meta-analysis

[Maximilian Steininger]

Dear all,

We conducted a multilevel meta-analysis with random effects specified for individual effect sizes (k = 90) nested within studies (n = 60). We preregistered a series of unimoderator analyses of 4 categorical predictors. Additionally, we conducted exploratory unimoderator analyses with 4 more categorical predictors and 2 continuous predictors – resulting in a total of 10 separate models.

In our manuscript, we reported these unimoderator analyses, identified two significant moderators, and subsequently conducted an exploratory moderator analysis using these two significant moderators as predictors.
A reviewer suggested we instead include all moderators in a single multimoderator meta-regression model – i.e., using all 10 predictors (8 categorical, 2 continuous).

I am open to this suggestion, but have some concerns, and I would be grateful for your insights.

Model overview:

- 5 categorical predictors with 2 levels
- 2 categorical predictors with 3 levels
- 1 categorical predictor with 4 levels
- 2 continuous (centred) predictors

Here is an example of the model specification in R:

metaregression = rma.mv(yi ~ cat1 + cat2 + cat3 +cat4 +
                                                 cat5 + cat6 + cat7 + cat8 +
                                                 con1 + con2,
					         Vmetaregression,
					 	 random = ~ 1 | study_id/es_id,
					         data = all_fx)

My concerns are the following:

1) The model requires an estimation of 15 regression parameters. With only 60 studies and 90 effects, this falls below the often mentioned minimum of 10 studies per predictor. I worry this may lead to overfitting and unstable estimates. Would this compromise the stability of the regression coefficients due to increased sampling error?

2) With 8 categorical moderators, interpretation becomes challenging. If I understand correctly, the model yields conditional effects, i.e., each moderator’s estimate is reported holding all other moderators at their reference level. Is this correct? If so, it seems the coefficients might be difficult to interpret, since they are related to a small hypothetical subset of studies.

3) Related to 2, we will only have very sparse data across these category combinations, with some of these combinations being non-existent or underrepresented. To what extent can the model handle such sparsity and still provide meaningful estimates?

4) Do we face power issues given the “moderate” number of effects relative to the number of moderators?

5) Could the limited sample size, coupled with the large amount of moderators, increase sensitivity to outlying studies or effect sizes, potentially distorting the results?

I’m seriously considering the reviewer’s suggestion but want to ensure that any expanded model is both statistically sound and interpretable.

Thanks in advance for your time and input - I appreciate any guidance or pointers to references that can help me tackle this issue.

Best and thanks,
Max

——

Mag. Maximilian Steininger
  PhD candidate

  Social, Cognitive, and Affective Neuroscience Unit
  Faculty of Psychology
  University of Vienna

  Liebiggasse 5
  1010 Vienna, Austria

  e: maximilian.steininger using univie.ac.at
  w: http://scan.psy.univie.ac.at