[R-meta] Seeking advice on multimoderator meta-regression in multilevel meta-analysis

[Michael Dewey]

Dear Maximilian

Comments in-line

On 22/05/2025 09:52, Maximilian Steininger via R-sig-meta-analysis wrote:
> Dear all,
> 
> We conducted a multilevel meta-analysis with random effects specified for individual effect sizes (k = 90) nested within studies (n = 60). We preregistered a series of unimoderator analyses of 4 categorical predictors. Additionally, we conducted exploratory unimoderator analyses with 4 more categorical predictors and 2 continuous predictors – resulting in a total of 10 separate models.
> 
> In our manuscript, we reported these unimoderator analyses, identified two significant moderators, and subsequently conducted an exploratory moderator analysis using these two significant moderators as predictors.
> A reviewer suggested we instead include all moderators in a single multimoderator meta-regression model – i.e., using all 10 predictors (8 categorical, 2 continuous).
> 
> I am open to this suggestion, but have some concerns, and I would be grateful for your insights.
> 
> Model overview:
> 
> - 5 categorical predictors with 2 levels
> - 2 categorical predictors with 3 levels
> - 1 categorical predictor with 4 levels
> - 2 continuous (centred) predictors
> 
> Here is an example of the model specification in R:
> 
> metaregression = rma.mv(yi ~ cat1 + cat2 + cat3 +cat4 +
>                                                   cat5 + cat6 + cat7 + cat8 +
>                                                   con1 + con2,
> 					         Vmetaregression,
> 					 	 random = ~ 1 | study_id/es_id,
> 					         data = all_fx)
> 
> My concerns are the following:
> 
> 1) The model requires an estimation of 15 regression parameters. With only 60 studies and 90 effects, this falls below the often mentioned minimum of 10 studies per predictor. I worry this may lead to overfitting and unstable estimates. Would this compromise the stability of the regression coefficients due to increased sampling error?

I suspect you will see large standard errors for the coefficients in 
your multimoderator analysis.>
> 2) With 8 categorical moderators, interpretation becomes challenging. If I understand correctly, the model yields conditional effects, i.e., each moderator’s estimate is reported holding all other moderators at their reference level. Is this correct? If so, it seems the coefficients might be difficult to interpret, since they are related to a small hypothetical subset of studies.
> 

I think that may be a scientific question - are we interested in such 
effects? Interpretation is also difficult if any of the moderators is 
strongly associated with others.

> 3) Related to 2, we will only have very sparse data across these category combinations, with some of these combinations being non-existent or underrepresented. To what extent can the model handle such sparsity and still provide meaningful estimates?
> 

I think that is covered by point 2.

> 4) Do we face power issues given the “moderate” number of effects relative to the number of moderators?
>

I am not sure powere is quite the right word here but your estimates 
will lack precision.


> 5) Could the limited sample size, coupled with the large amount of moderators, increase sensitivity to outlying studies or effect sizes, potentially distorting the results?
> 

Probably

> I’m seriously considering the reviewer’s suggestion but want to ensure that any expanded model is both statistically sound and interpretable.
> 
> Thanks in advance for your time and input - I appreciate any guidance or pointers to references that can help me tackle this issue.
>

I think it is worth arguing with the referee unless they have suggested 
a clear scientific question which correspond to the model.

Michael


> Best and thanks,
> Max
> 
> ——
> 
> Mag. Maximilian Steininger
>    PhD candidate
> 
>    Social, Cognitive, and Affective Neuroscience Unit
>    Faculty of Psychology
>    University of Vienna
> 
>    Liebiggasse 5
>    1010 Vienna, Austria
> 
>    e: maximilian.steininger using univie.ac.at
>    w: http://scan.psy.univie.ac.at
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-- 
Michael Dewey