[R-meta] rule of thumb miminum number of studies per factor level meta-regression

[Viechtbauer, Wolfgang (SP)]

Dear Lena,

Just for the record, the '10 studies per covariate' rule comes from here:

https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1

where it says:

"It is very unlikely that an investigation of heterogeneity will produce useful findings unless there is a substantial number of studies. Typical advice for undertaking simple regression analyses: that at least ten observations (i.e. ten studies in a meta-analysis) should be available for each characteristic modelled. However, even this will be too few when the covariates are unevenly distributed across studies."

I have no idea where the 10 per covariate rule comes from (there is also no reference in the Cochrane Handbook) and I am not aware of any empirical support for it. I suspect it was just taken over from similar rules that have been formulated in other contexts (e.g., regression models with primary data, prediction models, factor analysis) where these rules have often been formulated without much, if any, empirical support.

Given what it says in the Cochrane Handbook, one could read this to imply that at least 10 studies per covariate are needed to 'produce useful findings'. Without a definition of 'useful findings', I don't even know how to evaluate whether such a rule is sensible or not.

I am not trying to rag on the Cochrane Handbook. The question about 'k per moderator' (or k in general for a meta-analysis) is one of the questions that *always* comes up in any course on meta-analysis I teach. It is a good question and I have no good answer for it, except to mention that such rules exist (e.g., '10 per covariate'), but that they lack empirical support.

Analogously, I am not aware of any evidence-based guidelines with respect to your 'k per level' question.

So, in the end, I am doing again the same thing as I always do when I get this question, which is to provide no good answer.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Lena Pollerhoff
>Sent: Monday, 28 March, 2022 10:40
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] rule of thumb miminum number of studies per factor level meta-
>regression
>
>Dear list member,
>
>I am conducting meta-regressions in metafor at the moment and have a short
>question regarding rule of thumbs with respect to categorical predictors in meta-
>regression. While we are aware of one rule of thumb that meta-regressions should
>not be considered for fewer than ten studies per covariate (e.g., Cochrane
>Handbook), we were wondering whether such a rule of thumb also exists with
>respect to the minimum number of studies per factor level of a categorical
>variable?
>
>In my case, I am conducting meta-regressions, where the number of studies per
>factor level are sometimes unevenly distributed: For example, k = 22, and I have
>one categorical predictor with three factor levels, with the first one
>represented by only one study, the second one by three studies, and the third one
>including 18 studies.
>
>Thanks in advance and have a nice day!
>Lena Pollerhoff