[R-meta] rule of thumb miminum number of studies per factor level meta-regression

[Viechtbauer, Wolfgang (SP)]

Hi Frank,

I would say that essentially all conclusions from moderator analyses (whether done by meta-regression, some kind of ANOVA-analogue, subgrouping, or some other technique) are hypothesis generating. Cooper calls this 'synthesis-generated evidence' (in contrast to study-generated evidence, which can be obtained by randomly assigning participants to different levels of a potential moderator of a treatment effect). His book 'Research synthesis and meta-analysis' discusses this distinction.

Best,
Wolfgang

>-----Original Message-----
>From: Frank van Boven [mailto:f.vanboven5 using gmail.com]
>Sent: Monday, 28 March, 2022 13:13
>To: Viechtbauer, Wolfgang (SP)
>Cc: Lena Pollerhoff; r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] rule of thumb miminum number of studies per factor level
>meta-regression
>
>Dear all,
>
>In reply to this explanation, I am wondering.
>When subgrouping the studies (thus no meta-regression).
>Would it be an option to limit the aim of the meta-analysis to only generate
>hypotheses, irrespectfull the number of studies left in each subgroup?
>
>Kind regards,
>
>Frank van Boven
>
>> Op 28 mrt. 2022, om 12:04 heeft Viechtbauer, Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> het volgende geschreven:
>>
>> Dear Lena,
>>
>> Just for the record, the '10 studies per covariate' rule comes from here:
>>
>> https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1
>>
>> where it says:
>>
>> "It is very unlikely that an investigation of heterogeneity will produce useful
>findings unless there is a substantial number of studies. Typical advice for
>undertaking simple regression analyses: that at least ten observations (i.e. ten
>studies in a meta-analysis) should be available for each characteristic modelled.
>However, even this will be too few when the covariates are unevenly distributed
>across studies."
>>
>> I have no idea where the 10 per covariate rule comes from (there is also no
>reference in the Cochrane Handbook) and I am not aware of any empirical support
>for it. I suspect it was just taken over from similar rules that have been
>formulated in other contexts (e.g., regression models with primary data,
>prediction models, factor analysis) where these rules have often been formulated
>without much, if any, empirical support.
>>
>> Given what it says in the Cochrane Handbook, one could read this to imply that
>at least 10 studies per covariate are needed to 'produce useful findings'.
>Without a definition of 'useful findings', I don't even know how to evaluate
>whether such a rule is sensible or not.
>>
>> I am not trying to rag on the Cochrane Handbook. The question about 'k per
>moderator' (or k in general for a meta-analysis) is one of the questions that
>*always* comes up in any course on meta-analysis I teach. It is a good question
>and I have no good answer for it, except to mention that such rules exist (e.g.,
>'10 per covariate'), but that they lack empirical support.
>>
>> Analogously, I am not aware of any evidence-based guidelines with respect to
>your 'k per level' question.
>>
>> So, in the end, I am doing again the same thing as I always do when I get this
>question, which is to provide no good answer.
>>
>> Best,
>> Wolfgang