[R-meta] Sample size and continuity correction

[Röver, Christian]

Dear Nelly,

you may need to distinguish between frequentist and Bayesian methods
here.

Firstly, you may wonder how "representative" a small sample can
possibly be of some general population, however, when you think about
it, this is not necessarily an issue tied to small samples -- you could
also think of large samples that are not representative, e.g., due to
selection biases.

Secondly, small sample sizes (small numbers of studies or few events
within a study) may lead to "technical" difficulties for the meta-
analysis methods. Consider for example the normal approximation that is
often utilized in a normal model; this tends to break down e.g. if you
are looking at a log-OR endpoint and you only have one, two or no
events in one of the study arms. Continuity corrections then may help,
but only to a certain degree. Such issues are discussed e.g. by Jackson
and White (2018; https://doi.org/10.1002/bimj.201800071).

You can then substitute the Normal approximation by a more accurate
model (e.g., a Binomial likelihood; see e.g. the proposals discussed by
Seide et al. (2018; https://doi.org/10.1186/s12874-018-0618-3)).
However, many methods may still perform unsatisfactorily for few
studies or few events, essentially because they often rely on many-
study and/or many-event asymptotics.

This is where frequentist and Bayesian methods may behave somewhat
differently. Bayesian methods generally behave reasonably for any study
number or size, however, the asymptotics issue does not completely go
away. For many studes and many events, the prior information that is
formally included in the model tends to make little difference; but the
fewer studies or events you have, the more important the prior
assumptions will become. It hence crucial to convincingly motivate the
prior assumptions you make. A fully Bayesian approach for few studies
and events (based on a binomial model) is described e.g. by Günhan et
al. (2020; https://doi.org/10.1002/jrsm.1370). Within the common normal
model, you usually first of all have to worry about prior specification
for the heterogeneity parameter; we have recently summarized some
guidance here: https://arxiv.org/abs/2007.08352 .

Cheers,

Christian


On Wed, 2020-08-26 at 10:15 +0200, ne gic wrote:
> Dear List,
> 
> I have general meta-analysis questions that are not
> platform/software related.
> 
> *=======================*
> *1. Issue of few included studies *
> * =======================*
> It seems common to see published meta-analyses with few studies e.g.
> :
> 
> (A). An analysis of only 2 studies.
> (B). In another, subgroup analyses ending up with only one study in
> one of
> the subgroups.
> 
> Nevertheless, they still end up providing a pooled estimate in their
> respective forest plots.
> 
> So my question is, is there an agreed upon (or rule of thumb, or in
> your
> view) minimum number of studies below which meta-analysis becomes
> unacceptable?
> 
> What interpretations/conclusions can one really draw from such
> analyses?
> 
> *===================*
> *2. Continuity correction *
> * ===================*
> 
> In studies of rare events, zero events tend to occur and it seems
> common to
> add a small value so that the zero is taken care of somehow.
> 
> If for instance, the inclusion of this small value via continuity
> correction leads to differing results e.g. from non-significant
> results
> when not using correction, to significant results when using it, what
> does
> make of that? Can we trust such results?
> 
> If one instead opts to calculate a risk difference instead, and test
> that
> for significance, would this be a better solution (more reliable
> result?)
> to the continuity correction problem above?
> 
> 
> Looking forward to hearing your views as diverse as they may be in
> cases
> where there is no consensus.
> 
> Sincerely,
> nelly
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis