[R-meta] Sample size and continuity correction
doeb|er @end|ng |rom @t@t|@t|k@tu-dortmund@de
Fri Aug 28 10:43:44 CEST 2020
Gerta, In the case of two studies there is a caveat w.r.t. to the
overlapping CI heuristic (probably also in the three study case, but I do
not know a number for that):
If, say, the assumptions of the two-sample t-test hold, then the CIs might
overlap, but the t-test might be significant. The significance of the
t-test might be seen as an indicator of heterogeneity. Goldstein and Healy
(1995) argue in favour of 83% CIs because of this suggestion (I am not sure
I buy into that) and there is also a note by Cumming and Finch (2005). Even
if the assumptions of the two-sample t-test do not hold, but appropriate
CIs are available, the "overlap but significant differences" might still
Harvey Goldstein; Michael J. R. Healy. The Graphical Presentation of a
Collection of Means, Journal of the Royal Statistical Society, Vol. 158,
No. 1. (1995), p. 175-177.
Cumming, Geoff; Finch, Sue. Inference by Eye: Confidence Intervals and How
to Read Pictures of Data, American Psychologist, Vol 60(2), Feb-Mar 2005,
On Thu, Aug 27, 2020 at 9:24 PM Gerta Ruecker <ruecker using imbi.uni-freiburg.de>
> Dear Nelly and all,
> With respect to (only) the first question (sample size):
> I think nothing is wrong, at least in principle, with a meta-analysis of
> two studies. We analyze single studies, so why not combining two of
> them? They may even include hundreds of patients.
> Of course, it is impossible to obtain a decent estimate of the
> between-study variance/heterogeneity from two or three studies. But if
> the confidence intervals are overlapping, I don't see any reason to
> mistrust the pooled effect estimate.
> Am 27.08.2020 um 16:07 schrieb ne gic:
> > Many thanks for the insights Wolfgang.
> > Apologies for my imprecise questions. By "agreed upon" & "what
> > conclusions/interpretations", I was thinking if there is a minimum sample
> > size whose pooled estimate can be considered somewhat reliable to produce
> > robust inferences e.g. inferences drawn from just 2 studies can be
> > drastically changed by the publication of a third study for instance -
> > it seems like there isn't. But I guess readers have to then check this
> > themselves to access how much weight they can place on the conclusions of
> > specific meta-analyses.
> > Again, I appreciate it!
> > Sincerely,
> > nelly
> > On Thu, Aug 27, 2020 at 3:43 PM Viechtbauer, Wolfgang (SP) <
> > wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >> Dear nelly,
> >> See my responses below.
> >>> -----Original Message-----
> >>> From: R-sig-meta-analysis [mailto:
> >> r-sig-meta-analysis-bounces using r-project.org]
> >>> On Behalf Of ne gic
> >>> Sent: Wednesday, 26 August, 2020 10:16
> >>> To: r-sig-meta-analysis using r-project.org
> >>> Subject: [R-meta] Sample size and continuity correction
> >>> Dear List,
> >>> I have general meta-analysis questions that are not
> >>> platform/software related.
> >>> *=======================*
> >>> *1. Issue of few included studies *
> >>> * =======================*
> >>> It seems common to see published meta-analyses with few studies e.g. :
> >>> (A). An analysis of only 2 studies.
> >>> (B). In another, subgroup analyses ending up with only one study in
> one of
> >>> the subgroups.
> >>> Nevertheless, they still end up providing a pooled estimate in their
> >>> respective forest plots.
> >>> So my question is, is there an agreed upon (or rule of thumb, or in
> >>> view) minimum number of studies below which meta-analysis becomes
> >>> unacceptable?
> >> Agreed upon? Not that I am aware of. Some may want at least 5 studies
> >> group or overall), some 10, others may be fine with if one group only
> >> contains 1 or 2 studies.
> >>> What interpretations/conclusions can one really draw from such
> >> That's a vague question, so I can't really answer this in general. Of
> >> course, estimates will be imprecise when k is small (overall or within
> >> groups).
> >>> *===================*
> >>> *2. Continuity correction *
> >>> * ===================*
> >>> In studies of rare events, zero events tend to occur and it seems
> >> to
> >>> add a small value so that the zero is taken care of somehow.
> >>> If for instance, the inclusion of this small value via continuity
> >>> correction leads to differing results e.g. from non-significant results
> >>> when not using correction, to significant results when using it, what
> >>> make of that? Can we trust such results?
> >> If this happens, then the p-value is probably fluctuating around 0.05
> >> whatever cutoff is used for declaring results as significant). The
> >> difference between p=.06 and p=.04 is (very very unlikely) to be
> >> significant (Gelman & Stern, 2006). Or, to use the words of Rosnow and
> >> Rosenthal (1989): "[...] surely, God loves the .06 nearly as much as the
> >> .05".
> >> Gelman, A., & Stern, H. (2006). The difference between "significant" and
> >> "not significant" is not itself statistically significant. American
> >> Statistician, 60(4), 328-331.
> >> Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the
> >> justification of knowledge in psychological science. American
> >> 44, 1276-1284.
> >>> If one instead opts to calculate a risk difference instead, and test
> >>> for significance, would this be a better solution (more reliable
> >>> to the continuity correction problem above?
> >> If one is worried about the use of 'continuity corrections', then I
> >> the more appropriate reaction is to use 'exact likelihood' methods
> (such as
> >> using (mixed-effects) logistic regression models or beta-binomial
> >> instead of switching to risk differences (nothing wrong with the latter,
> >> but risk differences are really a fudamentally different effect size
> >> measure compared to risk/odds ratios).
> >>> Looking forward to hearing your views as diverse as they may be in
> >>> where there is no consensus.
> >>> Sincerely,
> >>> nelly
> > [[alternative HTML version deleted]]
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> Dr. rer. nat. Gerta Rücker, Dipl.-Math.
> Institute of Medical Biometry and Statistics,
> Faculty of Medicine and Medical Center - University of Freiburg
> Stefan-Meier-Str. 26, D-79104 Freiburg, Germany
> Phone: +49/761/203-6673
> Fax: +49/761/203-6680
> Mail: ruecker using imbi.uni-freiburg.de
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
Prof. Dr. Philipp Doebler
Technische Universität Dortmund
Tel.: +49 231-755 8259
Fax: +49 231-755 3918
doebler using statistik.tu-dortmund.de
Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie
ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der
für diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den
Absender und vernichten Sie diese Mail. Vielen Dank.
Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen
ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher
Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung
eines solchen Schriftstücks per Telefax erfolgen.
Important note: The information included in this e-mail is confidential.
It is solely intended for the recipient. If you are not the intended
recipient of this e-mail please contact the sender and delete this
message. Thank you.
Without prejudice of e-mail correspondence, our statements are only
legally binding when they are made in the conventional written form
(with personal signature) or when such documents are sent by fax.
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis