[R-meta] Sample size and continuity correction

Gerta Ruecker ruecker @end|ng |rom |mb|@un|-|re|burg@de
Fri Aug 28 11:48:38 CEST 2020


Hi Philipp,

Yes, of course. I never said that overlapping and non-significance of 
differences are equivalent. I even didn't define "overlapping" properly. 
My focus was the problem of few studies in a meta-analysis Nelly brought 
up, and my main point is that two studies in a meta-analysis is not the 
same problem as two individuals in a clinical trial: two studies can 
still mean we have thousands of individuals and much information about 
effect sizes. What we don't have is information about between-study 
variance, therefore I think the "overlapping CI heuristic" helpful as a 
caveat.

Best,

Gerta

Am 28.08.2020 um 10:43 schrieb Philipp Doebler:
> Gerta, In the case of two studies there is a caveat w.r.t. to the 
> overlapping CI heuristic (probably also in the three study case, but I 
> do not know a number for that):
>
> If, say, the assumptions of the two-sample t-test hold, then the CIs 
> might overlap, but the t-test might be significant. The significance 
> of the t-test might be seen as an indicator of heterogeneity. 
> Goldstein and Healy (1995) argue in favour of 83% CIs because of this 
> suggestion (I am not sure I buy into that) and there is also a note by 
> Cumming and Finch (2005). Even if the assumptions of the two-sample 
> t-test do not hold, but appropriate CIs are available, the "overlap 
> but significant differences" might still hold.
>
> Harvey Goldstein; Michael J. R. Healy. The Graphical Presentation of a 
> Collection of Means, Journal of the Royal Statistical Society, Vol. 
> 158, No. 1. (1995), p. 175-177.
>
> Cumming, Geoff; Finch, Sue. Inference by Eye: Confidence Intervals and 
> How to Read Pictures of Data, American Psychologist, Vol 60(2), 
> Feb-Mar 2005, p. 170-180.
>
>
> -Philipp
>
>
>
> On Thu, Aug 27, 2020 at 9:24 PM Gerta Ruecker 
> <ruecker using imbi.uni-freiburg.de <mailto:ruecker using imbi.uni-freiburg.de>> 
> wrote:
>
>     Dear Nelly and all,
>
>     With respect to (only) the first question (sample size):
>
>     I think nothing is wrong, at least in principle, with a
>     meta-analysis of
>     two studies. We analyze single studies, so why not combining two of
>     them? They may even include hundreds of patients.
>
>     Of course, it is impossible to obtain a decent estimate of the
>     between-study variance/heterogeneity from two or three studies.
>     But if
>     the confidence intervals are overlapping, I don't see any reason to
>     mistrust the pooled effect estimate.
>
>     Best,
>
>     Gerta
>
>
>
>     Am 27.08.2020 um 16:07 schrieb ne gic:
>     > Many thanks for the insights Wolfgang.
>     >
>     > Apologies for my imprecise questions. By "agreed upon" & "what
>     > conclusions/interpretations", I was thinking if there is a
>     minimum sample
>     > size whose pooled estimate can be considered somewhat reliable
>     to produce
>     > robust inferences e.g. inferences drawn from just 2 studies can be
>     > drastically changed by the publication of a third study for
>     instance - but
>     > it seems like there isn't. But I guess readers have to then
>     check this for
>     > themselves to access how much weight they can place on the
>     conclusions of
>     > specific meta-analyses.
>     >
>     > Again, I appreciate it!
>     >
>     > Sincerely,
>     > nelly
>     >
>     > On Thu, Aug 27, 2020 at 3:43 PM Viechtbauer, Wolfgang (SP) <
>     > wolfgang.viechtbauer using maastrichtuniversity.nl
>     <mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>> wrote:
>     >
>     >> Dear nelly,
>     >>
>     >> See my responses below.
>     >>
>     >>> -----Original Message-----
>     >>> From: R-sig-meta-analysis [mailto:
>     >> r-sig-meta-analysis-bounces using r-project.org
>     <mailto:r-sig-meta-analysis-bounces using r-project.org>]
>     >>> On Behalf Of ne gic
>     >>> Sent: Wednesday, 26 August, 2020 10:16
>     >>> To: r-sig-meta-analysis using r-project.org
>     <mailto:r-sig-meta-analysis using r-project.org>
>     >>> Subject: [R-meta] Sample size and continuity correction
>     >>>
>     >>> Dear List,
>     >>>
>     >>> I have general meta-analysis questions that are not
>     >>> platform/software related.
>     >>>
>     >>> *=======================*
>     >>> *1. Issue of few included studies *
>     >>> * =======================*
>     >>> It seems common to see published meta-analyses with few
>     studies e.g. :
>     >>>
>     >>> (A). An analysis of only 2 studies.
>     >>> (B). In another, subgroup analyses ending up with only one
>     study in one of
>     >>> the subgroups.
>     >>>
>     >>> Nevertheless, they still end up providing a pooled estimate in
>     their
>     >>> respective forest plots.
>     >>>
>     >>> So my question is, is there an agreed upon (or rule of thumb,
>     or in your
>     >>> view) minimum number of studies below which meta-analysis becomes
>     >>> unacceptable?
>     >> Agreed upon? Not that I am aware of. Some may want at least 5
>     studies (per
>     >> group or overall), some 10, others may be fine with if one
>     group only
>     >> contains 1 or 2 studies.
>     >>
>     >>> What interpretations/conclusions can one really draw from such
>     analyses?
>     >> That's a vague question, so I can't really answer this in
>     general. Of
>     >> course, estimates will be imprecise when k is small (overall or
>     within
>     >> groups).
>     >>
>     >>> *===================*
>     >>> *2. Continuity correction *
>     >>> * ===================*
>     >>>
>     >>> In studies of rare events, zero events tend to occur and it
>     seems common
>     >> to
>     >>> add a small value so that the zero is taken care of somehow.
>     >>>
>     >>> If for instance, the inclusion of this small value via continuity
>     >>> correction leads to differing results e.g. from
>     non-significant results
>     >>> when not using correction, to significant results when using
>     it, what does
>     >>> make of that? Can we trust such results?
>     >> If this happens, then the p-value is probably fluctuating
>     around 0.05 (or
>     >> whatever cutoff is used for declaring results as significant). The
>     >> difference between p=.06 and p=.04 is (very very unlikely) to be
>     >> significant (Gelman & Stern, 2006). Or, to use the words of
>     Rosnow and
>     >> Rosenthal (1989): "[...] surely, God loves the .06 nearly as
>     much as the
>     >> .05".
>     >>
>     >> Gelman, A., & Stern, H. (2006). The difference between
>     "significant" and
>     >> "not significant" is not itself statistically significant. American
>     >> Statistician, 60(4), 328-331.
>     >>
>     >> Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the
>     >> justification of knowledge in psychological science. American
>     Psychologist,
>     >> 44, 1276-1284.
>     >>
>     >>> If one instead opts to calculate a risk difference instead,
>     and test that
>     >>> for significance, would this be a better solution (more
>     reliable result?)
>     >>> to the continuity correction problem above?
>     >> If one is worried about the use of 'continuity corrections',
>     then I think
>     >> the more appropriate reaction is to use 'exact likelihood'
>     methods (such as
>     >> using (mixed-effects) logistic regression models or
>     beta-binomial models)
>     >> instead of switching to risk differences (nothing wrong with
>     the latter,
>     >> but risk differences are really a fudamentally different effect
>     size
>     >> measure compared to risk/odds ratios).
>     >>
>     >>> Looking forward to hearing your views as diverse as they may
>     be in cases
>     >>> where there is no consensus.
>     >>>
>     >>> Sincerely,
>     >>> nelly
>     >       [[alternative HTML version deleted]]
>     >
>     > _______________________________________________
>     > R-sig-meta-analysis mailing list
>     > R-sig-meta-analysis using r-project.org
>     <mailto:R-sig-meta-analysis using r-project.org>
>     > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>     -- 
>
>     Dr. rer. nat. Gerta Rücker, Dipl.-Math.
>
>     Institute of Medical Biometry and Statistics,
>     Faculty of Medicine and Medical Center - University of Freiburg
>
>     Stefan-Meier-Str. 26, D-79104 Freiburg, Germany
>
>     Phone:    +49/761/203-6673
>     Fax:      +49/761/203-6680
>     Mail: ruecker using imbi.uni-freiburg.de
>     <mailto:ruecker using imbi.uni-freiburg.de>
>     Homepage:
>     https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker
>
>     _______________________________________________
>     R-sig-meta-analysis mailing list
>     R-sig-meta-analysis using r-project.org
>     <mailto:R-sig-meta-analysis using r-project.org>
>     https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>
>
> -- 
> Prof. Dr. Philipp Doebler
> Technische Universität Dortmund
> Fakultät Statistik
> Vogelpothsweg 87
> 44227 Dortmund
>
> Tel.: +49 231-755 8259
> Fax: +49 231-755 3918
> doebler using statistik.tu-dortmund.de <mailto:doebler using statistik.tu-dortmund.de>
> www.statistik.tu-dortmund.de/1261.html 
> <http://www.statistik.tu-dortmund.de/1261.html>
>
>
> Wichtiger Hinweis: Die Information in dieser E-Mail ist vertraulich. Sie
> ist ausschließlich für den Adressaten bestimmt. Sollten Sie nicht der
> für diese E-Mail bestimmte Adressat sein, unterrichten Sie bitte den
> Absender und vernichten Sie diese Mail. Vielen Dank.
> Unbeschadet der Korrespondenz per E-Mail, sind unsere Erklärungen
> ausschließlich final rechtsverbindlich, wenn sie in herkömmlicher
> Schriftform (mit eigenhändiger Unterschrift) oder durch Übermittlung
> eines solchen Schriftstücks per Telefax erfolgen.
>
> Important note: The information included in this e-mail is confidential.
> It is solely intended for the recipient. If you are not the intended
> recipient of this e-mail please contact the sender and delete this
> message. Thank you.
> Without prejudice of e-mail correspondence, our statements are only
> legally binding when they are made in the conventional written form
> (with personal signature) or when such documents are sent by fax.

-- 

Dr. rer. nat. Gerta Rücker, Dipl.-Math.

Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Stefan-Meier-Str. 26, D-79104 Freiburg, Germany

Phone:    +49/761/203-6673
Fax:      +49/761/203-6680
Mail:     ruecker using imbi.uni-freiburg.de
Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker


	[[alternative HTML version deleted]]



More information about the R-sig-meta-analysis mailing list