[R-meta] Overlapping CIs with significant difference among subgroups

[Gerta Ruecker]

Dear Rafael,

First of all, the information content of standard errors and confidence 
intervals is identical, they can be transformed into each other. 
Secondly, to present standard errors in a graph, one would probably show 
x ± SE(x) instead of x ± 1.96*SE(x). But what would be the advantage? 
The interpretation of this intercval would mean that the true value is 
covered by 68% of all such intervals (=1-2*(1-pnorm(1))). I don't think 
that this is of more interest than a confidence interval.

The main aim of a forest plot is interval estimation, not statistically 
comparing different studies.

Best,

Gerta

Am 04.06.2020 um 08:26 schrieb Rafael Rios:
> Dear Dr. Wolfgang,
>
> Thank you for the feedback. I was wondering why meta-analysts did not
> exhibit standard errors instead of confidence intervals in graphs. I can
> understand the importance of showing that CIs did not include zero, but
> standard errors can be more informative when comparing subgroups of a
> moderator. This is just a curiosity.
>
> Best wishes,
>
> Rafael.
>
> Em qua, 3 de jun de 2020 às 05:02, Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> escreveu:
>
>> Dear Rafael,
>>
>> What specifically do you mean by "this practice"? Presenting estimated
>> (average) effects with their CIs when subgrouping the studies based on some
>> categorical variable? Indeed, one cannot directly infer based on the CIs
>> whether the subgroups are actually different from each other. For this, one
>> should conduct a proper test of subgroup differences. One can also directly
>> test whether the difference between two effects is significant or not or
>> present an estimate of the difference between two effects with a
>> corresponding CI (and if that CI excludes 0, then one knows that the test
>> of the difference is significant at alpha = (100 - CI level)/100). But I
>> see nothing generally wrong with the practice of presenting subgroup
>> effects with CIs.
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: Rafael Rios [mailto:biorafaelrm using gmail.com]
>>> Sent: Wednesday, 03 June, 2020 5:27
>>> To: Viechtbauer, Wolfgang (SP)
>>> Cc: r-sig-meta-analysis using r-project.org
>>> Subject: Re: Overlapping CIs with significant difference among subgroups
>>>
>>> Dear Dr. Wolfgang,
>>>
>>> Thank you very much! Since confidence intervals are not very informative
>> to
>>> exhibit diferences between subgroups, why is this practice so common among
>>> meta-analysts? Why not to present standard errors instead of CIs?
>>>
>>> Best wishes,
>>>
>>> Rafael.
>>>
>>> Em ter, 2 de jun de 2020 às 03:48, Viechtbauer, Wolfgang (SP)
>>> <wolfgang.viechtbauer using maastrichtuniversity.nl> escreveu:
>>> Dear Rafael,
>>>
>>> CIs can overlap and yet the difference between the two levels can be
>>> significant. See, for example:
>>>
>>> https://towardsdatascience.com/why-overlapping-confidence-intervals-mean-
>>> nothing-about-statistical-significance-48360559900a?gi=b673a691634d
>>>
>>> https://www.psychologicalscience.org/observer/understanding-confidence-
>>> intervals-cis-and-effect-size-estimation
>>>
>>> https://blog.minitab.com/blog/real-world-quality-improvement/common-
>>> statistical-mistakes-you-should-avoid
>>>
>>> and many more (just google for "test difference overlapping confidence
>>> intervals" or something along those lines). They don't talk about meta-
>>> analysis per se, but it's the same principle.
>>>
>>> So, you can trust the test of the difference between the levels of the
>>> moderators.
>>>
>>> Best,
>>> Wolfgang
>>>
>>>> -----Original Message-----
>>>> From: Rafael Rios [mailto:biorafaelrm using gmail.com]
>>>> Sent: Monday, 01 June, 2020 21:54
>>>> To: r-sig-meta-analysis using r-project.org; Viechtbauer, Wolfgang (SP)
>>>> Subject: Overlapping CIs with significant difference among subgroups
>>>>
>>>> ATTACHMENT(S) REMOVED: dataset.csv | pruned_super-tree.tre | script.R
>>>>
>>>> Dear Wolfgang and All,
>>>>
>>>> I conducted a multilevel mixed-effects meta-analysis and found
>> differences
>>>> between levels of two moderators. I was expecting to find non-overlapped
>>>> confidence intervals. However, I obtained overlapped confidence intervals
>>>> for all subgroups. How can I interpret these results? In such situation,
>>>> should I trust in the Q-test or in the CIs? I controlled for phylogenetic
>>>> non-independence. Is there a chance of this approach affect the
>> estimation
>>>> of CIs using predict function? My dataset and script are attached.
>>>>
>>>> Best wishes,
>>>> _______________________________________________________
>>>>
>>>> Prof. Dr. Rafael Rios Moura
>>>> Coordenador de Pesquisa e do NEPEE/CNPq
>>>> Laboratório de Ecologia e Zoologia (LEZ)
>>>> UEMG - Unidade Ituiutaba
>>>>
>>>> ORCID: http://orcid.org/0000-0002-7911-4734
>>>> Currículo Lattes: http://lattes.cnpq.br/4264357546465157
>>>> Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>>>> Rios de Ciência:
>> https://www.youtube.com/channel/UCu2186wIJKji22ai8tvlUfg
>>> --
>>> _______________________________________________________

-- 

Dr. rer. nat. Gerta Rücker, Dipl.-Math.

Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Stefan-Meier-Str. 26, D-79104 Freiburg, Germany

Phone:    +49/761/203-6673
Fax:      +49/761/203-6680
Mail:     ruecker using imbi.uni-freiburg.de
Homepage: https://www.uniklinik-freiburg.de/imbi.html