[R-meta] Overlapping CIs with significant difference among subgroups

[Viechtbauer, Wolfgang (SP)]

Dear Rafael,

CIs can overlap and yet the difference between the two levels can be significant. See, for example:

https://towardsdatascience.com/why-overlapping-confidence-intervals-mean-nothing-about-statistical-significance-48360559900a?gi=b673a691634d

https://www.psychologicalscience.org/observer/understanding-confidence-intervals-cis-and-effect-size-estimation

https://blog.minitab.com/blog/real-world-quality-improvement/common-statistical-mistakes-you-should-avoid

and many more (just google for "test difference overlapping confidence intervals" or something along those lines). They don't talk about meta-analysis per se, but it's the same principle.

So, you can trust the test of the difference between the levels of the moderators.

Best,
Wolfgang

>-----Original Message-----
>From: Rafael Rios [mailto:biorafaelrm using gmail.com]
>Sent: Monday, 01 June, 2020 21:54
>To: r-sig-meta-analysis using r-project.org; Viechtbauer, Wolfgang (SP)
>Subject: Overlapping CIs with significant difference among subgroups
>
>ATTACHMENT(S) REMOVED: dataset.csv | pruned_super-tree.tre | script.R
>
>Dear Wolfgang and All,
>
>I conducted a multilevel mixed-effects meta-analysis and found differences
>between levels of two moderators. I was expecting to find non-overlapped
>confidence intervals. However, I obtained overlapped confidence intervals
>for all subgroups. How can I interpret these results? In such situation,
>should I trust in the Q-test or in the CIs? I controlled for phylogenetic
>non-independence. Is there a chance of this approach affect the estimation
>of CIs using predict function? My dataset and script are attached.
>
>Best wishes,
>_______________________________________________________
>
>Prof. Dr. Rafael Rios Moura
>Coordenador de Pesquisa e do NEPEE/CNPq
>Laboratório de Ecologia e Zoologia (LEZ)
>UEMG - Unidade Ituiutaba
>
>ORCID: http://orcid.org/0000-0002-7911-4734
>Currículo Lattes: http://lattes.cnpq.br/4264357546465157
>Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>Rios de Ciência: https://www.youtube.com/channel/UCu2186wIJKji22ai8tvlUfg