[R-meta] Overlapping CIs with significant difference among subgroups

[Viechtbauer, Wolfgang (SP)]

Dear Rafael,

The SEs of the predicted average outcomes for the various levels can be quite different than the SEs of the difference between levels.

You can get the predicted average outcome for all four levels with:

pred.r = predict(res, transf=transf.ztor, digits=3, newmods = rbind(c(0,0,0), c(1,0,0), c(0,1,0), c(1,1,1)))
pred.r

And then you can plot them with:

forest(pred.r$pred, ci.lb=pred.r$ci.lb, ci.ub=pred.r$ci.ub, slab=c("Without grouping","Temporal grouping","Spatial grouping", "Both"))

Indeed, CIs are wide and overlap. But let's now compute the predicted average difference between levels when compared against the "Without grouping" level:

pred.r = predict(res, transf=transf.ztor, digits=3, newmods = rbind(c(1,0,0) - c(0,0,0), c(0,1,0) - c(0,0,0), c(1,1,1) - c(0,0,0)), intercept=FALSE)
pred.r

forest(pred.r$pred, ci.lb=pred.r$ci.lb, ci.ub=pred.r$ci.ub, slab=c("Diff Temporal grouping","Diff Spatial grouping", "Diff Both"))

Now the CIs are quite narrow and exlude 0.

This aside, I would recommend that you include random effects for species twice, once without and once with the phylogenetic correlation matrix:

h$speciesIDnon <- h$speciesID
res <- rma.mv(zf, vzf, mods=~sce_temporal*sce_spatial, random = list (~1|effectsizeID, ~1|studyID, ~1|speciesIDnon, ~1|speciesID), R=list(speciesID=corr), data=h)

This is model (15) from:

Nakagawa, S., & Santos, E. S. A. (2012). Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26(5), 1253-1274.

Conclusions do not change as far as I can tell, but I would still go with that model. A LRT also shows that this model fits significantly better:

res0 <- rma.mv(zf, vzf, mods=~sce_temporal*sce_spatial, random = list (~1|effectsizeID, ~1|studyID, ~1|speciesID), R=list(speciesID=corr), data=h)
anova(res, res0)

Best,
Wolfgang

>-----Original Message-----
>From: Rafael Rios Moura [mailto:biorafaelrm using gmail.com]
>Sent: Tuesday, 06 October, 2020 20:25
>To: r-sig-meta-analysis using r-project.org; Viechtbauer, Wolfgang (SP)
>Subject: Fwd: Overlapping CIs with significant difference among subgroups
>
>ATTACHMENT(S) REMOVED: dataset.csv | pruned_super-tree.tre | script.R
>
>Dear Wolfgang and All,
>
>Few months ago, I sent this email about a result obtained from a mixed
>effects MLMA, controlling for phylogenetic non-independence. I tested the
>difference between two levels of a moderator and obtained two close means
>(0.39 and 0.31) with highly overlapping CIs. However, the omnibus test
>detected a difference between estimates. Could it be a problem with my code
>or the test? Or am I not using the "predict" function correctly? My dataset
>and script are attached. I am grateful for contributions.
>
>Best wishes,
>_______________________________________________________
>
>Prof. Dr. Rafael Rios Moura
>scientia amabilis
>Coordenador de Pesquisa e do NEPEE/CNPq
>UEMG - Unidade Ituiutaba
>
>ORCID: http://orcid.org/0000-0002-7911-4734
>Currículo Lattes: http://lattes.cnpq.br/4264357546465157
>Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>Rios de Ciência: https://www.youtube.com/channel/UCu2186wIJKji22ai8tvlUfg
>
>---------- Forwarded message ---------
>De: Rafael Rios <biorafaelrm using gmail.com>
>Date: seg., 1 de jun. de 2020 às 16:53
>Subject: Overlapping CIs with significant difference among subgroups
>To: <r-sig-meta-analysis using r-project.org>, Viechtbauer Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl>
>
>Dear Wolfgang and All,
>
>I conducted a multilevel mixed-effects meta-analysis and found differences
>between levels of two moderators. I was expecting to find non-overlapped
>confidence intervals. However, I obtained overlapped confidence intervals
>for all subgroups. How can I interpret these results? In such situation,
>should I trust in the Q-test or in the CIs? I controlled for phylogenetic
>non-independence. Is there a chance of this approach affect the estimation
>of CIs using predict function? My dataset and script are attached.
>
>Best wishes,
>_______________________________________________________
>
>Prof. Dr. Rafael Rios Moura
>Coordenador de Pesquisa e do NEPEE/CNPq
>Laboratório de Ecologia e Zoologia (LEZ)
>UEMG - Unidade Ituiutaba
>
>ORCID: http://orcid.org/0000-0002-7911-4734
>Currículo Lattes: http://lattes.cnpq.br/4264357546465157
>Research Gate: https://www.researchgate.net/profile/Rafael_Rios_Moura2
>Rios de Ciência: https://www.youtube.com/channel/UCu2186wIJKji22ai8tvlUfg