[R-meta] Questions Re: I^2

[James Pustejovsky]

If you take the heterogeneity estimates at face value, then what you have
observed implies that there is a lot of within-study heterogeneity in the
effect sizes. Another way, perhaps more direct, to get a sense of the
heterogeneity is just to examine m_1$sigma2 directly and to calculate
profile likelihood confidence intervals using
confint(m_1)
In my view, high within-study heterogeneity does NOT indicate that you
should avoid moderator analysis or drop the within-study heterogeneity term
from the model. Rather, I think it suggests two directions for further
investigation:

1) The model you've specified assumes that the within-study heterogeneity
term is homoskedastic across the included studies. This assumption should
be examined, at minimum, by plotting the effect sizes grouped within study
(as in a forest plot, with one row per study but possibly with multiple
effect size estimates on each row) and looking at the degree of dispersion
in the cloud of effect sizes from each study. (It might be that one or two
of the studies are substantially more heterogeneous than others, perhaps
due to outliers.)

2) If the heterogeneity assumptions are reasonable and there is a lot of
within-study heterogeneity, then that raises the question of what factors
are producing the within-study heterogeneity. What features of the effect
size estimates vary _within_ studies, not just between? Identifying such
features could help you to explain the within-study heterogeneity. Such
features might also be very useful as control variables for examining other
moderators (even study-level moderators) because they could help to soak up
variation in the effect size estimates, making it easier to identify other
patterns of moderation.

One other note: If the effect size estimates that have different ESid's but
the same StudyID are calculated for different outcomes but are based on the
same sample of participants, then it would be useful to treat them as
correlated rather than conditionally independent. More details about how to
do this in metafor can be found here:
https://wviechtb.github.io/metafor/reference/misc-recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-structures
And a much more detailed discussion can be found here:
https://jepusto.com/publications/RVE-Meta-analysis-expanding-the-range/index.html
In your example, I would guess that treating the effect size estimates as
correlated will actually lead to even more within-study heterogeneity (and
possibly a bit less between-study heterogeneity), so it doesn't exactly
help, but still it might be a more plausible model for the data.

James


On Tue, Apr 22, 2025 at 11:24 AM Tori Peña via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Hi again,
>
> I just wanted to follow-up on this email -
>
> I am running a three-level rma.mv model in R. When I calculated I^2, it
> suggested that 78% of heterogeneity is *within *clusters in level two.  I
> wasn't sure if this suggests that we should not conduct moderator analyses
> or if it suggests we should consider a two-level model?  Thanks in advance
> for your guidance!
>
> See code below:
>
> m_1 <- rma.mv(yi = yi,
>                      V = vi,
>                      slab = StudyID,
>                      data = nestedAgg,
>                      random = ~ 1 | StudyID/ESid,
>                      test = "t",
>                      method = "REML")
>
> W <- diag(1/m_1$vi)
> X <- model.matrix(m_1)
> P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> 100 * sum(m_1$sigma2) / (sum(m_1$sigma2) + (m_1$k-m_1$p)/sum(diag(P)))
>
> 100 * m_1$sigma2 / (sum(m_1$sigma2) + (m_1$k-m_1$p)/sum(diag(P)))
>
> [OUTPUT: ## [1] 9.890334 77.989904]
>
> On Tue, Apr 15, 2025 at 11:20 AM Tori Peña <Tori.Pena using stonybrook.edu>
> wrote:
>
> > Hi everyone,
> >
> > I am running a three-level rma.mv model in R. When I calculated I^2, it
> > suggested that 78% of heterogeneity is *within *clusters in level two.  I
> > wasn't sure if this suggests that we should not conduct moderator
> analyses
> > or if it suggests we should consider a two-level model?  Thanks in
> advance
> > for your guidance!
> >
> > See code below:
> >
> > m_1 <- rma.mv(yi = yi,
> >                      V = vi,
> >                      slab = StudyID,
> >                      data = nestedAgg,
> >                      random = ~ 1 | StudyID/ESid,
> >                      test = "t",
> >                      method = "REML")
> >
> > W <- diag(1/m_1$vi)
> > X <- model.matrix(m_1)
> > P <- W - W %*% X %*% solve(t(X) %*% W %*% X) %*% t(X) %*% W
> > 100 * sum(m_1$sigma2) / (sum(m_1$sigma2) + (m_1$k-m_1$p)/sum(diag(P)))
> >
> > 100 * m_1$sigma2 / (sum(m_1$sigma2) + (m_1$k-m_1$p)/sum(diag(P)))
> >
> > [OUTPUT: ## [1] 9.890334 77.989904]
> >
> > Thanks,
> > Tori
> >
> > --
> > *Tori Peña, Ph.D. *(she/her/ella)
> > Cognitive Psychology
> > Dept. of Psychology
> > Stony Brook University
> > Stony Brook, NY 11790-2500
> > [image: Stony Brook University logo]
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]