[R-meta] Questions on Implementing CHE and SCE Models in R

[James Pustejovsky]

Hi Fée,

Let me take your questions a little bit out of order:

1. Heterogeneity

1a. I don't find I^2 a very useful statistic for interpreting the degree of
heterogeneity. I would suggest just reporting the estimated variance
components directly, e.g., between-study SD = X1, within-study SD = X2,
total heterogeneity SD = X3 = sqrt(X1^2 + X2^2).

1b. It's also useful to report prediction intervals (PIs) describing the
expected range of effect sizes that one would see in a new study with a new
outcome measure drawn from the same population as that represented by the
sample of included studies. I find it helpful to report PIs using a level
smaller than 95% (e.g., 67% or 80%), both because 95% seems extreme and
because using a different value provides another signal that PIs are not
the same as confidence intervals.

1c. Where does the var.comp() function come from? What package?

2. Moderator analysis: CHE or SCE? A very long-winded reply:

The SCE working model is structured as it is so that it is equivalent (or
nearly so) to running separate meta-analyses on different subsets of the
data (I wrote about this in gory technical detail in this recent paper:
https://jepusto.com/publications/Equivalences-between-ad-hoc-strategies-and-models/).
The average effect size or meta-regression coefficients estimated for each
subgroup are based only on the data for that subgroup. So in your analysis,
you should end up getting equivalent results whether you run the
meta-regression for each emotion subgroup separately versus running the
whole thing with an SCE working model with subgroups defined by emotion.
The only benefit of using SCE instead of just doing the subgroup analyses
is that it lets you test for contrasts between subgroups—such testing for
whether the effects differ by emotion.

In contrast, a multivariate working model such as the CHE involves some
borrowing of information across the subgroups. If, for example, some of
your studies have effects both on anger and sadness, then using the MV
working model will cause the average effects for each of these emotions to
be pulled toward each other; the average effect for anger will be estimated
in part by using indirect information from the distribution of effects for
sadness. This can be useful and appropriate if one it's theoretically
reasonable to expect that effects on anger should correlate with effects on
sadness. The CHE is a simplification of a fully multivariate working model.
Rather than allowing each pair of emotions to have a separate correlation,
we just assume there is a common (intraclass) correlation between every
pair of emotions, and we also assume that each emotion has the same degree
of heterogeneity.

The practical implication of all this is that it makes conceptual sense to
use the SCE model whenever it makes sense to do separate subgroup analyses
because you're dealing with dimensions that are not closely related (or
where you want to avoid borrowing information from one subgroup when
estimating results for another subgroup). It makes conceptual sense to use
a CHE model (or other multivariate model) whenever it makes sense to do a
multivariate analyses because you're dealing with related outcomes. Even if
using a CHE model for moderator analysis, I think it's important to
understand the extent to which your results rest on how the estimates
involve borrowing-of-information across outcome domains. Pragmatically,
that means comparing results between the CHE working model and the SCE
model. If you get consequentially different results, then your findings
hinge on assumptions about the dependence structure and you'll need to be
quite careful about checking those assumptions.

As an aside, If you're interested in exploring the more nuanced
multivariate specifications, you could try this by changing the struct
argument in rma.mv(). Instead of using struct = "DIAG" for the SCE model,
you can try structure = "UN" for a full multivariate working model or
struct = "HCS" for a heteroskedastic compound symmetric model that is
mid-way in between a full multivariate working model and CHE.

3. SCE specification: See inline comments on your code below.

4. Interpretation
4a. You asked "To determine whether group differences differ significantly
between specific emotions, is it correct to rely on the confidence
intervals (CIs) excluding zero and the Wald test results? Apologies if this
is a basic question—I just want to make sure I’m interpreting the findings
properly."

The CIs excluding zero answers a different question than the Wald tests.
Roughly, checking whether a CI includes zero is answering the question of
whether the average effect size (say for a given emotion) is zero. If the
CI excludes zero, then there is evidence to rule out that possibility. But
the Wald tests are answering the question of whether (more specifically,
testing the null hypothesis that) the average effect sizes are the same for
every category (say, that average effect sizes are the same for anger,
sadness, and happiness). If the test is significant, then there is evidence
to rule out the possibility that the effects are equal---at least one type
of emotion has different average effect size than the others. Because the
tests are getting at different null hypotheses, it's entirely possible that
one would be rejected but the other would not.

4b. You asked "I’ve noticed that, while I have significant effects for
single emotions in the SCE, the Wald test indicates no significance,
suggesting there is no difference between the emotions. Could you help
clarify why this might be the case? Is it common for these results to
differ, and how should I interpret these discrepancies in the context of my
model?"

When you're looking at the CIs for different levels of a moderator, it is
not valid to conclude that average effects differ because the CI for one
level includes zero but the CI for another level excludes zero. The
differences in significance levels do not necessarily imply significant
differences (https://doi.org/10.1198/000313006X152649). If you want to look
at differences between specific pairs of categories, you can calculate
these easily using the linear_contrast() function (
https://jepusto.github.io/clubSandwich/reference/linear_contrast.html),
which works similarly to Wald_test().

James

On Mon, Dec 16, 2024 at 9:32 AM Fée Ona Fuchs MSH Hamburg via
R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:

> (a) General Group Differences in ER (Overall Effect)
>
> # Calculate SE^2
> df_g_acc$vi <- df_g_acc$hedges_g_se^2
>
> # Constant sampling correlation assumption
> rho <- 0.6
>
> # Impute covariance matrix
> V_g_acc <- with(df_g_acc,
>           impute_covariance_matrix(vi = vi,
>                                    cluster = study,
>                                    r = rho))
>
> # Fit model
> che.g_acc <- rma.mv(effectsize_hedges_g ~ 1,
>                     V = V_g_acc,
>                     random = ~ 1 | study/no,
>                     data = df_g_acc,
>                     sparse = TRUE)
>
> # Confidence Intervals
> conf_int(che.g_acc,
>          vcov = "CR2")
> coef_test(che.g_acc,
>           vcov = "CR2")
>
>
You don't need to do both conf_int() and coef_test(). If you want both CIs
and p-values for individual coefficients, use conf_int() with the argument
p_values = TRUE.


> # I²
> i2_che_g_acc <- var.comp(che.g_acc)
> i2_che_g_acc
>
> See questions and comments above about I^2.


> # Robust F-test
> Wald_g_acc <- Wald_test(che.g_acc,
>                         constraints = constrain_zero(1),
>                         vcov = "CR2")
> Wald_g_acc
>
> It's not necessary to do the Wald test for the overall average effect. It
provides the same information as the t-test above.


>
> (b) Emotion-Specific Group Differences (SCE)
>
> # Impute covariance matrix for subgroup emotions
> V_g_acc_emo <- impute_covariance_matrix(df_g_acc_filtered$vi,
>                                         cluster = df_g_acc_filtered$study,
>                                         r = rho,
>                                         smooth_vi = TRUE,
>                                         subgroup = df_g_acc_filtered$av)
>
> # Fit random effects working model
> sce_g_acc_emo <- rma.mv(effectsize_hedges_g ~ 0 + av,
>                       V = V_g_acc_emo,
>                       random = list(~ av | study), struct = "DIAG",
>                       data = df_g_acc_filtered, sparse = TRUE)
> sce_g_acc_emo
>
> # Confidence Intervals
> CI_g_acc_emo <- conf_int(sce_g_acc_emo, vcov = "CR2")
> CI_g_acc_emo
>
> # Robust F-test
> Wald_g_acc_emo <- Wald_test(sce_g_acc_emo,
>                             constraints = constrain_equal(1:7),
>                             vcov = "CR2")
> Wald_g_acc_emo
>
>
This looks correct as an implementation of the SCE working model.

	[[alternative HTML version deleted]]