[R-meta] Questions on Implementing CHE and SCE Models in R

[Fée Ona Fuchs MSH Hamburg]

Dear James,

thank you so much for your detailed and insightful response – this is incredibly helpful!

To clarify, the varcomp() function comes from the {dmetar} package.

I appreciate the suggestion to report the variance components directly, and will definitely consider that approach.

Looking forward to incorporating your recommendations into my analysis!

Best regards,
Fée O. Fuchs

__________________________________________________________________________

Fée O. Fuchs
PhD student

MSH Medical School Hamburg
University of Applied Sciences and Medical University
Am Kaiserkai 1
20457 Hamburg

________________________________________
Von: James Pustejovsky <jepusto using gmail.com>
Gesendet: Samstag, 21. Dezember 2024 19:25:07
An: R Special Interest Group for Meta-Analysis
Cc: Fée  Ona Fuchs MSH Hamburg
Betreff: Re: [R-meta] Questions on Implementing CHE and SCE Models in R

Hi Fée,

Let me take your questions a little bit out of order:

1. Heterogeneity

1a. I don't find I^2 a very useful statistic for interpreting the degree of heterogeneity. I would suggest just reporting the estimated variance components directly, e.g., between-study SD = X1, within-study SD = X2, total heterogeneity SD = X3 = sqrt(X1^2 + X2^2).

1b. It's also useful to report prediction intervals (PIs) describing the expected range of effect sizes that one would see in a new study with a new outcome measure drawn from the same population as that represented by the sample of included studies. I find it helpful to report PIs using a level smaller than 95% (e.g., 67% or 80%), both because 95% seems extreme and because using a different value provides another signal that PIs are not the same as confidence intervals.

1c. Where does the var.comp() function come from? What package?

2. Moderator analysis: CHE or SCE? A very long-winded reply:

The SCE working model is structured as it is so that it is equivalent (or nearly so) to running separate meta-analyses on different subsets of the data (I wrote about this in gory technical detail in this recent paper: https://jepusto.com/publications/Equivalences-between-ad-hoc-strategies-and-models/). The average effect size or meta-regression coefficients estimated for each subgroup are based only on the data for that subgroup. So in your analysis, you should end up getting equivalent results whether you run the meta-regression for each emotion subgroup separately versus running the whole thing with an SCE working model with subgroups defined by emotion. The only benefit of using SCE instead of just doing the subgroup analyses is that it lets you test for contrasts between subgroups—such testing for whether the effects differ by emotion.

In contrast, a multivariate working model such as the CHE involves some borrowing of information across the subgroups. If, for example, some of your studies have effects both on anger and sadness, then using the MV working model will cause the average effects for each of these emotions to be pulled toward each other; the average effect for anger will be estimated in part by using indirect information from the distribution of effects for sadness. This can be useful and appropriate if one it's theoretically reasonable to expect that effects on anger should correlate with effects on sadness. The CHE is a simplification of a fully multivariate working model. Rather than allowing each pair of emotions to have a separate correlation, we just assume there is a common (intraclass) correlation between every pair of emotions, and we also assume that each emotion has the same degree of heterogeneity.

The practical implication of all this is that it makes conceptual sense to use the SCE model whenever it makes sense to do separate subgroup analyses because you're dealing with dimensions that are not closely related (or where you want to avoid borrowing information from one subgroup when estimating results for another subgroup). It makes conceptual sense to use a CHE model (or other multivariate model) whenever it makes sense to do a multivariate analyses because you're dealing with related outcomes. Even if using a CHE model for moderator analysis, I think it's important to understand the extent to which your results rest on how the estimates involve borrowing-of-information across outcome domains. Pragmatically, that means comparing results between the CHE working model and the SCE model. If you get consequentially different results, then your findings hinge on assumptions about the dependence structure and you'll need to be quite careful about checking those assumptions.

As an aside, If you're interested in exploring the more nuanced multivariate specifications, you could try this by changing the struct argument in rma.mv<http://rma.mv>(). Instead of using struct = "DIAG" for the SCE model, you can try structure = "UN" for a full multivariate working model or struct = "HCS" for a heteroskedastic compound symmetric model that is mid-way in between a full multivariate working model and CHE.

3. SCE specification: See inline comments on your code below.

4. Interpretation
4a. You asked "To determine whether group differences differ significantly between specific emotions, is it correct to rely on the confidence intervals (CIs) excluding zero and the Wald test results? Apologies if this is a basic question—I just want to make sure I’m interpreting the findings properly."

The CIs excluding zero answers a different question than the Wald tests. Roughly, checking whether a CI includes zero is answering the question of whether the average effect size (say for a given emotion) is zero. If the CI excludes zero, then there is evidence to rule out that possibility. But the Wald tests are answering the question of whether (more specifically, testing the null hypothesis that) the average effect sizes are the same for every category (say, that average effect sizes are the same for anger, sadness, and happiness). If the test is significant, then there is evidence to rule out the possibility that the effects are equal---at least one type of emotion has different average effect size than the others. Because the tests are getting at different null hypotheses, it's entirely possible that one would be rejected but the other would not.

4b. You asked "I’ve noticed that, while I have significant effects for single emotions in the SCE, the Wald test indicates no significance, suggesting there is no difference between the emotions. Could you help clarify why this might be the case? Is it common for these results to differ, and how should I interpret these discrepancies in the context of my model?"

When you're looking at the CIs for different levels of a moderator, it is not valid to conclude that average effects differ because the CI for one level includes zero but the CI for another level excludes zero. The differences in significance levels do not necessarily imply significant differences (https://doi.org/10.1198/000313006X152649). If you want to look at differences between specific pairs of categories, you can calculate these easily using the linear_contrast() function (https://jepusto.github.io/clubSandwich/reference/linear_contrast.html), which works similarly to Wald_test().

James

On Mon, Dec 16, 2024 at 9:32 AM Fée Ona Fuchs MSH Hamburg via R-sig-meta-analysis <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>> wrote:
(a) General Group Differences in ER (Overall Effect)

# Calculate SE^2
df_g_acc$vi <- df_g_acc$hedges_g_se^2

# Constant sampling correlation assumption
rho <- 0.6

# Impute covariance matrix
V_g_acc <- with(df_g_acc,
          impute_covariance_matrix(vi = vi,
                                   cluster = study,
                                   r = rho))

# Fit model
che.g_acc <- rma.mv<http://rma.mv>(effectsize_hedges_g ~ 1,
                    V = V_g_acc,
                    random = ~ 1 | study/no,
                    data = df_g_acc,
                    sparse = TRUE)

# Confidence Intervals
conf_int(che.g_acc,
         vcov = "CR2")
coef_test(che.g_acc,
          vcov = "CR2")


You don't need to do both conf_int() and coef_test(). If you want both CIs and p-values for individual coefficients, use conf_int() with the argument p_values = TRUE.

# I²
i2_che_g_acc <- var.comp(che.g_acc)
i2_che_g_acc

See questions and comments above about I^2.

# Robust F-test
Wald_g_acc <- Wald_test(che.g_acc,
                        constraints = constrain_zero(1),
                        vcov = "CR2")
Wald_g_acc

It's not necessary to do the Wald test for the overall average effect. It provides the same information as the t-test above.


(b) Emotion-Specific Group Differences (SCE)

# Impute covariance matrix for subgroup emotions
V_g_acc_emo <- impute_covariance_matrix(df_g_acc_filtered$vi,
                                        cluster = df_g_acc_filtered$study,
                                        r = rho,
                                        smooth_vi = TRUE,
                                        subgroup = df_g_acc_filtered$av)

# Fit random effects working model
sce_g_acc_emo <- rma.mv<http://rma.mv>(effectsize_hedges_g ~ 0 + av,
                      V = V_g_acc_emo,
                      random = list(~ av | study), struct = "DIAG",
                      data = df_g_acc_filtered, sparse = TRUE)
sce_g_acc_emo

# Confidence Intervals
CI_g_acc_emo <- conf_int(sce_g_acc_emo, vcov = "CR2")
CI_g_acc_emo

# Robust F-test
Wald_g_acc_emo <- Wald_test(sce_g_acc_emo,
                            constraints = constrain_equal(1:7),
                            vcov = "CR2")
Wald_g_acc_emo


This looks correct as an implementation of the SCE working model.


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5508 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20250305/01f12540/attachment-0001.p7s>