[R-meta] Multilevel model between-study variance

[James Pustejovsky]

Hi Frederik,

Your interpretation of the parameter estimates is correct, but the
estimator of between-study heterogeneity may be quite imprecise if you have
only 12 studies. You can get a sense of this by computing profile
likelihood  confidence intervals for the variance components:
confint(che.model)
Note that these confidence intervals are based on the assumptions of the
working model, so they are not robust to mis-specification in the same
sense that robust CIs for the average effect size are robust.

Both the point estimate of between-study heterogeneity and its confidence
interval could be pretty sensitive to the assumed value of rho (the
correlation of the sampling errors). Using a smaller value of rho will tend
to produce larger estimates of between-study heterogeneity and somewhat
smaller estimates of within-study heterogeneity. So it's definitely helpful
to use empirical data to inform this assumption (as you've done) and to
conduct sensitivity analyses (as you've also noted).

To your broader question about whether aggregating is state-of-the-art, I
have a new paper (with Man Chen) about exactly this question:
https://jepusto.com/publications/Equivalences-between-ad-hoc-strategies-and-models/
We argue that aggregating is neither correct nor incorrect, but is rather
just a different working model that might or might not be reasonable for
your data. Specifically, we show that aggregating is exactly equivalent to
using a multivariate working model that has a between-study variance
component but no within-study variance component (which we call the
"correlated effects" working model), as in the following:
ce.model <- rma.mv(yi = yi,
                    V = V,
                    random = ~ 1 | Study,
                    data = all_study_designs_combined)
This way of representing the model is helpful because it puts the CE model
on equal footing with the CHE model and allows them to be directly compared
via likelihood ratio tests:
anova(che.model, ce.model)
Or otherwise compared and discussed as alternatives.

One other note: the sensitivity of the variance component estimates is also
affected by the distribution of effect sizes per study. It is therefore
useful to examine and report the range of effect estimates per study. If 10
of your studies each have a single effect and 2 each have many effect
sizes, then the CHE will be much more tenuous than if all twelve studies
have 3 or 4 effect sizes each.

James

On Mon, Jul 8, 2024 at 5:08 AM Frederik Zirn via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Dear R-sig-meta-analysis community,
>
> I am a PhD student conducting a meta-analysis of 12 studies with 41 effect
> sizes. There is dependency among my effect sizes as several studies measure
> the same outcome at multiple time points or multiple outcomes are measured
> in the same study.
>
> My first approach was to aggregate effect sizes per study using the
> agg-function of the Mad package.
> all_study_designs_combined_agg <- agg(id = Study, es = yi, var = vi,
> method = "BHHR", cor = 0.59, data=all_study_designs_combined)
>
> cor = 0.59 is based on the correlations reported within a study of my
> meta-analysis. I am conducting sensitivity analyses with other values.
>
> 1) However, aggregating within studies is no longer considered
> state-of-the-art practice, correct? Or is this still a valid approach to
> handle dependent effect sizes?
>
> Consequently, I aimed to create a multilevel model. Here is the code I
> used:
>
> Fitting a CHE Model with Robust Variance Estimation
> constant sampling correlation assumption rho <- 0.59
>
> constant sampling correlation working model
> V <- with(all_study_designs_combined,
>           impute_covariance_matrix(vi = vi,
>                                    cluster = Study,
>                                    r = rho))
>
> che.model <- rma.mv(yi = yi,
>                     V = V,
>                     random = ~ 1 | Study/ES_ID,
>                     data = all_study_designs_combined)
> che.model
>
> robust variance
> full.model.robust <- robust(che.model,
> cluster=all_study_designs_combined$Study, clubSandwich = TRUE)
> summary(full.model.robust)
>
> Doing so, I receive the following variance components:
> sigma^2.1  0.0000  0.0000     12     no        Study
> sigma^2.2  0.1286  0.3587     41     no  Study/ES_ID
>
> 2) I have trouble interpreting those findings. Does that indicate that all
> of the variance in my model comes from within-study variance, and I have no
> between-study variance? This does not seem plausible to me. Am I overseeing
> something here? Could that be due to the limited sample size (12 studies)?
>
> Thanks in advance,
> Frederik Zirn
> PhD student
> Chair of Corporate Education
> University of Konstanz
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]