[R-meta] Questions about multilevel meta-analysis structure

[James Pustejovsky]

Hi Isaac,

Comments inline below. (You've hit on something I'm interested in, so
apologies in advance!)

James

On Thu, Jul 20, 2023 at 12:17 AM Isaac Calvin Saywell via
R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:

>
> 1. Is my model correctly structured to account for dependency using the
> inner | outer formula (see MODEL 1 CODE below) or should I just specify
> random effects at the study and unique effect size level (see MODEL 2 CODE
> below).
>
>
The syntax looks correct to me except for two things. First, the first
argument of each model should presumably be yi = yi rather than vi. Second,
in Model 2, the struct argument is not necessary and will be ignored (it's
only relevant for models where the random effects have inner | outer
structure).

Conceptually, this is an interesting question. Model 1 is theoretically
appealing because it uses a more flexible, general structure than Model 2.
Model 1 is saying that there are different average effects for each
cognitive domain, and each study has a unique set of effects per cognitive
domain that are distinct from each other but can be inter-correlated. In
contrast, Model 2 is saying that the study-level random effects apply
equally to all cognitive domains---if study X has higher-than-average
effects in domain A, then it will have effects in domain B that are equally
higher-than-average.

The big caveat with Model 2 is that it can be hard to fit unless you have
lots of studies, and specifically lots of studies that report effects for
multiple cognitive domains. To figure out if it is feasible to estimate
this model, it can be useful to do some descriptives where you count the
number of studies that include effect sizes from each possible *pair* of
cognitive domains. If some pairs have very few studies, then it's going to
be difficult or impossible to fit the multivariate random effects structure
without imposing further restrictions.

In case it's looking infeasible, there are some other random effects
structures that are intermediate between Model 1 and Model 2, which might
be worth trying:
Model 1.0: random = list(~ cog_domain | study_id, ~ 1 | effectsize_id),
struct = "UN"
Model 1.1: random = list(~ cog_domain | study_id, ~ 1 | effectsize_id),
struct = "HCS"
Model 1.2: random = list(~ cog_domain | study_id, ~ 1 | effectsize_id),
struct = "CS"
Model 1.2 (equivalent specification, I think): random = ~ 1 | study_id /
cog_domain / effectsize_id
Model 2.0: random = list(~ 1 | study_id, ~ 1 | effectsize_id)
Model 2.0 (equivalent specification): random = ~ 1 | study_id /
effectsize_id

So perhaps there is something in between 1.0 and 2.0 that will strike a
balance between theoretical appeal and feasibility.


> 2. If I do need to specify an inner | outer formula to compare effect
> sizes across cognitive domains, then is an unstructured variance-covariance
> matrix ("UN") most appropriate (allowing tau^2 to differ among cognitive
> domains) or should another structure be specified?
>
> See previous response.


> 3. To account for effect size dependency is a variance-covariance matrix
> necessary (this is what my model currently uses) or is it ok to use
> sampling variance of each in the multilevel model.
>

This has been discussed previously on the listserv. My perspective is that
you should use whatever assumptions are most plausible. If you expect that
there really is correlation in the sampling errors (e.g., because the
effect size estimates are based on correlated outcomes measured on the same
set of respondents), then I think it is more defensible to use a
non-diagonal V matrix, as in your current syntax.


>
> 4. When subsetting my data by one cognitive domain and investigating this
> same cognitive domain in a univariate multilevel model the effect estimate
> tends to be lower compared to when all cognitive domains are included in a
> single multilevel model as a moderator, is there a reason for this?
>

Is this true for *all* of the cognitive domains or only one or a few of
them? Your Model 1 and Model 2 use random effects models that assume effect
sizes from different cognitive domains are somewhat related (i.e., the
random effects are correlated within study) and so the average effect for a
given domain will be estimated based in part on the effect size estimates
for that domain and in part by "borrowing information" from other domains
that are correlated with it. Broadly speaking, the consequence of this
borrowing of information is that the average effects will tend to be pulled
toward each other, and thus will be a little less dispersed than if you
estimate effects through subgroup analysis.

The above would explain why some domains would get pulled downward in the
multivariate model compared to the univariate model, but it would not
explain why *all* of the domains are pulled down. If it's really all of
them, then I suspect your data must have some sort of association between
average effect size and the number of effect size estimates per study.
That'd be weird and I'm not really sure how to interpret it. You could
check on this by calculating a variable (call it k_j) that is the number of
effect size estimates reported per study (across any cognitive domain) and
then including that variable as a predictor in Model 1 or Model 2 above.
This would at least tell you if there's something funky going on...

As a bit of an aside, you can do the equivalent of a subgroup analysis
within the framework of a multivariate working model, which might be
another thing to explore to figure out what's going on. To do this, you'll
first need to recalculate your V matrix, setting the subgroup argument to
be equal to cog_domain. This amounts to making the assumption that there is
correlation between effect size estimates *within* the same domain but not
between domains of a given study. Call this new V matrix V_sub. Then try
the following model specifications:

Model 2.1: V = V_sub, random = list(~ cog_domain | study_id, ~ cog_domain |
effectsize_id), struct = c("DIAG","DIAG")
Model 2.2: V = V_sub, random = list(~ cog_domain | study_id, ~ 1 |
effectsize_id), struct = "DIAG",

Model 2.1 should reproduce what you get from running separate models by
subgroup.
Model 2.2 is a slight tweak on that, which assumes that there is a common
within-study, within-subgroup variance instead of allowing this to differ
by subgroup. Model 2.2 is nested in Models 1.0 and 1.1, but not in 1.2.

	[[alternative HTML version deleted]]