[R-meta] Dealing with effect size dependance with a small number of studies

[Danka Puric]

Hi everyone,

Apologies for the long post and lots of questions.

We are doing a meta-analysis where a single study sometimes included more
than one subsample and also the same subsample (same group of participants)
sometimes yielded more than one effect size.

1. Following the Berkey et al. (1998) example in metafor, we tried fitting
the following “basic” model:
nested_UN <-rma.mv(ES_corrected, SV, random = ~ IDeffect | IDstudy, struct
= "UN", data=MA_dat_raw)
where individual effect sizes are nested within studies. This model,
however, produces profile likelihood plots which have flat parts (both for
sigma2.1 and sigma2.2), which (if I’m not mistaken) indicates model
overparametrization. We believe this is most likely due to a small number
of effect sizes (k = 69, from 53 subsamples, from 20 studies).
We tried a similar model with random = ~ IDeffect | IDsubsample, but this
model did not even converge (I assume because the number of effect sizes
per subsample is even smaller than the number of ES per study).
Are we correct in concluding that a multi-level model can not be properly
fit with the data that we have and an alternative approach (RVE or effect
size aggregation) is better suited to the data?

2. If we want to use RVE, would the following model which includes random
effects at all three levels (effect size, subsample, study) be appropriate
in combination with clubSandwich package robust coefficient estimates?
model <-rma.mv(ES_corrected, SV, random =  ~ 1 | IDstudy / IDsubsample/
IDeffect, data=MA_dat_raw)
coef_test(model, vcov = "CR2")
Or should something else be done in order to adequately address the issue
of effect size dependence?

3. The variances for this model are:
Variance Components:
            estim    sqrt  nlvls  fixed                        factor

sigma^2.1  0.0589  0.2427     20     no                       IDstudy

sigma^2.2  0.0250  0.1583     53     no           IDstudy/IDsubsample

sigma^2.3  0.0014  0.0373     69     no  IDstudy/IDsubsample/IDeffect
In other words, there is very little variance at the level of IDeffect,
after Study and Subsample have been taken into account. The profile
likelihood plot for sigma^2.3 does, however, appear to peak at the
corresponding value when “zoomed in” (with xlim=c(0,0.01)).
Should we consider this a satisfactory model, or is the variance at the
level of IDeffect too small to be meaningful? Presumably, this has to do
with the fact that the majority of subsamples (43 out of 53) only
contribute to the MA with one effect size, for 8 subsamples there are 2 ES
per subsample, and in two instances 5 ESs per subsample.

Would an acceptable alternative model be:
nested <- rma.mv(ES_corrected, SV, random = ~ 1 | IDstudy/IDeffect,
data=MA_dat_raw)
Here, we’ve excluded random effects at the subsample level, because it made
more sense to include random effects at the level of individual effect
sizes and the two variables have a substantial overlap. The variances for
this model seem adequate (and their profile plots look fine, too).
Variance Components:
            estim    sqrt  nlvls  fixed            factor

sigma^2.1  0.0678  0.2604     20     no           IDstudy

sigma^2.2  0.0150  0.1223     69     no  IDstudy/IDeffect

4. Finally, we are also interested in examining the effects of a moderator
variable which defines different outcomes. So, in cases when one subsample
produces more than one effect size – sometimes these effect sizes belong to
the same level of the moderator variable (same outcome under different
circumstances) and sometimes they belong to different levels of the
moderator (different outcomes). Theoretically, we would expect “same-level”
ESs to be more correlated than “different-level” ones, but with the small
number of subsamples that report more than one ES this seems impossible to
model. Does the use of clubSandwich robust coefficient already take care of
this?


Thanks a lot in advance!

Happy holidays and all the best,
Danka Purić

	[[alternative HTML version deleted]]