[R-meta] Multilevel random effects structure in rma.mv for MetaLab

[Viechtbauer, Wolfgang (SP)]

Hi Sho, Hi Christina,

If I understand the structure correctly, you have studies that may contain one or multiple participant groups and each participant group may contribute one or multiple effect sizes. So:

study  group  effectid  d
-------------------------
1      1      1         .
1      1      2         .
2      1      3         .
3      1      4         .
3      1      5         .
3      2      6         .
3      2      7         .

so in study 1, there is one group contributing two effect sizes, in study 2 there is one group contributing a single effect size, and in study 3 there are two groups, each contributing two effect sizes.

I am a bit confused by your use of "participant". I interpreted this above as meaning groups of participants, but maybe I am misunderstanding and you have effect sizes on single participants (but then I am not sure how you are computing a standardized mean difference on a single subject, unless you are computing a d-like measure for single subject designs). And since you mentioned working with standardized mean differences, I interpret this as indicating that a group of participants actually consists of two groups, for two different treatments, but again, I may be misunderstanding you.

Then one basic structure would be:

res1 <- rma.mv(yi, vi, random = ~ 1 | study/group/effectid, data=dat)

For groups contributing multiple effects, the sampling errors are correlated, so technically one would want to account for this by computing the covariances and including them in the V matrix, but if this is not possible, then the approach above at least accounts for the clustering through the random effects structure. Using cluster-robust inferences methods after fitting such a model would then be a good sensitivity check.

The same model as above can be fitted with:

res2 <- rma.mv(yi, vi, random = list(~ factor(group) | study, ~ 1 | effectid), data=dat)

or with:

dat$study.group <- paste(dat$study, dat$group)
res3 <- rma.mv(yi, vi, random = list(~ 1 | study, ~ factor(effectid) | study.group), data=dat)

These are all equivalent models, just parameterized differently. And to answer one of your question, no, the order of the elements in the list does not matter.

You didn't define what "paper.participant" is supposed to stand for, but I assume you mean such a construction as above. But given the data structure above,

random = list(~ participant | paper, ~ unique_row | paper.participant)

would be overparameterized and

random = list(~ participant | paper, ~ 1 | paper)

doesn't account for heterogeneity at the lowest level, plus you would essentially be adding random effects twice at the paper level, so that's not a good idea.

I am not sure if this really helps.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Christina Bergmann
Sent: Saturday, 06 July, 2019 21:19
To: sho tsuji; r-sig-meta-analysis using r-project.org
Subject: [R-meta] Multilevel random effects structure in rma.mv for MetaLab

Dear all,

We are part of the team behind MetaLab (metalab.stanford.edu), an open
platform for interactive, dynamic meta-analyses from developmental
psychology. (We'd be happy to tell you more, just get in touch!)

The meta-analyses on MetaLab are largely based on standardized mean
differences, comparing responses either within or between participants.
There is, of course, more structure in our data that we would like to model
correctly, so we have a question concerning the correct random effects
structure for our needs in rma.mv.

Typically, we have 1-n effect sizes per participant group, and 1-n effect
sizes from papers. Participant groups do usually not cross papers.

The variables concerned are thus

"unique_row": corresponds to unique effect sizes, equivalent to a row in
the data file

"paper": corresponds to unique paper, which can contain one or multiple
effect sizes ("unique_row")

"participant": corresponds to unique participants, can be different for
each "unique_row" or contribute several "unique_row" within a "paper".

We want to achieve two things:

1. Nest "unique_row" within "paper" (clustering of effect sizes within a
paper) - i.e. have effect-size level random effects for each record in our
meta-analyses

2. Allow correlated random effects for "participant" within "paper" (in
case the same participant contributes multiple effect sizes)

One formula that was proposed to us is the following:

random = list(~ participant | paper, ~ unique_row | paper.participant)

Where, as we understand, the former corresponds to the ~inner | outer
structure. As to the latter, we are wondering whether it would correspond
to the ~ 1 | id structure, and in this case we would not rather want to
write

random = list(~ participant | paper, ~ 1 | paper).

Another question is whether the order in the list of random effects matters.

If you would like to dig deeper, our code lives here:
https://github.com/langcog/metalab2 - There is a github issue, if you would
like to discuss more (because some datasets are more intricate and require
a more complex approach - how fun)
https://github.com/langcog/metalab2/issues/135

An example dataset can be downloaded here:
https://langcog.github.io/metalab2/dataset/inworddb.html

It would then typically be analyzed with:

rma.mv(d_calc, d_var_calc, random = list(~ same_infant_calc | short_cite, 1
| short_cite), data = inworddb.csv)

(where same_infant_calc = participant; short_cite = paper)

Thanks in advance, everyone!

Sho Tsuji and Christina Bergmann

-- 
Researcher at the Max Planck Institute for Psycholinguistics, Language
Development Department, Nijmegen, The Netherlands
Website: sites.google.com/site/chbergma
Blog: cogtales.wordpress.com