[R-meta] Meta-analysis of two multi-level studies

[Wojciech Raczyński]

Dear Group,

I am new to the Metafor package but eager to learn how to use it properly.
Saying that, please excuse my (probably silly) questions, but I'd be really
grateful if you could provide some suggestions on how to run my very first
meta-analysis across two studies I conducted.

I have conducted two studies: Study 1 and Study 2. Study 2 is a replication
of Study 1.
I want to run a meta-analysis to estimate the true effects of the links
between my three DVs and each of the IVs. I am now wondering how to
approach it best. Allow me to say a few words about Study 1:
There were ~3000 students (level 1) from 30 schools (level 2). DV:
self-concept scale consisting of three subscales: Family, Friends, and
School. All three subscales are highly related to each other (r = ~0.8).
For theoretical reasons, I run three separate multi-level models, with each
of the subscales as DV. I regressed these three DVs on 8 IVs, which I
believe are linked to the DV (multi-level design). Of important note - two
of those IVs are in squared form, that is, IV and IV^2 (as I observed
U-shaped relationships). Thus, I ended up with 10 standardized coefficients
plus an intercept.
I replicated Study 1 (Study 2) on a set of 25 different schools and
different 2500 students, keeping all others the same (IVs and DVs).

initially, I thought about converting my standardized coefficients into r,
but I searched through the Archives and found this response (
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2019-December/001887.html).
So, I figured I'd simply use the standardized coefficient in my
meta-analysis (yi). I computed variance by squaring SE (vi). But now I'm
puzzled what to do next.

Possible scenarios I figured would be to:
1) Run 10 x 3 = 30 separate meta-analyses for 10 predictors for 3 subscales
separately
res_DVsubscale1_IV1 <- rma(yi, vi, data = dat_DVsubscale1_IV1)
res_DVsubscale1_IV1
(repeat 30 times)

2) Run a multi-level meta-analysis and cluster the data in IVs and Subscales
res <- rma.mv(yi, vi, mods = ~ IV, random = ~ 1 | IV/DV_Subscale, data=dat)
res

But then I'd get general estimates of IVs across three subscales, and I
want separate ones for each of the Subscales... So maybe an interaction:
res_interaction <- rma.mv(yi, vi, mods = ~ IV * DV_Subscale, random = ~ 1 |
IV/DV_Subscale, data=dat)
res_interaction

But then I'd get estimates for IVs for Subscale 2 and Subscale 3 but not
Subscale 1. Plus, it would compare the effects of IVs for DV2 and DV3 to
those of DV1, which is not what I want...

3) Run three multi-level meta-analyses (for each of the three subscales)
and cluster the data in IVs
res_DVsubscale1 <- rma.mv(yi, vi, mods = ~ IV, random = ~ 1 | IV,
data=dat_DVsubscale1)
res_DVsubscale1

And although I was leaning towards this option, I saw weird results
regarding my squared terms. Although in two multi-level models in Study 1
and Study 2 they are both significantly related to DV (i.e., IV p < .001,
and IV^2 p < .001), when running the above, they are grossly not
significant (i.e., p = .9). and with completely different estimates (IV β =
-.2, IV^2 β = .2 in Study 1 and IV β = -.4, IV^2 β = .4 in Study 2). What's
going on?
Extra questions:
a) Should I include the original intercept from my multi-level models from
Study 1 and Study 2? And exclude the intercept from the rma/rma.mv ("-1")?
Or maybe manually set the rma/rma.mv intercept to ... like a mean of the
two original intercepts?

b) Regardless of how to approach it, I would love to do a Foster Plot
showing all IVs divided by Subscale. I searched through the website, and
the closest forest plot example I could find was
https://www.metafor-project.org/doku.php/tips:forest_plot_with_aggregated_values.
Still, it's not what I was looking for - I'd want on the left to be IV (in
the example provided "Study", and next to it, my three DVs (In the example
provided "Estimates"). I also found in Archive the suggestion to use
forest() from meta package, but maybe anyone of you has manager to plot it
with metafor package?

Kind regards,
Wojciech Raczyński

	[[alternative HTML version deleted]]