[R-meta] Meta-analyzing gain effects

[Zhouhan Jin]

Thank you, Wolfgang, for the excellent answer! My goal was to understand which general approach (momentarily leaving the V matrix, and random-effects structure aside) is methodologically more appropriate.

Your last paragraph regarding the equivalence alluded to what I was tinkering with recently.

Specifically, I expected the following 2 approaches to be equivalent, but they are not. The pvals, SEs and CIs are quite larger in the second approach, why?

dat.smc <- read.table(header=T, text="
study time_interval   group      ni   mpre    mpost  sdpre  sdpost
1     pre-post1       treat      28   0.89     5.07   1.40   3.20
1     pre-post2       treat      28   0.89     3.64   1.40   3.15
1     pre-post1       contl      58   1.22     3.52   1.76   2.58
1     pre-post2       contl      58   1.22     2.86   1.76   2.80
2     pre-post1       treat      38   1.89     4.07   0.40   2.20
2     pre-post1       contl      48   2.22     2.52   0.76   1.58
")

# Approach 1:
datT <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,3), data=subset(dat.smc,group=="treat"))

datC <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,3), data=subset(dat.smc,group=="contl"))

dat <- data.frame(yi = datT$yi - datC$yi, vi = datT$vi + datC$vi, time_interval=datT$time_interval)

rma(yi~time_interval+0, vi, data = dat)

# Approach 2:
smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,6), data=dat.smc)

a2 <- rma(yi~time_interval*group, vi, data = smcc)

gr2 <- emmprep(a2)

contrast(gr2, list(gain_dif1 = c(-1,0,1,0), gain_dif2 = c(0,-1,0,1)),
         infer=c(T,T))

Best wishes,

Zhouhan

On Mar 14, 2024 at 10:19 -0400, Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>, wrote:
Dear Zhouhan,

Thanks -- that makes things a lot clearer.

First of all, as presented, I would say that neither approach is appropriate. Both approaches ignore the within-study correlation of multiple effect sizes computed based on the same subjects. In approach 1, the SMD values within studies are correlated. In approach 2, the SMCC values for the same group within studies are correlated. So unless steps are taken to account for this, I would consider both approaches flawed.

Also, using SMDs in the first and SMCCs in the second approach makes the results non-comparable, as the effects are in different metrics. To make things more comparable, one could use SMCR in the second approach. And to make things even more comparable, I would recommend to use SMCRP - which is the standardized mean change using raw score standardization using the pooled SD of the pre- and post-test SDs for the standardization. This is something that I just added recently, so it is in the 'devel' version of metafor, not the CRAN one.

Furthermore, I think a few minor mistakes snuck into the smcc_dat. So let's fix these, use SMCRP, and also allow for correlated estimates. *Very naively*, I will just use 0.5 for all correlations below (to be clear, this is not what I would recommend to do by default). With such a small dataset, I am not going to be adding random effects, but in a larger dataset, this is something one would also have to consider. So here is the code:

############################

library(metafor) # need the 'devel' version for SMCRP below

dat.smd <- read.table(header=T, text="
study time nt nc mt mc sdt sdc
1 pre 28 58 0.89 1.22 1.40 1.76
1 post1 28 58 5.07 3.52 3.20 2.58
1 post2 28 58 3.64 2.86 3.15 2.80
2 pre 38 48 1.89 2.22 0.40 0.76
2 post1 38 48 4.07 2.52 2.20 1.58
")

dat.smc <- read.table(header=T, text="
study time_interval group ni mpre mpost sdpre sdpost
1 pre-post1 treat 28 0.89 5.07 1.40 3.20
1 pre-post2 treat 28 0.89 3.64 1.40 3.15
1 pre-post1 contl 58 1.22 3.52 1.76 2.58
1 pre-post2 contl 58 1.22 2.86 1.76 2.80
2 pre-post1 treat 38 1.89 4.07 0.40 2.20
2 pre-post1 contl 48 2.22 2.52 0.76 1.58
")

if (F) {
dat.smd$sdt <- 2
dat.smd$sdc <- 2
dat.smc$sdpre <- 2
dat.smc$sdpost <- 2
}

library(emmeans)

dat.smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc, data=dat.smd)
dat.smc <- escalc("SMCRP", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(0.5,6), data=dat.smc)

# Approach 1
V1 <- vcalc(vi, cluster=study, obs=time, rho=0.5, data=dat.smd)
res1 <- rma.mv(yi ~ time-1, V1, data=dat.smd)
gr1 <- emmprep(res1)
contrast(gr1, list("diffgain1"=c(1,0,-1), "diffgain2"=c(0,1,-1)))

# Approach 2
V2 <- vcalc(vi, cluster=study, subgroup=group, obs=time_interval, rho=0.5, data=dat.smc)
res2 <- rma.mv(yi ~ 0 + group:time_interval, V2, data=dat.smc)
data=dat.smc)
gr2 <- emmprep(res2)
contrast(gr2, list("diffgain1"=c(-1,1,0,0), "diffgain2"=c(0,0,-1,1)))

############################

The results are not all too dissimilar here, but this is not guaranteed. Differences arise due to the different ways the mean differences/changes are standardized (either based on the SDs of the two groups at a single time point or based on the SDs at the two time points within a single group) and how these effect sizes are then weighted in the model fitting.

Note that not requiring the pre-post correlations is not a reason to prefer the first approach in my opinion, since one needs to know these anyway in constructing an appropriate V matrix.

Between these approaches, I might actually have a preference for the second one. That approach would in fact be identical (if one uses fixed study effects) to directly computing the difference between the standardized mean changes between the two groups within studies and then meta-analyzing those differences (if multiple such differences can be computed, like in study 1, then one would again have to account for their correlation). That also appears to be the preferred approach of others who have discussed the meta-analysis of such study designs in the past; for example:

Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278. https://doi.org/10.1111/j.2044-8317.1988.tb00901.x

Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364-386. https://doi.org/10.1177/1094428106291059

Best,
Wolfgang

-----Original Message-----
From: Zhouhan Jin <zjin65 using uwo.ca>
Sent: Monday, March 11, 2024 18:20
To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
project.org>; Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer using maastrichtuniversity.nl>
Subject: RE: Meta-analyzing gain effects

Dear Wolfgang,

here is a simple example based on my original post (quasi-experimental studies
i.e., unequal baselines). Q: Which approach to estimating the gain effects is
methodologically more appropriate and common?

# Approach 1: Data to compute SMDs at each time point
smd_dat <- read.table(header=T, text="
study time nt nc mt mc sdt sdc
1 pre 28 58 0.89 1.22 1.40 1.76
1 post1 28 58 5.07 3.52 3.20 2.58
1 post2 28 58 3.64 2.86 3.15 2.80
2 pre 38 48 1.89 2.22 0.40 0.76
2 post1 38 48 4.07 2.52 2.20 1.58
")

# Approach 2: Same data reformatted to compute gains before meta-analysis
smcc_dat <- read.table(header=T, text="
study time_interval group ni mpre mpost sdpre sdpost
1 pre-post1 treat 28 .89 5.07 1.40 3.2
1 pre-post2 treat 28 .89 3.64 1.40 3.15
1 pre-post1 contl 58 1.22 3.52 1.76 2.58
1 pre-post2 contl 58 1.22 2.86 1.76 3.15
2 pre-post1 treat 38 2.22 4.07 0.40 2.20
2 pre-post1 contl 48 2.22 1.58 0.76 1.58
")

library(emmeans)

smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc,
data=smd_dat)
smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
ri=rep(.5,6), data=smcc_dat) # needs ri

# Approach 1
a1 <- rma(yi ~ time-1, vi, data = smd)

gr1 <- emmprep(a1)

# Contrast hypotheses to estimate gains meta-analytically
contrast(gr1, list("gain1"=c(1,0,-1), "gain2"=c(0,1,-1)))

# Approach 2
a2 <- rma(yi ~ time_interval*group, vi, data = smcc)

gr2 <- emmprep(a2)

# Just get the EMMs, don't run hypothesis:
emmeans(gr2, ~ time_interval*group)

Best wishes,

Zhouhan
On Mar 11, 2024 at 10:51 -0400, Viechtbauer, Wolfgang (NP)
<wolfgang.viechtbauer using maastrichtuniversity.nl>, wrote:

Dear Zhouhan,

Could you provide a small reproducible toy example illustrating the two
different approaches you are contrasting below? I could provide me own
interpretation of what it is that you are describing, but it would be a lot
easier if you show an example.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
Of Zhouhan Jin via R-sig-meta-analysis
Sent: Monday, March 11, 2024 15:27
To: r-sig-meta-analysis using r-project.org
Cc: Zhouhan Jin <zjin65 using uwo.ca>
Subject: [R-meta] Meta-analyzing gain effects

Dear R meta Community,
(reposting this as I think my first message fell through the cracks)

When meta-analyzing quasi-experimental longitudinal studies, I wonder which
approach I should take to estimate the gains:

1- Meta-analyze the effects (e.g., SMDs) at each time point and then after
modeling, run appropriate hypotheses to estimate treatments' gains meta-
analytically?

OR

2- Compute the gain effects (e.g., SMCCs in escalc) in the dataset, and meta-
analyze them by a model to estimate the treatments' gains directly?

PS. I personally prefer the first approach as it doesn't directly require the
pre-post correlations.

Best wishes,

Zhouhan

	[[alternative HTML version deleted]]