[R-meta] Meta-analyzing gain effects

[Viechtbauer, Wolfgang (NP)]

As I mentioned, one needs to use fixed study effects in the second approach. Actually, since there are two timepoints in study 1, one needs to add study by timepoint interaction effects. See below:

dat.smc <- read.table(header=T, text="
study time_interval   group      ni   mpre    mpost  sdpre  sdpost
1     pre-post1       treat      28   0.89     5.07   1.40   3.20
1     pre-post2       treat      28   0.89     3.64   1.40   3.15
1     pre-post1       contl      58   1.22     3.52   1.76   2.58
1     pre-post2       contl      58   1.22     2.86   1.76   2.80
2     pre-post1       treat      38   1.89     4.07   0.40   2.20
2     pre-post1       contl      48   2.22     2.52   0.76   1.58
")

# Approach 1:
datT <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,3), data=subset(dat.smc,group=="treat"))
datC <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,3), data=subset(dat.smc,group=="contl"))
dat <- data.frame(yi = datT$yi - datC$yi, vi = datT$vi + datC$vi, time_interval=datT$time_interval)
rma(yi ~ 0 + time_interval, vi, data=dat)

# Approach 2:
smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(.5,6), data=dat.smc)
smcc$treat <- ifelse(smcc$group == "treat", 1, 0)
rma(yi ~ 0 + factor(study):time_interval + time_interval:treat, vi, data=smcc)

One can ignore the warning about redundant predictors in the second approach. As you can see, the estimated average effects for treat vs. contl (at post1 and post2) are identical in the two approaches.

While some have a strong preference for using fixed study effects, one can also use random study effects (there has been a long debate about this issue in the literature, not in the context of this specific application, but the use of bivariate models in general). Here, this would amount to:

rma.mv(yi ~ 0 + time_interval + time_interval:group, vi, random = ~ 1 | interaction(study,time_interval), data=smcc)

The results are almost identical. This is not guaranteed though, as the use of random study effects (or here: random study-timepoint effects) does allow for some sharing of information across studies in how the effect size estimates (i.e., the difference between treat vs. contl within studies) would need to be computed for the two approaches to be exactly identical. So the argument is that one should only use information from a given study when computing the observed effect sizes for that study (as in approach 1) and hence using random study effects with approach 2 is flawed. In practice, it often makes very little difference (under certain conditions). Again, if you are interested in this topic, you will have to dig into the literature. But as shown above, approach 2 with fixed study effects is identical to approach 1, so one always go with that if one wants to have this exact equivalence.

Best,
Wolfgang

> -----Original Message-----
> From: Zhouhan Jin <zjin65 using uwo.ca>
> Sent: Friday, March 15, 2024 04:38
> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
> project.org>; Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: RE: Meta-analyzing gain effects
>
> Thank you, Wolfgang, for the excellent answer! My goal was to understand which
> general approach (momentarily leaving the V matrix, and random-effects structure
> aside) is methodologically more appropriate.
>
> Your last paragraph regarding the equivalence alluded to what I was tinkering
> with recently.
>
> Specifically, I expected the following 2 approaches to be equivalent, but they
> are not. The pvals, SEs and CIs are quite larger in the second approach, why?
>
> dat.smc <- read.table(header=T, text="
> study time_interval   group      ni   mpre    mpost  sdpre  sdpost
> 1     pre-post1       treat      28   0.89     5.07   1.40   3.20
> 1     pre-post2       treat      28   0.89     3.64   1.40   3.15
> 1     pre-post1       contl      58   1.22     3.52   1.76   2.58
> 1     pre-post2       contl      58   1.22     2.86   1.76   2.80
> 2     pre-post1       treat      38   1.89     4.07   0.40   2.20
> 2     pre-post1       contl      48   2.22     2.52   0.76   1.58
> ")
>
> # Approach 1:
> datT <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(.5,3), data=subset(dat.smc,group=="treat"))
>
> datC <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(.5,3), data=subset(dat.smc,group=="contl"))
>
> dat <- data.frame(yi = datT$yi - datC$yi, vi = datT$vi + datC$vi,
> time_interval=datT$time_interval)
>
> rma(yi~time_interval+0, vi, data = dat)
>
> # Approach 2:
> smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(.5,6), data=dat.smc)
>
> a2 <- rma(yi~time_interval*group, vi, data = smcc)
>
> gr2 <- emmprep(a2)
>
> contrast(gr2, list(gain_dif1 = c(-1,0,1,0), gain_dif2 = c(0,-1,0,1)),
>          infer=c(T,T))
>
> Best wishes,
>
> Zhouhan
> On Mar 14, 2024 at 10:19 -0400, Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>, wrote:
>
> Dear Zhouhan,
>
> Thanks -- that makes things a lot clearer.
>
> First of all, as presented, I would say that neither approach is appropriate.
> Both approaches ignore the within-study correlation of multiple effect sizes
> computed based on the same subjects. In approach 1, the SMD values within
> studies are correlated. In approach 2, the SMCC values for the same group within
> studies are correlated. So unless steps are taken to account for this, I would
> consider both approaches flawed.
>
> Also, using SMDs in the first and SMCCs in the second approach makes the results
> non-comparable, as the effects are in different metrics. To make things more
> comparable, one could use SMCR in the second approach. And to make things even
> more comparable, I would recommend to use SMCRP - which is the standardized mean
> change using raw score standardization using the pooled SD of the pre- and post-
> test SDs for the standardization. This is something that I just added recently,
> so it is in the 'devel' version of metafor, not the CRAN one.
>
> Furthermore, I think a few minor mistakes snuck into the smcc_dat. So let's fix
> these, use SMCRP, and also allow for correlated estimates. *Very naively*, I
> will just use 0.5 for all correlations below (to be clear, this is not what I
> would recommend to do by default). With such a small dataset, I am not going to
> be adding random effects, but in a larger dataset, this is something one would
> also have to consider. So here is the code:
>
> ############################
>
> library(metafor) # need the 'devel' version for SMCRP below
>
> dat.smd <- read.table(header=T, text="
> study time nt nc mt mc sdt sdc
> 1 pre 28 58 0.89 1.22 1.40 1.76
> 1 post1 28 58 5.07 3.52 3.20 2.58
> 1 post2 28 58 3.64 2.86 3.15 2.80
> 2 pre 38 48 1.89 2.22 0.40 0.76
> 2 post1 38 48 4.07 2.52 2.20 1.58
> ")
>
> dat.smc <- read.table(header=T, text="
> study time_interval group ni mpre mpost sdpre sdpost
> 1 pre-post1 treat 28 0.89 5.07 1.40 3.20
> 1 pre-post2 treat 28 0.89 3.64 1.40 3.15
> 1 pre-post1 contl 58 1.22 3.52 1.76 2.58
> 1 pre-post2 contl 58 1.22 2.86 1.76 2.80
> 2 pre-post1 treat 38 1.89 4.07 0.40 2.20
> 2 pre-post1 contl 48 2.22 2.52 0.76 1.58
> ")
>
> if (F) {
> dat.smd$sdt <- 2
> dat.smd$sdc <- 2
> dat.smc$sdpre <- 2
> dat.smc$sdpost <- 2
> }
>
> library(emmeans)
>
> dat.smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc,
> data=dat.smd)
> dat.smc <- escalc("SMCRP", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(0.5,6), data=dat.smc)
>
> # Approach 1
> V1 <- vcalc(vi, cluster=study, obs=time, rho=0.5, data=dat.smd)
> res1 <- rma.mv(yi ~ time-1, V1, data=dat.smd)
> gr1 <- emmprep(res1)
> contrast(gr1, list("diffgain1"=c(1,0,-1), "diffgain2"=c(0,1,-1)))
>
> # Approach 2
> V2 <- vcalc(vi, cluster=study, subgroup=group, obs=time_interval, rho=0.5,
> data=dat.smc)
> res2 <- rma.mv(yi ~ 0 + group:time_interval, V2, data=dat.smc)
> data=dat.smc)
> gr2 <- emmprep(res2)
> contrast(gr2, list("diffgain1"=c(-1,1,0,0), "diffgain2"=c(0,0,-1,1)))
>
> ############################
>
> The results are not all too dissimilar here, but this is not guaranteed.
> Differences arise due to the different ways the mean differences/changes are
> standardized (either based on the SDs of the two groups at a single time point
> or based on the SDs at the two time points within a single group) and how these
> effect sizes are then weighted in the model fitting.
>
> Note that not requiring the pre-post correlations is not a reason to prefer the
> first approach in my opinion, since one needs to know these anyway in
> constructing an appropriate V matrix.
>
> Between these approaches, I might actually have a preference for the second one.
> That approach would in fact be identical (if one uses fixed study effects) to
> directly computing the difference between the standardized mean changes between
> the two groups within studies and then meta-analyzing those differences (if
> multiple such differences can be computed, like in study 1, then one would again
> have to account for their correlation). That also appears to be the preferred
> approach of others who have discussed the meta-analysis of such study designs in
> the past; for example:
>
> Becker, B. J. (1988). Synthesizing standardized mean-change measures. British
> Journal of Mathematical and Statistical Psychology, 41(2), 257-278.
> https://doi.org/10.1111/j.2044-8317.1988.tb00901.x
>
> Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control
> group designs. Organizational Research Methods, 11(2), 364-386.
> https://doi.org/10.1177/1094428106291059
>
> Best,
> Wolfgang
>
>
> -----Original Message-----
> From: Zhouhan Jin <zjin65 using uwo.ca>
> Sent: Monday, March 11, 2024 18:20
> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
> project.org>; Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: RE: Meta-analyzing gain effects
>
> Dear Wolfgang,
>
> here is a simple example based on my original post (quasi-experimental studies
> i.e., unequal baselines). Q: Which approach to estimating the gain effects is
> methodologically more appropriate and common?
>
> # Approach 1: Data to compute SMDs at each time point
> smd_dat <- read.table(header=T, text="
> study time nt nc mt mc sdt sdc
> 1 pre 28 58 0.89 1.22 1.40 1.76
> 1 post1 28 58 5.07 3.52 3.20 2.58
> 1 post2 28 58 3.64 2.86 3.15 2.80
> 2 pre 38 48 1.89 2.22 0.40 0.76
> 2 post1 38 48 4.07 2.52 2.20 1.58
> ")
>
> # Approach 2: Same data reformatted to compute gains before meta-analysis
> smcc_dat <- read.table(header=T, text="
> study time_interval group ni mpre mpost sdpre sdpost
> 1 pre-post1 treat 28 .89 5.07 1.40 3.2
> 1 pre-post2 treat 28 .89 3.64 1.40 3.15
> 1 pre-post1 contl 58 1.22 3.52 1.76 2.58
> 1 pre-post2 contl 58 1.22 2.86 1.76 3.15
> 2 pre-post1 treat 38 2.22 4.07 0.40 2.20
> 2 pre-post1 contl 48 2.22 1.58 0.76 1.58
> ")
>
> library(emmeans)
>
> smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc,
> data=smd_dat)
> smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(.5,6), data=smcc_dat) # needs ri
>
> # Approach 1
> a1 <- rma(yi ~ time-1, vi, data = smd)
>
> gr1 <- emmprep(a1)
>
> # Contrast hypotheses to estimate gains meta-analytically
> contrast(gr1, list("gain1"=c(1,0,-1), "gain2"=c(0,1,-1)))
>
> # Approach 2
> a2 <- rma(yi ~ time_interval*group, vi, data = smcc)
>
> gr2 <- emmprep(a2)
>
> # Just get the EMMs, don't run hypothesis:
> emmeans(gr2, ~ time_interval*group)
>
> Best wishes,
>
> Zhouhan
> On Mar 11, 2024 at 10:51 -0400, Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>, wrote:
>
> Dear Zhouhan,
>
> Could you provide a small reproducible toy example illustrating the two
> different approaches you are contrasting below? I could provide me own
> interpretation of what it is that you are describing, but it would be a lot
> easier if you show an example.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Zhouhan Jin via R-sig-meta-analysis
> Sent: Monday, March 11, 2024 15:27
> To: r-sig-meta-analysis using r-project.org
> Cc: Zhouhan Jin <zjin65 using uwo.ca>
> Subject: [R-meta] Meta-analyzing gain effects
>
> Dear R meta Community,
> (reposting this as I think my first message fell through the cracks)
>
> When meta-analyzing quasi-experimental longitudinal studies, I wonder which
> approach I should take to estimate the gains:
>
> 1- Meta-analyze the effects (e.g., SMDs) at each time point and then after
> modeling, run appropriate hypotheses to estimate treatments' gains meta-
> analytically?
>
> OR
>
> 2- Compute the gain effects (e.g., SMCCs in escalc) in the dataset, and meta-
> analyze them by a model to estimate the treatments' gains directly?
>
> PS. I personally prefer the first approach as it doesn't directly require the
> pre-post correlations.
>
> Best wishes,
>
> Zhouhan