[R-meta] Meta-analyzing gain effects
Viechtbauer, Wolfgang (NP)
wo||g@ng@v|echtb@uer @end|ng |rom m@@@tr|chtun|ver@|ty@n|
Thu Mar 14 15:19:12 CET 2024
Dear Zhouhan,
Thanks -- that makes things a lot clearer.
First of all, as presented, I would say that neither approach is appropriate. Both approaches ignore the within-study correlation of multiple effect sizes computed based on the same subjects. In approach 1, the SMD values within studies are correlated. In approach 2, the SMCC values for the same group within studies are correlated. So unless steps are taken to account for this, I would consider both approaches flawed.
Also, using SMDs in the first and SMCCs in the second approach makes the results non-comparable, as the effects are in different metrics. To make things more comparable, one could use SMCR in the second approach. And to make things even more comparable, I would recommend to use SMCRP - which is the standardized mean change using raw score standardization using the pooled SD of the pre- and post-test SDs for the standardization. This is something that I just added recently, so it is in the 'devel' version of metafor, not the CRAN one.
Furthermore, I think a few minor mistakes snuck into the smcc_dat. So let's fix these, use SMCRP, and also allow for correlated estimates. *Very naively*, I will just use 0.5 for all correlations below (to be clear, this is not what I would recommend to do by default). With such a small dataset, I am not going to be adding random effects, but in a larger dataset, this is something one would also have to consider. So here is the code:
############################
library(metafor) # need the 'devel' version for SMCRP below
dat.smd <- read.table(header=T, text="
study time nt nc mt mc sdt sdc
1 pre 28 58 0.89 1.22 1.40 1.76
1 post1 28 58 5.07 3.52 3.20 2.58
1 post2 28 58 3.64 2.86 3.15 2.80
2 pre 38 48 1.89 2.22 0.40 0.76
2 post1 38 48 4.07 2.52 2.20 1.58
")
dat.smc <- read.table(header=T, text="
study time_interval group ni mpre mpost sdpre sdpost
1 pre-post1 treat 28 0.89 5.07 1.40 3.20
1 pre-post2 treat 28 0.89 3.64 1.40 3.15
1 pre-post1 contl 58 1.22 3.52 1.76 2.58
1 pre-post2 contl 58 1.22 2.86 1.76 2.80
2 pre-post1 treat 38 1.89 4.07 0.40 2.20
2 pre-post1 contl 48 2.22 2.52 0.76 1.58
")
if (F) {
dat.smd$sdt <- 2
dat.smd$sdc <- 2
dat.smc$sdpre <- 2
dat.smc$sdpost <- 2
}
library(emmeans)
dat.smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc, data=dat.smd)
dat.smc <- escalc("SMCRP", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost, ri=rep(0.5,6), data=dat.smc)
# Approach 1
V1 <- vcalc(vi, cluster=study, obs=time, rho=0.5, data=dat.smd)
res1 <- rma.mv(yi ~ time-1, V1, data=dat.smd)
gr1 <- emmprep(res1)
contrast(gr1, list("diffgain1"=c(1,0,-1), "diffgain2"=c(0,1,-1)))
# Approach 2
V2 <- vcalc(vi, cluster=study, subgroup=group, obs=time_interval, rho=0.5, data=dat.smc)
res2 <- rma.mv(yi ~ 0 + group:time_interval, V2, data=dat.smc)
data=dat.smc)
gr2 <- emmprep(res2)
contrast(gr2, list("diffgain1"=c(-1,1,0,0), "diffgain2"=c(0,0,-1,1)))
############################
The results are not all too dissimilar here, but this is not guaranteed. Differences arise due to the different ways the mean differences/changes are standardized (either based on the SDs of the two groups at a single time point or based on the SDs at the two time points within a single group) and how these effect sizes are then weighted in the model fitting.
Note that not requiring the pre-post correlations is not a reason to prefer the first approach in my opinion, since one needs to know these anyway in constructing an appropriate V matrix.
Between these approaches, I might actually have a preference for the second one. That approach would in fact be identical (if one uses fixed study effects) to directly computing the difference between the standardized mean changes between the two groups within studies and then meta-analyzing those differences (if multiple such differences can be computed, like in study 1, then one would again have to account for their correlation). That also appears to be the preferred approach of others who have discussed the meta-analysis of such study designs in the past; for example:
Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278. https://doi.org/10.1111/j.2044-8317.1988.tb00901.x
Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364-386. https://doi.org/10.1177/1094428106291059
Best,
Wolfgang
> -----Original Message-----
> From: Zhouhan Jin <zjin65 using uwo.ca>
> Sent: Monday, March 11, 2024 18:20
> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
> project.org>; Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Subject: RE: Meta-analyzing gain effects
>
> Dear Wolfgang,
>
> here is a simple example based on my original post (quasi-experimental studies
> i.e., unequal baselines). Q: Which approach to estimating the gain effects is
> methodologically more appropriate and common?
>
> # Approach 1: Data to compute SMDs at each time point
> smd_dat <- read.table(header=T, text="
> study time nt nc mt mc sdt sdc
> 1 pre 28 58 0.89 1.22 1.40 1.76
> 1 post1 28 58 5.07 3.52 3.20 2.58
> 1 post2 28 58 3.64 2.86 3.15 2.80
> 2 pre 38 48 1.89 2.22 0.40 0.76
> 2 post1 38 48 4.07 2.52 2.20 1.58
> ")
>
> # Approach 2: Same data reformatted to compute gains before meta-analysis
> smcc_dat <- read.table(header=T, text="
> study time_interval group ni mpre mpost sdpre sdpost
> 1 pre-post1 treat 28 .89 5.07 1.40 3.2
> 1 pre-post2 treat 28 .89 3.64 1.40 3.15
> 1 pre-post1 contl 58 1.22 3.52 1.76 2.58
> 1 pre-post2 contl 58 1.22 2.86 1.76 3.15
> 2 pre-post1 treat 38 2.22 4.07 0.40 2.20
> 2 pre-post1 contl 48 2.22 1.58 0.76 1.58
> ")
>
> library(emmeans)
>
> smd <- escalc("SMD", n1i=nt, n2i=nc, m1i=mt, m2i=mc, sd1i=sdt, sd2i=sdc,
> data=smd_dat)
> smcc <- escalc("SMCC", ni=ni, m1i=mpost, m2i=mpre, sd1i=sdpre, sd2i=sdpost,
> ri=rep(.5,6), data=smcc_dat) # needs ri
>
> # Approach 1
> a1 <- rma(yi ~ time-1, vi, data = smd)
>
> gr1 <- emmprep(a1)
>
> # Contrast hypotheses to estimate gains meta-analytically
> contrast(gr1, list("gain1"=c(1,0,-1), "gain2"=c(0,1,-1)))
>
> # Approach 2
> a2 <- rma(yi ~ time_interval*group, vi, data = smcc)
>
> gr2 <- emmprep(a2)
>
> # Just get the EMMs, don't run hypothesis:
> emmeans(gr2, ~ time_interval*group)
>
> Best wishes,
>
> Zhouhan
> On Mar 11, 2024 at 10:51 -0400, Viechtbauer, Wolfgang (NP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl>, wrote:
>
> Dear Zhouhan,
>
> Could you provide a small reproducible toy example illustrating the two
> different approaches you are contrasting below? I could provide me own
> interpretation of what it is that you are describing, but it would be a lot
> easier if you show an example.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Zhouhan Jin via R-sig-meta-analysis
> Sent: Monday, March 11, 2024 15:27
> To: r-sig-meta-analysis using r-project.org
> Cc: Zhouhan Jin <zjin65 using uwo.ca>
> Subject: [R-meta] Meta-analyzing gain effects
>
> Dear R meta Community,
> (reposting this as I think my first message fell through the cracks)
>
> When meta-analyzing quasi-experimental longitudinal studies, I wonder which
> approach I should take to estimate the gains:
>
> 1- Meta-analyze the effects (e.g., SMDs) at each time point and then after
> modeling, run appropriate hypotheses to estimate treatments' gains meta-
> analytically?
>
> OR
>
> 2- Compute the gain effects (e.g., SMCCs in escalc) in the dataset, and meta-
> analyze them by a model to estimate the treatments' gains directly?
>
> PS. I personally prefer the first approach as it doesn't directly require the
> pre-post correlations.
>
> Best wishes,
>
> Zhouhan
More information about the R-sig-meta-analysis
mailing list