[R-meta] Computing Effect Size for Difference in Differences with Different Populations

[Viechtbauer, Wolfgang (NP)]

Hi Mika,

Depends on what you mean by 'best'. But note that escalc(measure="SMCRH, ...) computes (m1i-m2i)/sd1i, so you are using the post-treatment SD to standardize, which is a bit unusual and different from your second approach where you use the average pre-treatment SD to standardize. This aside, the two approaches should lead to rather similar estimates, especially if the pre-treatment SDs (assuming those are used in both approaches) are similar across the two groups.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Mika Manninen via R-sig-meta-analysis
>Sent: Monday, 13 March, 2023 17:28
>To: R meta
>Cc: Mika Manninen
>Subject: [R-meta] Computing Effect Size for Difference in Differences with
>Different Populations
>
>Dear community,
>
>I am currently working on a meta-analysis that aims to examine the
>difference in training effects between two populations. Both
>populations underwent the same training, but at pre-test, the groups
>have significantly different means and standard deviations (about
>1-2sd difference in means).
>
>I am interested in computing the effect size for the difference in
>differences between the two groups. Specifically, I would like to know
>what is the best way to calculate the effect size given the
>significant difference in means and standard deviations at pre-test.
>
>Would the below be roughly accurate (Option 1):
>
>Option 1.
>
>G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1,
>sd1i=postsd_G1,ni=n_G1, sd2i = presd_G1, ri=c(rep(0.7,10)), data=G)
>G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2,
>sd1i=postsd_G2, ni=n_G2, sd2i = presd_G2, ri=c(rep(0.7,10)), data=G)
>dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)
>
>Option 2.
>
>ES = (G1 post_mean - G2 pre_mean) - (G2 post_mean - G2 pre_mean) / pldpre_sd
>pldpre_sd = sqrt((presdG1^2 + presdG2^2) / 2)
>
>### dataset
>
>set.seed(123)
>
>postm_G1 <- rnorm(100, mean = 14, sd = 2.5)
>prem_G1 <- rnorm(100, mean = 10, sd = 2)
>postsd_G1 <- rnorm(100, mean = 1.4, sd = 0.2)
>presd_G1 <- rnorm(100, mean = 1, sd = 0.2)
>n_G1 <- rpois(100, lambda = 50)
>
>postm_G2 <- rnorm(100, mean = 7.5, sd = 1.8)
>prem_G2 <- rnorm(100, mean = 5, sd = 1.2)
>postsd_G2 <- rnorm(100, mean = 0.9, sd = 0.2)
>presd_G2 <- rnorm(100, mean = 0.6, sd = 0.2)
>n_G2 <- rpois(100, lambda = 50)
>
>G <- data.frame(postm_G1, prem_G1, postsd_G1, n_G1, presd_G1,
>postm_G2, prem_G2, postsd_G2, n_G2, presd_G2)
>
>###
>
>Thank you in advance for your time and help.
>
>Best wishes,
>Mika