[R-meta] Computing Effect Size for Difference in Differences with Different Populations

[Viechtbauer, Wolfgang (NP)]

The difference comes down to how the mean changes are standardized. You will find that

G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1, sd1i=pldpre_sd, ni=n_G1, sd2i=postsd_G1, ri=c(rep(0.7,50)), data=G)
G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2, sd1i=pldpre_sd, ni=n_G2, sd2i=postsd_G2, ri=c(rep(0.7,50)), data=G)
dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)

plot(dat$yi, ES, pch=19)
abline(0,1)

gives nearly identical results, the only difference arising due to the bias correction that is not applied to ES. If you have the 'devel' version of metafor installed, you can switch that off with:

G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1, sd1i=pldpre_sd, ni=n_G1, sd2i=postsd_G1, ri=c(rep(0.7,50)), data=G, correct=FALSE)
G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2, sd1i=pldpre_sd, ni=n_G2, sd2i=postsd_G2, ri=c(rep(0.7,50)), data=G, correct=FALSE)
dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)

and then the results are identical.

So, in the first approach (using sd1i=presd_G1 and sd1i=presd_G2, respectively, for the two groups), the mean changes are standardized by the group-specific (pre-treatment) SDs. In the 'ES-approach', the mean changes are standardized by the square root of the average (pre-treatment) variances. That is why I wrote that the two approaches should lead to similar estimates if the pre-treatment SDs are similar across the two groups. But

G$presd_G1 / G$presd_G2

shows that this is not really the case, so the two approaches give different estimates.

In the first approach, you are quantifying the effect size as the difference in the amount of change relative to the variability at the pre-treatment assessment within each group individually. In the second approach, you are quantifying the effect size as the difference in the amount of change relative to the average variability at the pre-treatment assessment.

Neither is right or wrong I would say.

One practical issue: With the ES-approach, you would still have the challenge of deriving the appropriate equation for the sampling variance of ES.

Best,
Wolfgang

>-----Original Message-----
>From: Mika Manninen [mailto:mixu89 using gmail.com]
>Sent: Thursday, 16 March, 2023 23:46
>To: Viechtbauer, Wolfgang (NP)
>Cc: R Special Interest Group for Meta-Analysis
>Subject: Re: [R-meta] Computing Effect Size for Difference in Differences with
>Different Populations
>
>Hey Wolfgang,
>
>Thank you very much for the reply.
>
>My mistake with the sd1i argument, I wasn't supposed to be using the post sds.
>Thank you for pointing that out.
>
>In my data the pre-treatment SDs are not similar (neither are the pre-treatment
>means) between the groups. That is why I am a bit unsure regarding the most
>appropriate ES computation method. I have mostly meta-analysed RCT data in which
>the two groups are almost identical at pre-treatment. I could not really find
>examples in which the difference in treatment response is being compared between
>two different populations receiving the same treatment.
>
>In any case, the following is an okay representation of the actual data and
>depending on which ES computation approach I use, the result looks quite
>different. So, I was wondering if you could help me in deciding which of the two
>is more appropriate or perhaps there is a third option. The data (or a subsection
>of it) can also be meta-analysed with raw effect sizes which does lead to
>different conclusions as well compared to the SMCRH/SMCC approach).
>
>Thank you very much in advance,
>Mika
>
>### dataset
>set.seed(123)
>n_G1 <- rpois(50, lambda = 50)
>n_G2 <- n_G1
>
>postm_G1 <- rnorm(50, mean = 14, sd = 2.5)
>prem_G1 <- rnorm(50, mean = 10, sd = 2)
>postsd_G1 <- rnorm(50, mean = 2.4, sd = 0.4)
>presd_G1 <- rnorm(50, mean = 1.9, sd = 0.3)
>
>postm_G2 <- rnorm(50, mean = postm_G1 - 7.5, sd = 1.8)
>prem_G2 <- rnorm(50, mean = prem_G1 - 5, sd = 1.2)
>postsd_G2 <- rnorm(50, mean = 1.2, sd = 0.2)
>presd_G2 <- rnorm(50, mean = 1, sd = 0.2)
>
>G <- data.frame(prem_G1,presd_G1, postm_G1, postsd_G1, n_G1, prem_G2,presd_G2,
>postm_G2,postsd_G2, n_G2)
>G
>
># Option 1 (could be SMCC as well I suppose)
>G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1, sd1i=presd_G1, ni=n_G1,
>sd2i = postsd_G1, ri=c(rep(0.7,50)), data=G)
>G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2, sd1i=presd_G2, ni=n_G2,
>sd2i = postsd_G2, ri=c(rep(0.7,50)), data=G)
>dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)
>dat
># Crude mean of Effect sizes
>mean(dat$yi)
>
># Option 2
>pldpre_sd = sqrt((presd_G1^2 + presd_G2^2) / 2)
>ES = ((postm_G1 - prem_G1) - (postm_G2 - prem_G2)) / pldpre_sd
>ES
># Crude mean of Effect sizes from this formula
>mean(ES)
>
>to 16. maalisk. 2023 klo 17.36 Viechtbauer, Wolfgang (NP)
>(wolfgang.viechtbauer using maastrichtuniversity.nl) kirjoitti:
>Hi Mika,
>
>Depends on what you mean by 'best'. But note that escalc(measure="SMCRH, ...)
>computes (m1i-m2i)/sd1i, so you are using the post-treatment SD to standardize,
>which is a bit unusual and different from your second approach where you use the
>average pre-treatment SD to standardize. This aside, the two approaches should
>lead to rather similar estimates, especially if the pre-treatment SDs (assuming
>those are used in both approaches) are similar across the two groups.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Mika Manninen via R-sig-meta-analysis
>>Sent: Monday, 13 March, 2023 17:28
>>To: R meta
>>Cc: Mika Manninen
>>Subject: [R-meta] Computing Effect Size for Difference in Differences with
>>Different Populations
>>
>>Dear community,
>>
>>I am currently working on a meta-analysis that aims to examine the
>>difference in training effects between two populations. Both
>>populations underwent the same training, but at pre-test, the groups
>>have significantly different means and standard deviations (about
>>1-2sd difference in means).
>>
>>I am interested in computing the effect size for the difference in
>>differences between the two groups. Specifically, I would like to know
>>what is the best way to calculate the effect size given the
>>significant difference in means and standard deviations at pre-test.
>>
>>Would the below be roughly accurate (Option 1):
>>
>>Option 1.
>>
>>G1 <- escalc(measure="SMCRH", m1i=postm_G1, m2i=prem_G1,
>>sd1i=postsd_G1,ni=n_G1, sd2i = presd_G1, ri=c(rep(0.7,10)), data=G)
>>G2 <- escalc(measure="SMCRH", m1i=postm_G2, m2i=prem_G2,
>>sd1i=postsd_G2, ni=n_G2, sd2i = presd_G2, ri=c(rep(0.7,10)), data=G)
>>dat <- data.frame(yi = G1$yi - G2$yi, vi = G1$vi + G2$vi)
>>
>>Option 2.
>>
>>ES = (G1 post_mean - G2 pre_mean) - (G2 post_mean - G2 pre_mean) / pldpre_sd
>>pldpre_sd = sqrt((presdG1^2 + presdG2^2) / 2)
>>
>>### dataset
>>
>>set.seed(123)
>>
>>postm_G1 <- rnorm(100, mean = 14, sd = 2.5)
>>prem_G1 <- rnorm(100, mean = 10, sd = 2)
>>postsd_G1 <- rnorm(100, mean = 1.4, sd = 0.2)
>>presd_G1 <- rnorm(100, mean = 1, sd = 0.2)
>>n_G1 <- rpois(100, lambda = 50)
>>
>>postm_G2 <- rnorm(100, mean = 7.5, sd = 1.8)
>>prem_G2 <- rnorm(100, mean = 5, sd = 1.2)
>>postsd_G2 <- rnorm(100, mean = 0.9, sd = 0.2)
>>presd_G2 <- rnorm(100, mean = 0.6, sd = 0.2)
>>n_G2 <- rpois(100, lambda = 50)
>>
>>G <- data.frame(postm_G1, prem_G1, postsd_G1, n_G1, presd_G1,
>>postm_G2, prem_G2, postsd_G2, n_G2, presd_G2)
>>
>>###
>>
>>Thank you in advance for your time and help.
>>
>>Best wishes,
>>Mika