[R-meta] calculate effect size for pre-post control design

[Viechtbauer, Wolfgang (SP)]

Aside from the excellent advice from James, I just want to add that there is no "true" effect size here (both versions, that is, using raw or change score standardization, are perfectly fine). Also, both versions can be obtained with metafor:

library(metafor)

datA <- escalc(measure="SMCR", m1i=83.91, m2i=83.24, sd1i=14.9, ni=40, ri=.995)
datB <- escalc(measure="SMCR", m1i=80.54, m2i=80.71, sd1i=15.28, ni=41, ri=.995)
dat <- data.frame(yi = datB$yi - datA$yi, vi = datB$vi + datA$vi)
round(dat, 4)

# This yields:
#
#       yi    vi                                                     
# 1 -0.055 5e-04

datA <- escalc(measure="SMCC", m1i=83.91, m2i=83.24, sd1i=14.9, sd2i=14.88, ni=40, ri=.995)
datB <- escalc(measure="SMCC", m1i=80.54, m2i=80.71, sd1i=15.28, sd2i=15.36, ni=41, ri=.995)
dat <- data.frame(yi = datB$yi - datA$yi, vi = datB$vi + datA$vi)
round(dat, 4)

# This yields:
#
#        yi    vi
# 1 -0.5499 0.052  

The slight discrepancy (-.5499 vs -.56) results from escalc() applying a bias correction.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of James Pustejovsky
Sent: Tuesday, 22 January, 2019 5:43
To: Yingkai Yang
Cc: R meta
Subject: Re: [R-meta] calculate effect size for pre-post control design

Yang,

As you noted, the difference between the two effect size metrics is in the
denominator of the standardized mean difference. Standardizing by the
change-score SD is what it sounds like: the denominator is the standard
deviation of the (Post - Pre) change scores. If the SD of the pre-test
scores is equal to the SD of the post-test scores, then SD(Post - Pre) =
SD(Pre) * sqrt(2 * (1 - r)), where r = corr(Post, Pre). So d_prepost =
d_pre / sqrt(2 * (1 - r)) and d_prepost will be larger than d_pre (or
d_post) whenever r > 0.5.

Whether to use change score standardization or pretest/post-test
standardization depends on the characteristics of the studies you are
trying to synthesize. Suppose that your pool of studies includes some that
use pre-post designs and some that use post-test only designs. Change score
standardization is not viable here because there's no way to compute change
scores from the post-test only designs, but post-test standardization will
work so long as the post-test SDs can be obtained for all of the studies.
In syntheses I've worked on, we've used pre-test standardization for
pre-post designs and post-test standardization for studies where pre-test
data is not available, on the assumption that the pre-test and post-test
SDs will usually be similar.

Another consideration is measurement error. If the studies that are
included in your synthesis use different outcome scales with varying
degrees of reliability, then change score standardization will be less
defensible because taking change scores amplifies the effects of
measurement error. In comparison, differential reliability will have a
weaker effect on effect sizes based on pre-test standardization.

One other note: your r seems unrealistically high--was that a typo? If not,
I would recommend checking through the study again to see if there might be
some error.

James

On Mon, Jan 21, 2019 at 7:37 PM Yingkai Yang <yykjiyisuipian using gmail.com>
wrote:

> Hi,
>
> I am wondering how to calculate effect size for pre-post control design.
> For example, I have a data:
>
>
> group A: pre mean= 83.91, post mean= 83.24, preSD = 14.9, postSD= 14.88, n=
> 40;
>
> group B: pre mean= 80.54, post mean= 80.71, preSD = 15.28, postSD= 15.36,
> n= 41;
>
> pre-post r= 0.995 (all data)
> When I used the code in Metafor website
> http://www.metafor-project.org/doku.php/analyses:morris2008?s[]=morris, I
> got an effect size of -0.055.
>
> This is confusing，since the the true effect size [The intervention
> (between-group) effect for change] I got from the raw data is -0.56.
>
> Then, I used the software of Comprehensive Meta-analysis 3. I found that,
> when calculating effect size from pre-post control design, you can select
> to use change score SD, or post score SD. When I choose change score SD,
> the effect size is indeed -0.56, which is correct.
> So, my question is what is the difference between change score SD and post
> score SD (or pre score SD). How to select these SDs?
>
> best,
> Yang