[R-meta] The correct formula for variance when using SMCR

[Viechtbauer, Wolfgang (SP)]

Dear Colin,

I am referring to Borenstein et al. (2009), Introduction to Meta-Analysis, in this note. The same also applies to the Handbook of Research Synthesis and Meta-Analysis, 2nd and 3rd edition. In these references, we can find discussions of a 'standardized mean change' type of effect size measure (eq. 4.26 in the Intro book, 12.19 in the 2nd, and 11.23 in the 3rd edition of the Handbook). The measure is

d = (mean_1 - mean_2) / S_within

where mean_1 and mean_2 refer to the means at two different timepoints and S_within refers to the SD of the measurements at a single timepoint, so NOT the SD of the change scores. In metafor, I describe this type of measure as a 'standardized mean change using raw score standardization'. This type of measure was first described by:

Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278. https://doi.org/10.1111/j.2044-8317.1988.tb00901.x

See equation 2. The (large-sample) sampling variance for this type of measure was also derived in this article and is given by:

v(d) = 2(1-r)/n + d^2 / (2n),           [1]

where n is the sample size of the group being measured twice. I have double-checked this derivation.

Sidenote: There is some notational inconsistency in the literature as to the use of 'd' and 'g' to describe biased and bias-corrected versions of 'standardized mean difference/change' type measures, but this is not relevant for this discussion.

In the books noted above, the sampling variance is however given as

v(d) = (1/n + d^2/(2n)) * 2(1-r)        [2]
     = 2(1-r)/n + 2(1-r)*d^2 / (2n),

which is similar but not correct (d^2 should NOT be multiplied by 2(1-r)).

Sidenote: The equation in the books seems to be motivated as follows. First, let d_c = (mean_1 - mean_2) / SD_c denote the 'standardized mean change using change score standardization' (so SD_c is the SD of the change scores). The large-sample variance of d_c is v(d_c) = 1/n + d_c^2 / (2n). Now under the assumption that the variance of the measurements at the two measurement occasions is the same, then d = d_c * sqrt(2(1-r)). Hence, this implies v(d) = (1/n + d_c^2 / (2n)) * 2(1-r) (which is identical to [1]), revealing again the mistake in [2], since in that equation, d and not d_c is used.

I hope this clarifies things.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Colin Tredoux
>Sent: Thursday, 09 September, 2021 12:55
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] The correct formula for variance when using SMCR
>
>Dear Wolfgang Viechtbauer, or other parties who may be able to comment:
>
>Thanks so much for your very useful R package, Metafor.
>
>One question, if I may ask it, concerns a section in the code file escalc.R lines
>1878 to 1883, for computing SMCR
>(https://github.com/wviechtb/metafor/blob/master/R/escalc.r)
>
>### large sample approximation to the sampling variance (using corrected (!)
>equation from Borenstein, 2009)
>               if (vtype[i] == "LS2")
>                  vi[i] <- cmi[i]^2 * (2*(1-ri[i])/ni[i] + di[i]^2 / (2*ni[i]))
>                  #vi[i] <- cmi[i]^2 * 2*(1-ri[i]) * (1/ni[i] + di[i]^2 /
>(2*ni[i])) # as in  Borenstein (2009) but this is incorrect
>
>            }
>
>I am wondering which article by Borenstein you are referring to?  The chapter in
>the second edition of the Handbook?
>Has it been corrected in the third edition?
>I see that there might be an inconsistency between the formula reported by Becker
>(1988) and by Borenstein, but am wondering if that is because it is not an SMCR he
>outlines there.
>
>Bottom line - if you have a moment to indicate what the mistake is, I would
>appreciate that.  I will be using Metafor to run a meta-analysis very soon, and am
>considering using SMCR as the effect size, but am a little concerned about
>potential formula issues.
>
>--
>Colin Tredoux