[R-meta] Standard Error of an effect size for use in longitudinal meta-analysis

[Viechtbauer, Wolfgang (SP)]

1) Yes, d = (m_1 - m_2) / SD_d (where m_1 and m_2 are the observed means at time 1 and 2 and SD_d is the standard deviation of the change scores) is a biased estimator of (mu_1-mu_2)/sigma_D. An approximatly unbiased estimator is given by:

g = (1 - 3/(4*(n-1) - 1) * d

See, for example:

Gibbons, R. D., Hedeker, D. R., & Davis, J. M. (1993). Estimation of effect size from a series of experiments involving paired comparisons. Journal of Educational Statistics, 18(3), 271-279.

where you can also find the exact correction factor.

Not sure what you mean by correcting the sampling variance (and which variance estimate you are referring to). You can find the equation for an unbiased estimator of the sampling variance of d and g in:

Viechtbauer, W. (2007). Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design. Journal of Educational and Behavioral Statistics, 32(1), 39-60.

In particular, see equations 25 and 26.

2) I don't understand your question.

3) The idea of using 'd_dif' for computing an effect size for independent-groups pretest–posttest design is not new. See Gibbons et al. (1993) and:

Becker, B. J. (1988). Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278.

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105-125.

In this context, one also needs to consider whether one should compute d or g using raw-score or change-score standardization (the former being (m1-m2)/SD_1 where SD_1 is the standard deviations at the first time point). Becker (1988) discusses the use of raw-score standardization, while Gibbons et al. (1993) is focused on change-score standardization. In any case, I think you will find some useful discussions in these articles.

Best,
Wolfgang

>-----Original Message-----
>From: Simon Harmel [mailto:sim.harmel using gmail.com]
>Sent: Tuesday, 11 August, 2020 23:22
>To: Viechtbauer, Wolfgang (SP)
>Cc: R meta
>Subject: Re: Standard Error of an effect size for use in longitudinal meta-
>analysis
>
>Dear Wolfang,
>
>Many thanks for your confirmation! I have some follow-up questions.
>
>(1) I believe "dppc" is biased and requires a correction, so does its
>sampling variance, right?
>
>(2) If "dppc" and "dppt" (along with their sampling variances) are each
>biased and must be corrected, then, we don't need to again correct the
>sampling variance of "d_dif" (i.e., dppt - dppc)?
>
>(3) Do you see any advantage or disadvantage for using "d_dif" in
>longitudinal meta-analysis where studies that have a control group?
>
>[In terms of advantages:
> a- I think logistically using "d_dif" will reduce the number of effect
>sizes that are otherwise usually computed from such studies by half,
> b- "d_dif"'s metric seems to better suit the repeated measures design used
>in the primary studies,
> c- "d_dif" seems to allow for investigating (by using appropriate
>moderators) the threats to the internal validity (regression to mean,
>history, maturation, testing) say when the primary studies didn't use random
>assignment of subjects (i.e., nonequivalent groups designs)
>
>In terms disadvantages:
> a- I think obtaining "r", the correlation bet. pre- and post-tests to
>compute "dppc" and "dppt" is a bit difficult,
> b- while "d_dif"'s metric seems to better suit the repeated measures design
>used in the primary studies, most applied meta-analysts are not used to such
>a metric.
>]
>
>On Tue, Aug 11, 2020 at 2:04 PM Viechtbauer, Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Simon,
>
>Based on what you describe, dppc and dppt appear to be standardized mean
>changes using what I would call "change-score standardization" (i.e., (m1 -
>m2)/SDd, where SDd is the standard deviation of the change scores).
>
>I haven't really tried to figure out the details of what you are doing (and
>I am not familiar with the distr package), but the large-sample sampling
>variances of dppc and dppt are:
>
>vc = 1/nc + dppc^2/(2*nc)
>
>and
>
>vt = 1/nt + dppt^2/(2*nt)
>
>(or to be precise, these are the estimates of the sampling variances, since
>dppc and dppt have been plugged in for the unknown true values).
>
>Hence, the SE of dppt - dppc is simply:
>
>sqrt(vt + vc)
>
>For the "example use" data, this is:
>
>sqrt((1/40 + 0.2^2 / (2*40)) + (1/40 + 0.4^2 / (2*40)))
>
>which yields 0.2291288.
>
>Running your code yields 0.2293698. The former is a large-sample
>approximation while you seem to be using an 'exact' approach, so they are
>not expected to coincide, but should be close, which they are.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Simon Harmel [mailto:sim.harmel using gmail.com]
>>Sent: Tuesday, 11 August, 2020 14:59
>>To: R meta
>>Cc: Viechtbauer, Wolfgang (SP)
>>Subject: Standard Error of an effect size for use in longitudinal meta-
>>analysis
>>
>>Dear All,
>>
>>Suppose I know that the likelihood function for an estimate of effect size
>>(called `dppc`) measuring the change in a "control" group from pre-test to
>>post-test in R language is given by:
>>
>>    like1 <- function(x) dt(dppc*sqrt(nc), df = nc - 1, ncp = x*sqrt(nc))
>>
>>where `dppc` is the observed estimate of effect size, and `nc` is the
>>"control" group's sample size.
>>
>>Similarly, the likelihood function for an estimate of effect size (called
>>`dppt`) measuring the change in a "treatment" group from pre-test to post-
>>test in `R` language is given by:
>>
>>    like2 <- function(x) dt(dppt*sqrt(nt), df = nt - 1, ncp = x*sqrt(nt))
>>
>>where `dppt` is the observed estimate of effect size, and `nt` is the
>>"treatment" group's sample size.
>>
>>>>>"Question:" Is there any way to find the "Standard Error (SE)" of the
>>`d_dif = dppt - dppc` (in `R`)?
>>
>>Below, I tried to first get the likelihood function of `d_dif` and then get
>>the *Standard Deviation* of that likelihood function. In a sense, I assumed
>>I have a Bayesian problem with a "flat prior" and thus "SE" is the standard
>>deviation of the likelihood of `d_dif`.
>>
>>>>> But I am not sure if my work below is at least approximately correct? -
>-
>>Thank you, Simon
>>
>>    library(distr)
>>
>>    d_dif <- Vectorize(function(dppc, dppt, nc, nt){
>>
>>     like1 <- function(x) dt(dppc*sqrt(nc), df = nc - 1, ncp = x*sqrt(nc))
>>     like2 <- function(x) dt(dppt*sqrt(nt), df = nt - 1, ncp = x*sqrt(nt))
>>
>>      d1 <- distr::AbscontDistribution(d = like1, low1 = -15, up1 = 15,
>>withStand = TRUE)
>>      d2 <- distr::AbscontDistribution(d = like2, low1 = -15, up1 = 15,
>>withStand = TRUE)
>>
>>     like.dif <- function(x) distr::d(d2 - d1)(x) ## Gives likelihood of
>the
>>difference i.e., `d_dif`
>>
>>     Mean <- integrate(function(x) x*like.dif(x), -Inf, Inf)[[1]]
>>       SE <- sqrt(integrate(function(x) x^2*like.dif(x), -Inf, Inf)[[1]] -
>>Mean^2)
>>
>>     return(c(SE = SE))
>>    })
>>
>>     # EXAMPLE OF USE:
>>     d_dif(dppc = .2, dppt = .4, nc = 40, nt = 40)