[R-meta] differences in within subject standardized mean difference estimated via means / SD's compared with test statistics

[Viechtbauer, Wolfgang (NP)]

Dear Brendan,

The means/SEs you report just don't match up with the F-statistic. Let's first compute the SDs for the two conditions:

sd_condition1 <- se_condition1 * sqrt(sample_size)
sd_condition2 <- se_condition2 * sqrt(sample_size)

Then the SD of the differences is:

sd_d <- sqrt(sd_condition1^2 + sd_condition2^2 - 2*assumed_r*sd_condition1*sd_condition2)

And then the F-statistic (assuming a one-way repeated-measures ANOVA or, equivalently, a paired-samples t-test, whose test statistic we square) is:

((mean_condition1 - mean_condition2) / (sd_d / sqrt(sample_size)))^2

which yields 52.5, a far cry from 5.78.

The F-statistic will be smallest when assuming r = -1, in which case we get an F of about 8.6, which is still larger.

So this F-statistic must come from some other model. Accordingly, 

d_via_t <- t_value / sqrt(sample_size)

isn't going to give you SMCC and it is no surprise that the two don't match up (leaving aside that escalc() will apply a bias correction, which does make a bit of a difference when n=15, but not to the extent that it could explain this discrepancy).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Brendan Hutchinson via R-sig-meta-analysis
>Sent: Saturday, 22 April, 2023 16:53
>To: r-sig-meta-analysis using r-project.org
>Cc: Brendan Hutchinson
>Subject: [R-meta] differences in within subject standardized mean difference
>estimated via means / SD's compared with test statistics
>
>Hi all,
>
>I'm currently conducting a meta-analysis that will be using primarily within-
>subject / repeated measures standardised mean difference as the effect size.
>
>As is common, many studies in my sample do not provide means/standard deviations
>for estimating the effect size, therefore I'm required to use test statistics
>(p's, t's, F's) for some studies.
>
>My issue is that I'm getting, in some instances, considerably large differences
>in effect sizes calculated between the two methods. For example, one study in my
>meta-analysis provides both, which allows me to compare the two. When calculated
>via means / SD's (assuming a correlation of 0.7 between the two repeated
>measures), the effect size estimate is approx 1.77, whereas the effect size
>derived via t test statistic is 0.62 (reproducible example below).
>
>I imagine slight differences between the two would be normal, but differences
>this large have me scratching my head. I'm wondering if this is a problem on my
>end (for example, potential sources of error could be e.g. in the assumed
>correlation value, that I've transformed SE's to SD's, and have transformed F to
>t), or are these sorts of differences to be expected? More broadly and notably,
>in what circumstance would differences between the two be within the normal
>range, and when would they be sufficient to raise a red flag?
>
>If anybody has any insight, advice, or recommended literature they can point me
>to, that would be much appreciated!
>
>reproducible example:
>
>library(metafor)
>
>mean_condition1 <- 0.197
>se_condition1 <- 0.082
>mean_condition2 <- -0.350
>se_condition2 <- 0.105
>sample_size <- 15
>assumed_r <- 0.7
>F_value <- 5.78
>t_value <- sqrt(F_value)
>
>#via means/SD
>escalc("SMCC", m1i = mean_condition1, sd1i = se_condition1 * sqrt(sample_size),
>       m2i = mean_condition2, sd2i = se_condition2 * sqrt(sample_size),
>       ni = sample_size,
>       ri = assumed_r)
>
>#via t
>d_via_t <- t_value / sqrt(sample_size)
>
>(Note that I've used escalc here simply for streamlining the reproducible
>example. I'm actually using calculations for these derived from Lakens (2013,
>https://doi.org/10.3389/fpsyg.2013.00863) and Borenstein et al (2009
>https://doi.org/10.1002/9780470743386.ch4), and estimates are largely similar.
>Also the example estimates are not the same (since d_via_t is uncorrected) but my
>question remains since that, of course, doesn't explain the difference).
>
>Thanks so much,
>Brendan