[R-meta] differences in within subject standardized mean difference estimated via means / SD's compared with test statistics

[Michael Dewey]

Dear Brendan

I think the safest course of action is not to use that study. I suppose 
if it is a recent study the authors might be able to help you.

Michael

On 23/04/2023 01:59, Brendan Hutchinson via R-sig-meta-analysis wrote:
> Dear Wolfgang,
> 
> Thank you for the response. Much appreciated.
> 
> This was largely my thinking, as when I manually calculated t I got 7.25:
> 
> mean_dif <- abs(mean_condition1 - mean_condition2)
> sd_dif <- sqrt(sd_condition1^2 + sd_condition2^2 - 2 * assumed_r * sd_condition1 * sd_condition2)
> t_stat_manual <- mean_dif / (sd_dif / sqrt(sample_size))
> 
> Still leaves me scratching my head since these are the numbers reported. I suppose that would suggest some issue in the reporting of the statistics in the original publication itself, meaning I probably have to make a decision about which to trust "more" (or perhaps, trust neither).
> 
> Thank you again,
> Brendan
> ________________________________
> From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
> Sent: Sunday, 23 April 2023 1:49 AM
> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
> Cc: Brendan Hutchinson <Brendan.Hutchinson using alumni.anu.edu.au>
> Subject: RE: differences in within subject standardized mean difference estimated via means / SD's compared with test statistics
> 
> Dear Brendan,
> 
> The means/SEs you report just don't match up with the F-statistic. Let's first compute the SDs for the two conditions:
> 
> sd_condition1 <- se_condition1 * sqrt(sample_size)
> sd_condition2 <- se_condition2 * sqrt(sample_size)
> 
> Then the SD of the differences is:
> 
> sd_d <- sqrt(sd_condition1^2 + sd_condition2^2 - 2*assumed_r*sd_condition1*sd_condition2)
> 
> And then the F-statistic (assuming a one-way repeated-measures ANOVA or, equivalently, a paired-samples t-test, whose test statistic we square) is:
> 
> ((mean_condition1 - mean_condition2) / (sd_d / sqrt(sample_size)))^2
> 
> which yields 52.5, a far cry from 5.78.
> 
> The F-statistic will be smallest when assuming r = -1, in which case we get an F of about 8.6, which is still larger.
> 
> So this F-statistic must come from some other model. Accordingly,
> 
> d_via_t <- t_value / sqrt(sample_size)
> 
> isn't going to give you SMCC and it is no surprise that the two don't match up (leaving aside that escalc() will apply a bias correction, which does make a bit of a difference when n=15, but not to the extent that it could explain this discrepancy).
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>> Behalf Of Brendan Hutchinson via R-sig-meta-analysis
>> Sent: Saturday, 22 April, 2023 16:53
>> To: r-sig-meta-analysis using r-project.org
>> Cc: Brendan Hutchinson
>> Subject: [R-meta] differences in within subject standardized mean difference
>> estimated via means / SD's compared with test statistics
>>
>> Hi all,
>>
>> I'm currently conducting a meta-analysis that will be using primarily within-
>> subject / repeated measures standardised mean difference as the effect size.
>>
>> As is common, many studies in my sample do not provide means/standard deviations
>> for estimating the effect size, therefore I'm required to use test statistics
>> (p's, t's, F's) for some studies.
>>
>> My issue is that I'm getting, in some instances, considerably large differences
>> in effect sizes calculated between the two methods. For example, one study in my
>> meta-analysis provides both, which allows me to compare the two. When calculated
>> via means / SD's (assuming a correlation of 0.7 between the two repeated
>> measures), the effect size estimate is approx 1.77, whereas the effect size
>> derived via t test statistic is 0.62 (reproducible example below).
>>
>> I imagine slight differences between the two would be normal, but differences
>> this large have me scratching my head. I'm wondering if this is a problem on my
>> end (for example, potential sources of error could be e.g. in the assumed
>> correlation value, that I've transformed SE's to SD's, and have transformed F to
>> t), or are these sorts of differences to be expected? More broadly and notably,
>> in what circumstance would differences between the two be within the normal
>> range, and when would they be sufficient to raise a red flag?
>>
>> If anybody has any insight, advice, or recommended literature they can point me
>> to, that would be much appreciated!
>>
>> reproducible example:
>>
>> library(metafor)
>>
>> mean_condition1 <- 0.197
>> se_condition1 <- 0.082
>> mean_condition2 <- -0.350
>> se_condition2 <- 0.105
>> sample_size <- 15
>> assumed_r <- 0.7
>> F_value <- 5.78
>> t_value <- sqrt(F_value)
>>
>> #via means/SD
>> escalc("SMCC", m1i = mean_condition1, sd1i = se_condition1 * sqrt(sample_size),
>>        m2i = mean_condition2, sd2i = se_condition2 * sqrt(sample_size),
>>        ni = sample_size,
>>        ri = assumed_r)
>>
>> #via t
>> d_via_t <- t_value / sqrt(sample_size)
>>
>> (Note that I've used escalc here simply for streamlining the reproducible
>> example. I'm actually using calculations for these derived from Lakens (2013,
>> https://doi.org/10.3389/fpsyg.2013.00863) and Borenstein et al (2009
>> https://doi.org/10.1002/9780470743386.ch4), and estimates are largely similar.
>> Also the example estimates are not the same (since d_via_t is uncorrected) but my
>> question remains since that, of course, doesn't explain the difference).
>>
>> Thanks so much,
>> Brendan
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html