[R-meta] Question about Cohen's d and meta-regression

[Viechtbauer Wolfgang (SP)]

Hi Angeline,

I don't quite understand what your concern is. Neither approach (of computing a standardized mean change) is right or wrong -- they are just different ways of standardizing the mean change. Also, a large treatment effect does not automatically translate into a small correlation. If the effect is such that the mean goes up (or down) by a large constant for everybody, then the correlation will be 1. It is only when there is large variability in the size of the treatment effect that the correlation will be small. But again, I am not able to follow what you are concerned about, so I cannot really answer your question.

Best,
Wolfgang

-----Original Message-----
From: Angeline Tsui [mailto:angelinetsui at gmail.com] 
Sent: Monday, 08 January, 2018 0:15
To: Viechtbauer Wolfgang (SP)
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Question about Cohen's d and meta-regression

Hello Wolfgang,

Thank you for your reply. Just a quick follow up question: I think that we need to take into account of the correlation between pre-test and post-test when we need to calculate the SD of difference score. I see this may be important for my study because if there is a large treatment effect at post-test, the correlation between pre-test and post-test would likely to be small. In this case, it looks like that I should not use pre-test SD for the SD of difference score. In this case, what is your recommendation? Should I use post-test SD instead if SD of difference score is not provided?

Thanks,
Angeline

On Sun, Jan 7, 2018 at 5:24 PM, Viechtbauer Wolfgang (SP) <wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
A "within-subject Cohen's d" can be computed in two ways:

1) by dividing the mean difference by the SD of the change scores (standardized mean change using change score standardization) or

2) by dividing the mean difference by the SD of the raw scores (standardized mean change using raw score standardization). Here, the pre-test SD is often used (although one could also use the post-test SD or pool pre- and post-test SDs, although for the latter one needs to do some extra work to get a correct estimate of the sampling variance).

The second approach yields a d value that is conceptually the same as a "between-subject Cohen's d" (if we 'equate' the pre condition with the control condition and the post condition with the experimental/treatment condition and assume that there are no time effects). So, I see no need for changing a within-subject Cohen's d to between-subject Cohen's d (leaving aside how that would actually be done).

Best,
Wolfgang

-----Original Message-----
From: Angeline Tsui [mailto:angelinetsui at gmail.com]
Sent: Thursday, 04 January, 2018 13:11
To: Philipp Doebler; Viechtbauer Wolfgang (SP)
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Question about Cohen's d and meta-regression

Dear Wolfgang and Philipp,

Thank you so much for the detailed response. This is exactly what I am looking for. I will look into the Knapp & Hartung correction and also do a power analysis of meta-regression. Thanks again for the recommended resources.

Re Wolfgang as for the question about changing within-subject Cohen's d to between-subject Cohen's d: To be honest, I have struggled quite a lot before making this decision. The primary reason is due to recommendations from researchers in my field. They once advocated meta-analysis in my field to change within-subject Cohen's d to between-subject Cohen's d. I think it is simply related to converting meta-analysis mean effect size to a common metric, so that we can directly compare mean effect sizes across meta-analyses. Another reason (I think I may be wrong and please feel free to correct me) is that I was worried if I can still apply the Cohen's rule of thumb (0.2, 0.5, 0.8) to interpret within-subject Cohen's d. I know that this rule of thumb applies to between-subject Cohen's d, but I really am not sure if we can do this for within-subject Cohen's d. Specifically, the standard deviations are different between within-subject and between-subject designs. The standard deviations for within-subject design are theoretically smaller since we are controlling for individual differences. Thus, I changed my metric to within-subject Cohen's d just for the ease of interpretation as well.

Best,
Angeline