[R-meta] Correcting gain effects in nested studies

[Viechtbauer, Wolfgang (NP)]

Just to answer the question why escalc() needs the pre-post correlation for measure SMCR: It doesn't affect the estimate, but it is relevant for computing the sampling variance. To be precise:

yi = (m1i - m2i) / sd1i
vi = 2*(1-ri)/ni + yi^2 / (2*ni)

(leaving out the bias-correction).

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Zhouhan Jin via R-sig-meta-analysis
> Sent: Thursday, May 9, 2024 05:24
> To: James Pustejovsky <jepusto using gmail.com>
> Cc: Zhouhan Jin <zjin65 using uwo.ca>; R Special Interest Group for Meta-Analysis <r-
> sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Correcting gain effects in nested studies
>
> Hi James,
>
> Why should ICC be smaller for change scores, because subtraction removes some of
> the dependency with clusters, kind of like centering within clusters?
>
> Anyway, knowing now that nothing exists for SMCC, I am actually open to
> standardizing my pre-post mean changes using other denominators including
> standard deviation of the pre-test as in SMCR (sidenote: why escalc() then needs
> "ri=" to calculate an SMCR estimate?).
>
> My data only consists of pre-post Means and their respective Sds.
>
> But, what equation in the WWC handbook specifically gives the formula to adjust
> the SMCRs and its SEs calculated by metafor::escalc() for (one level of)
> nesting?
>
> WWC's notations are often confusing.
>
> Best wishes,
>
> Zhouhan
>
> On May 8, 2024 at 21:59 -0400, James Pustejovsky <jepusto using gmail.com>, wrote:
>
> Ah, sorry I missed that you want SMCC rather than SMCR. Taylor and WWC focus on
> SMCR. I'm not aware of anything developed specifically for SMCC. Apart from the
> regular critiques of SMCC, there's also the difficulty of determining an
> appropriate ICC for the change scores. Just speculating, but I would guess that
> ICCs for change scores might be quite different---and probably smaller?---than
> ICCs for outcomes measured cross-sectionally (for which there are many sources
> of empirical data available).
>
> James
>
> On Wed, May 8, 2024 at 8:58 AM Zhouhan Jin <zjin65 using uwo.ca<mailto:zjin65 using uwo.ca>>
> wrote:
> Thank you, James! I might be missing something, but I couldn't specifically
> locate the formulas to adjust SMCC (and its SE) in ?metafor::escalc, certainly
> in Taylor et al.
>
> Regarding WWC, is there a specific equation that allows for nestedness-adjusting
> of SMCC (defined below)?
>
> SMCC = (postM - preM) / SD_of_change_scores
>
> with its SE in metafor by default (vtype='LS') being calculated without a small-
> sample correction as:
>
> SE_SMCC = sqrt( (1/n) + ((SMCC^2) / (2*n)) )
>
> Many thanks!
>
> Best wishes,
>
> Zhouhan
>
>
> On May 7, 2024 at 21:30 -0400, James Pustejovsky
> <jepusto using gmail.com<mailto:jepusto using gmail.com>>, wrote:
> See Taylor, Pigott, and Williams (2022;
> https://doi.org/10.3102/0013189X211051319) for how to handle cluster-randomized
> trials that involve gain scores or covariate adjustment. They provide a shiny
> app too. The technical details are also described in Appendix E of the What
> Works Clearinghouse handbook (Version 5;
> https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-
> 508.pdf). See pp. 173-174
>
> The methods described in these sources are consistent with the "general recipe"
> for standardized mean difference estimates as described here:
> https://www.jepusto.com/alternative-formulas-for-the-smd/
>
> James
>
> On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca<mailto:zjin65 using uwo.ca>>
> wrote:
> Hello All,
>
> Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for
> when primary studies have a nested design (below).
>
> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested
> studies, not SMDs.
>
> So, how can I adjust my SMCCs and their SEs for nestedness in the primary
> studies?
>
>
> adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
>                   (n_cluster * n_bar - 2)))
>
> adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
>     ( g^2 * (
>       (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
>          (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
>         ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
>     )  )
>
> Thanks a lot!
>
> Best wishes,
>
> Zhouhan