[R-meta] Correcting gain effects in nested studies

[James Pustejovsky]

Ah, sorry I missed that you want SMCC rather than SMCR. Taylor and WWC
focus on SMCR. I'm not aware of anything developed specifically for SMCC.
Apart from the regular critiques of SMCC, there's also the difficulty of
determining an appropriate ICC for the change scores. Just speculating, but
I would guess that ICCs for change scores might be quite different---and
probably smaller?---than ICCs for outcomes measured cross-sectionally (for
which there are many sources of empirical data available).

James

On Wed, May 8, 2024 at 8:58 AM Zhouhan Jin <zjin65 using uwo.ca> wrote:

> Thank you, James! I might be missing something, but I couldn't
> specifically locate the formulas to adjust SMCC (and its SE) in
> ?metafor::escalc, certainly in Taylor et al.
>
> Regarding WWC, is there a specific equation that allows for
> nestedness-adjusting of SMCC (defined below)?
>
> SMCC = (postM - preM) / SD_of_change_scores
>
> with its SE in metafor by default (vtype='LS') being calculated without a
> small-sample correction as:
>
> SE_SMCC = sqrt( (1/n) + ((SMCC^2) / (2*n)) )
>
> Many thanks!
>
> Best wishes,
>
> Zhouhan
>
>
> On May 7, 2024 at 21:30 -0400, James Pustejovsky <jepusto using gmail.com>,
> wrote:
>
> See Taylor, Pigott, and Williams (2022;
> https://doi.org/10.3102/0013189X211051319) for how to handle
> cluster-randomized trials that involve gain scores or covariate adjustment.
> They provide a shiny app too. The technical details are also described in
> Appendix E of the What Works Clearinghouse handbook (Version 5;
> https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf).
> See pp. 173-174
>
> The methods described in these sources are consistent with the "general
> recipe" for standardized mean difference estimates as described here:
> https://www.jepusto.com/alternative-formulas-for-the-smd/
>
> James
>
> On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca> wrote:
>
>> Hello All,
>>
>> Hedges (2007) provides formulas for adjusting SMD effects (g) and their
>> SEs for when primary studies have a nested design (below).
>>
>> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my
>> nested studies, not SMDs.
>>
>> So, how can I adjust my SMCCs and their SEs for nestedness in the primary
>> studies?
>>
>>
>> adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
>>                   (n_cluster * n_bar - 2)))
>>
>> adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
>>     ( g^2 * (
>>       (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
>>          (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
>>         ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
>>     )  )
>>
>> Thanks a lot!
>>
>> Best wishes,
>>
>> Zhouhan
>>
>>

	[[alternative HTML version deleted]]