[R-meta] Correcting gain effects in nested studies

[Zhouhan Jin]

Thank you, James! I might be missing something, but I couldn't specifically locate the formulas to adjust SMCC (and its SE) in ?metafor::escalc, certainly in Taylor et al.

Regarding WWC, is there a specific equation that allows for nestedness-adjusting of SMCC (defined below)?

SMCC = (postM - preM) / SD_of_change_scores

with its SE in metafor by default (vtype='LS') being calculated without a small-sample correction as:

SE_SMCC = sqrt( (1/n) + ((SMCC^2) / (2*n)) )

Many thanks!

Best wishes,

Zhouhan


On May 7, 2024 at 21:30 -0400, James Pustejovsky <jepusto using gmail.com>, wrote:
See Taylor, Pigott, and Williams (2022; https://doi.org/10.3102/0013189X211051319) for how to handle cluster-randomized trials that involve gain scores or covariate adjustment. They provide a shiny app too. The technical details are also described in Appendix E of the What Works Clearinghouse handbook (Version 5; https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf). See pp. 173-174

The methods described in these sources are consistent with the "general recipe" for standardized mean difference estimates as described here:
https://www.jepusto.com/alternative-formulas-for-the-smd/

James

On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca<mailto:zjin65 using uwo.ca>> wrote:
Hello All,

Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).

But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.

So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?


adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
                  (n_cluster * n_bar - 2)))

adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
    ( g^2 * (
      (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
         (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
        ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
    )  )

Thanks a lot!

Best wishes,

Zhouhan


	[[alternative HTML version deleted]]