[R-meta] Correcting gain effects in nested studies

[Zhouhan Jin]

Hi James,

Why should ICC be smaller for change scores, because subtraction removes some of the dependency with clusters, kind of like centering within clusters?

Anyway, knowing now that nothing exists for SMCC, I am actually open to standardizing my pre-post mean changes using other denominators including standard deviation of the pre-test as in SMCR (sidenote: why escalc() then needs "ri=" to calculate an SMCR estimate?).

My data only consists of pre-post Means and their respective Sds.

But, what equation in the WWC handbook specifically gives the formula to adjust the SMCRs and its SEs calculated by metafor::escalc() for (one level of) nesting?

WWC's notations are often confusing.

Best wishes,

Zhouhan

On May 8, 2024 at 21:59 -0400, James Pustejovsky <jepusto using gmail.com>, wrote:
Ah, sorry I missed that you want SMCC rather than SMCR. Taylor and WWC focus on SMCR. I'm not aware of anything developed specifically for SMCC. Apart from the regular critiques of SMCC, there's also the difficulty of determining an appropriate ICC for the change scores. Just speculating, but I would guess that ICCs for change scores might be quite different---and probably smaller?---than ICCs for outcomes measured cross-sectionally (for which there are many sources of empirical data available).

James

On Wed, May 8, 2024 at 8:58 AM Zhouhan Jin <zjin65 using uwo.ca<mailto:zjin65 using uwo.ca>> wrote:
Thank you, James! I might be missing something, but I couldn't specifically locate the formulas to adjust SMCC (and its SE) in ?metafor::escalc, certainly in Taylor et al.

Regarding WWC, is there a specific equation that allows for nestedness-adjusting of SMCC (defined below)?

SMCC = (postM - preM) / SD_of_change_scores

with its SE in metafor by default (vtype='LS') being calculated without a small-sample correction as:

SE_SMCC = sqrt( (1/n) + ((SMCC^2) / (2*n)) )

Many thanks!

Best wishes,

Zhouhan


On May 7, 2024 at 21:30 -0400, James Pustejovsky <jepusto using gmail.com<mailto:jepusto using gmail.com>>, wrote:
See Taylor, Pigott, and Williams (2022; https://doi.org/10.3102/0013189X211051319) for how to handle cluster-randomized trials that involve gain scores or covariate adjustment. They provide a shiny app too. The technical details are also described in Appendix E of the What Works Clearinghouse handbook (Version 5; https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf). See pp. 173-174

The methods described in these sources are consistent with the "general recipe" for standardized mean difference estimates as described here:
https://www.jepusto.com/alternative-formulas-for-the-smd/

James

On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca<mailto:zjin65 using uwo.ca>> wrote:
Hello All,

Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).

But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.

So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?


adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
                  (n_cluster * n_bar - 2)))

adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
    ( g^2 * (
      (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
         (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
        ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
    )  )

Thanks a lot!

Best wishes,

Zhouhan


	[[alternative HTML version deleted]]