[R-meta] Correcting Hedges' g vs. Log response ratio in nested studies

[Yuhang Hu]

Hi James,

If you look at Eq. number E.5.1 on p1 of this document: (
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf)
they define the correction factor as: sqrt( 1-((2*(n-1)*icc)/(N-2)) )
where N is n1 + n2 (total sample size), and n as s the average number of
individuals per cluster.

Am I missing something? Or is the correction factor linked above from WWC
inaccurate?

Thank you,
Yuhang

On Thu, Nov 2, 2023 at 1:51 PM James Pustejovsky <jepusto using gmail.com> wrote:

> Responses inline below.
>
> On Thu, Nov 2, 2023 at 3:30 PM Yuhang Hu <yh342 using nau.edu> wrote:
>
>> Regarding your first message, it looks like the correction factor for SMD
>> is: sqrt( 1-((2*(n-1)*icc)/(N-2)) ) where n is the average cluster size for
>> each comparison in a study, and N is the sum of the two groups' sample
>> sizes. So, I wonder how the number of clusters is impacting the correction
>> factor for SMD as you indicated?
>>
>> N = n * m, where m is the number of clusters. So the correction factor is
> sqrt( 1-((2*(n-1)*icc)/(m * n - 2)) ~=  sqrt( 1- 2 * icc /m)
>
>
>> Regarding my initial question, my hunch was that for SMD, the SMD
>> estimate and its sampling variance are (non-linearly) related to one
>> another. Therefore, correcting the sampling variance for a design issue
>> will necessitate correcting the SDM estimate as well.
>>
>> On the other hand, the LRR estimate and its sampling variance are not as
>> much related to one another. Therefore, correcting the sampling variance
>> for a design issue will not necessitate correcting the LRR estimate as well.
>>
>>
> No, the issue you've described here is pretty much unrelated to the bias
> correction problem.
>
>
>> On Thu, Nov 2, 2023 at 8:41 AM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>> One other thought on this question, for the extra-nerdy.
>>>
>>> The formulas for the Hedges' g SMD estimator involve what statisticians
>>> would call "second-order" bias corrections, meaning corrections arising
>>> from having a limited sample size. In contrast, the usual estimator of the
>>> LRR is just a "plug-in" estimator that works for large sample sizes but can
>>> have small biases with limited sample sizes. Lajeunesse (2015;
>>> https://doi.org/10.1890/14-2402.1) provides formulas for the
>>> second-order bias correction of the LRR estimator with independent samples.
>>> These bias correction formulas actually *would* need to be different if you
>>> have clustered observations. So, the two effect size metrics are maybe more
>>> similar than it initially seemed:
>>> - Both metrics have plug-in estimators that are not really affected by
>>> the dependence structure of the sample, but whose variance estimators do
>>> need to take into account the dependence structure
>>> - Both metrics have second-order corrected estimators, the exact form
>>> for which does need to take into account the dependence structure.
>>>
>>> James
>>>
>>> On Thu, Nov 2, 2023 at 8:14 AM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>>
>>>>
>>>> Wolfgang is correct. The WWC correction factor arises because the
>>>> sample variance is not quite unbiased as an estimator for the total
>>>> population variance in a design with clusters of dependent observations,
>>>> which leads to a small bias in the SMD.
>>>>
>>>> The thing is, though, this correction factor is usually negligible. Say
>>>> you’ve got a clustered design with n = 21 kids per cluster and 20 clusters,
>>>> and an ICC of 0.2. Then the correction factor is going to be about 0.99 and
>>>> so will make very little difference for the effect size estimate. It only
>>>> starts to matter if you’re looking at studies with very few clusters and
>>>> non-trivial ICCs.
>>>>
>>>> James
>>>>
>>>> > On Nov 2, 2023, at 3:04 AM, Viechtbauer, Wolfgang (NP) via
>>>> R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>>> > Dear Yuhang,
>>>> >
>>>> > I haven't looked deeply into this, but an immediate thought I have is
>>>> that for SMDs, you divide by some measure of variability within the groups.
>>>> If that measure of variability is affected by your study design, then this
>>>> will also affect the SMD value. On the other hand, this doesn't have any
>>>> impact on LRRs since they are only the (log-transformed) ratio of the means.
>>>> >
>>>> > Best,
>>>> > Wolfgang
>>>> >
>>>> >> -----Original Message-----
>>>> >> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org>
>>>> On Behalf
>>>> >> Of Yuhang Hu via R-sig-meta-analysis
>>>> >> Sent: Thursday, November 2, 2023 05:42
>>>> >> To: R meta <r-sig-meta-analysis using r-project.org>
>>>> >> Cc: Yuhang Hu <yh342 using nau.edu>
>>>> >> Subject: [R-meta] Correcting Hedges' g vs. Log response ratio in
>>>> nested studies
>>>> >>
>>>> >> Hello All,
>>>> >>
>>>> >> I know that when correcting Hedges' g (i.e., bias-corrected SMD, aka
>>>> "g")
>>>> >> in nested studies, we have to **BOTH** adjust our initial "g" and its
>>>> >> sampling variance "vi_g"
>>>> >> (
>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-
>>>> >> 508_09212020.pdf).
>>>> >>
>>>> >> But when correcting Log Response Ratios (LRR) in nested studies, we
>>>> have to
>>>> >> **ONLY** adjust its initial sampling variance "vi_LRR" but not "LRR"
>>>> itself
>>>> >> (
>>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html
>>>> ).
>>>> >>
>>>> >> I wonder why the two methods of correction differ for Hedge's g and
>>>> LRR?
>>>> >>
>>>> >> Thanks,
>>>> >> Yuhang
>>>> >
>>>> > _______________________________________________
>>>> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>>> > To manage your subscription to this mailing list, go to:
>>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>
>>>

	[[alternative HTML version deleted]]