[R-meta] Correcting Hedges' g vs. Log response ratio in nested studies

[James Pustejovsky]

Total sample size is the same thing as the average sample size per cluster
times the number of clusters. My previous message is just a restatement of
the formula to show how it is related to the number of clusters.

On Thu, Nov 2, 2023 at 4:11 PM Yuhang Hu <yh342 using nau.edu> wrote:

> Hi James,
>
> If you look at Eq. number E.5.1 on p1 of this document: (
> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf)
> they define the correction factor as: sqrt( 1-((2*(n-1)*icc)/(N-2)) )
> where N is n1 + n2 (total sample size), and n as s the average number of
> individuals per cluster.
>
> Am I missing something? Or is the correction factor linked above from WWC
> inaccurate?
>
> Thank you,
> Yuhang
>
> On Thu, Nov 2, 2023 at 1:51 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Responses inline below.
>>
>> On Thu, Nov 2, 2023 at 3:30 PM Yuhang Hu <yh342 using nau.edu> wrote:
>>
>>> Regarding your first message, it looks like the correction factor for
>>> SMD is: sqrt( 1-((2*(n-1)*icc)/(N-2)) ) where n is the average cluster size
>>> for each comparison in a study, and N is the sum of the two groups' sample
>>> sizes. So, I wonder how the number of clusters is impacting the correction
>>> factor for SMD as you indicated?
>>>
>>> N = n * m, where m is the number of clusters. So the correction factor is
>> sqrt( 1-((2*(n-1)*icc)/(m * n - 2)) ~=  sqrt( 1- 2 * icc /m)
>>
>>
>>> Regarding my initial question, my hunch was that for SMD, the SMD
>>> estimate and its sampling variance are (non-linearly) related to one
>>> another. Therefore, correcting the sampling variance for a design issue
>>> will necessitate correcting the SDM estimate as well.
>>>
>>> On the other hand, the LRR estimate and its sampling variance are not as
>>> much related to one another. Therefore, correcting the sampling variance
>>> for a design issue will not necessitate correcting the LRR estimate as well.
>>>
>>>
>> No, the issue you've described here is pretty much unrelated to the bias
>> correction problem.
>>
>>
>>> On Thu, Nov 2, 2023 at 8:41 AM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>>
>>>> One other thought on this question, for the extra-nerdy.
>>>>
>>>> The formulas for the Hedges' g SMD estimator involve what statisticians
>>>> would call "second-order" bias corrections, meaning corrections arising
>>>> from having a limited sample size. In contrast, the usual estimator of the
>>>> LRR is just a "plug-in" estimator that works for large sample sizes but can
>>>> have small biases with limited sample sizes. Lajeunesse (2015;
>>>> https://doi.org/10.1890/14-2402.1) provides formulas for the
>>>> second-order bias correction of the LRR estimator with independent samples.
>>>> These bias correction formulas actually *would* need to be different if you
>>>> have clustered observations. So, the two effect size metrics are maybe more
>>>> similar than it initially seemed:
>>>> - Both metrics have plug-in estimators that are not really affected by
>>>> the dependence structure of the sample, but whose variance estimators do
>>>> need to take into account the dependence structure
>>>> - Both metrics have second-order corrected estimators, the exact form
>>>> for which does need to take into account the dependence structure.
>>>>
>>>> James
>>>>
>>>> On Thu, Nov 2, 2023 at 8:14 AM James Pustejovsky <jepusto using gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Wolfgang is correct. The WWC correction factor arises because the
>>>>> sample variance is not quite unbiased as an estimator for the total
>>>>> population variance in a design with clusters of dependent observations,
>>>>> which leads to a small bias in the SMD.
>>>>>
>>>>> The thing is, though, this correction factor is usually negligible.
>>>>> Say you’ve got a clustered design with n = 21 kids per cluster and 20
>>>>> clusters, and an ICC of 0.2. Then the correction factor is going to be
>>>>> about 0.99 and so will make very little difference for the effect size
>>>>> estimate. It only starts to matter if you’re looking at studies with very
>>>>> few clusters and non-trivial ICCs.
>>>>>
>>>>> James
>>>>>
>>>>> > On Nov 2, 2023, at 3:04 AM, Viechtbauer, Wolfgang (NP) via
>>>>> R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>>>> > Dear Yuhang,
>>>>> >
>>>>> > I haven't looked deeply into this, but an immediate thought I have
>>>>> is that for SMDs, you divide by some measure of variability within the
>>>>> groups. If that measure of variability is affected by your study design,
>>>>> then this will also affect the SMD value. On the other hand, this doesn't
>>>>> have any impact on LRRs since they are only the (log-transformed) ratio of
>>>>> the means.
>>>>> >
>>>>> > Best,
>>>>> > Wolfgang
>>>>> >
>>>>> >> -----Original Message-----
>>>>> >> From: R-sig-meta-analysis <
>>>>> r-sig-meta-analysis-bounces using r-project.org> On Behalf
>>>>> >> Of Yuhang Hu via R-sig-meta-analysis
>>>>> >> Sent: Thursday, November 2, 2023 05:42
>>>>> >> To: R meta <r-sig-meta-analysis using r-project.org>
>>>>> >> Cc: Yuhang Hu <yh342 using nau.edu>
>>>>> >> Subject: [R-meta] Correcting Hedges' g vs. Log response ratio in
>>>>> nested studies
>>>>> >>
>>>>> >> Hello All,
>>>>> >>
>>>>> >> I know that when correcting Hedges' g (i.e., bias-corrected SMD,
>>>>> aka "g")
>>>>> >> in nested studies, we have to **BOTH** adjust our initial "g" and
>>>>> its
>>>>> >> sampling variance "vi_g"
>>>>> >> (
>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-
>>>>> >> 508_09212020.pdf).
>>>>> >>
>>>>> >> But when correcting Log Response Ratios (LRR) in nested studies, we
>>>>> have to
>>>>> >> **ONLY** adjust its initial sampling variance "vi_LRR" but not
>>>>> "LRR" itself
>>>>> >> (
>>>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html
>>>>> ).
>>>>> >>
>>>>> >> I wonder why the two methods of correction differ for Hedge's g and
>>>>> LRR?
>>>>> >>
>>>>> >> Thanks,
>>>>> >> Yuhang
>>>>> >
>>>>> > _______________________________________________
>>>>> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>>>> > To manage your subscription to this mailing list, go to:
>>>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>

	[[alternative HTML version deleted]]