[R-meta] Correcting Hedges' g vs. Log response ratio in nested studies

[James Pustejovsky]

Responses inline below.

On Thu, Nov 2, 2023 at 3:30 PM Yuhang Hu <yh342 using nau.edu> wrote:

> Regarding your first message, it looks like the correction factor for SMD
> is: sqrt( 1-((2*(n-1)*icc)/(N-2)) ) where n is the average cluster size for
> each comparison in a study, and N is the sum of the two groups' sample
> sizes. So, I wonder how the number of clusters is impacting the correction
> factor for SMD as you indicated?
>
> N = n * m, where m is the number of clusters. So the correction factor is
sqrt( 1-((2*(n-1)*icc)/(m * n - 2)) ~=  sqrt( 1- 2 * icc /m)


> Regarding my initial question, my hunch was that for SMD, the SMD estimate
> and its sampling variance are (non-linearly) related to one another.
> Therefore, correcting the sampling variance for a design issue will
> necessitate correcting the SDM estimate as well.
>
> On the other hand, the LRR estimate and its sampling variance are not as
> much related to one another. Therefore, correcting the sampling variance
> for a design issue will not necessitate correcting the LRR estimate as well.
>
>
No, the issue you've described here is pretty much unrelated to the bias
correction problem.


> On Thu, Nov 2, 2023 at 8:41 AM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> One other thought on this question, for the extra-nerdy.
>>
>> The formulas for the Hedges' g SMD estimator involve what statisticians
>> would call "second-order" bias corrections, meaning corrections arising
>> from having a limited sample size. In contrast, the usual estimator of the
>> LRR is just a "plug-in" estimator that works for large sample sizes but can
>> have small biases with limited sample sizes. Lajeunesse (2015;
>> https://doi.org/10.1890/14-2402.1) provides formulas for the
>> second-order bias correction of the LRR estimator with independent samples.
>> These bias correction formulas actually *would* need to be different if you
>> have clustered observations. So, the two effect size metrics are maybe more
>> similar than it initially seemed:
>> - Both metrics have plug-in estimators that are not really affected by
>> the dependence structure of the sample, but whose variance estimators do
>> need to take into account the dependence structure
>> - Both metrics have second-order corrected estimators, the exact form for
>> which does need to take into account the dependence structure.
>>
>> James
>>
>> On Thu, Nov 2, 2023 at 8:14 AM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>>
>>> Wolfgang is correct. The WWC correction factor arises because the sample
>>> variance is not quite unbiased as an estimator for the total population
>>> variance in a design with clusters of dependent observations, which leads
>>> to a small bias in the SMD.
>>>
>>> The thing is, though, this correction factor is usually negligible. Say
>>> you’ve got a clustered design with n = 21 kids per cluster and 20 clusters,
>>> and an ICC of 0.2. Then the correction factor is going to be about 0.99 and
>>> so will make very little difference for the effect size estimate. It only
>>> starts to matter if you’re looking at studies with very few clusters and
>>> non-trivial ICCs.
>>>
>>> James
>>>
>>> > On Nov 2, 2023, at 3:04 AM, Viechtbauer, Wolfgang (NP) via
>>> R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:
>>> > Dear Yuhang,
>>> >
>>> > I haven't looked deeply into this, but an immediate thought I have is
>>> that for SMDs, you divide by some measure of variability within the groups.
>>> If that measure of variability is affected by your study design, then this
>>> will also affect the SMD value. On the other hand, this doesn't have any
>>> impact on LRRs since they are only the (log-transformed) ratio of the means.
>>> >
>>> > Best,
>>> > Wolfgang
>>> >
>>> >> -----Original Message-----
>>> >> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org>
>>> On Behalf
>>> >> Of Yuhang Hu via R-sig-meta-analysis
>>> >> Sent: Thursday, November 2, 2023 05:42
>>> >> To: R meta <r-sig-meta-analysis using r-project.org>
>>> >> Cc: Yuhang Hu <yh342 using nau.edu>
>>> >> Subject: [R-meta] Correcting Hedges' g vs. Log response ratio in
>>> nested studies
>>> >>
>>> >> Hello All,
>>> >>
>>> >> I know that when correcting Hedges' g (i.e., bias-corrected SMD, aka
>>> "g")
>>> >> in nested studies, we have to **BOTH** adjust our initial "g" and its
>>> >> sampling variance "vi_g"
>>> >> (
>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-
>>> >> 508_09212020.pdf).
>>> >>
>>> >> But when correcting Log Response Ratios (LRR) in nested studies, we
>>> have to
>>> >> **ONLY** adjust its initial sampling variance "vi_LRR" but not "LRR"
>>> itself
>>> >> (
>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html
>>> ).
>>> >>
>>> >> I wonder why the two methods of correction differ for Hedge's g and
>>> LRR?
>>> >>
>>> >> Thanks,
>>> >> Yuhang
>>> >
>>> > _______________________________________________
>>> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> > To manage your subscription to this mailing list, go to:
>>> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>

	[[alternative HTML version deleted]]