<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">This correction applies to a single effect size estimate from a given study. All of these values (n, N, n1, n2) are therefore specific to the study and need to be recorded for each row in a meta-analysis database.</div><div dir="ltr"><br><blockquote type="cite">On Nov 2, 2023, at 5:14 PM, Yuhang Hu <yh342@nau.edu> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline">Sure, I thought <span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">N is n1 + n2 which is unique to each row in the dataset. </span></div><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline"><span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit"><br></span></div><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline"><span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">But it looks like I should compute N as</span><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit"> n * m where "n" is (average of n1, n2) for each row in the data but "m" is constant across all rows in the dataset.</span></div><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline"><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit"><br></span></div><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline"><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">Thanks,</span></div><div style="color:inherit;font:inherit;margin:0px;padding:0px;border:0px;vertical-align:baseline"><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">Yuhang</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 2:16 PM James Pustejovsky <<a href="mailto:jepusto@gmail.com">jepusto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Total sample size is the same thing as the average sample size per cluster times the number of clusters. My previous message is just a restatement of the formula to show how it is related to the number of clusters.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 4:11 PM Yuhang Hu <<a href="mailto:yh342@nau.edu" target="_blank">yh342@nau.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr" style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline"><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">Hi James,</div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"><br></div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">If you look at Eq. number E.5.1 on p1 of this document: (<a href="https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf" target="_blank">https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf</a>) they define the correction factor as: <span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">sqrt( 1-((2*(n-1)*icc)/(N-2)) )</span><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit"> </span></div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"><span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">where N is n1 + n2 (total sample size), and n as </span><span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">s the </span><span style="font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">average number of individuals per cluster.</span></div></div><div dir="ltr" style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline"><br></div><div style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline">Am I missing something? Or is the correction factor linked above from WWC inaccurate?</div><div style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline"><br></div><div style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline">Thank you,</div><div style="margin:0px;padding:0px;border:0px;font-stretch:inherit;line-height:inherit;vertical-align:baseline">Yuhang</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 1:51 PM James Pustejovsky <<a href="mailto:jepusto@gmail.com" target="_blank">jepusto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Responses inline below.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 3:30 PM Yuhang Hu <<a href="mailto:yh342@nau.edu" target="_blank">yh342@nau.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><span style="color:inherit;font-family:inherit;font-size:inherit;font-style:inherit;font-variant-ligatures:inherit;font-variant-caps:inherit;font-weight:inherit">Regarding your first message, it looks like the correction factor for SMD is: sqrt( 1-((2*(n-1)*icc)/(N-2)) ) where n is the average cluster size for each comparison in a study, and N is the sum of the two groups' sample sizes. So, I wonder how the number of clusters is impacting the correction factor for SMD as you indicated?</span><br></div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"><br></div></div></blockquote><div>N = n * m, where m is the number of clusters. So the correction factor is</div><div>sqrt( 1-((2*(n-1)*icc)/(m * n - 2)) ~= sqrt( 1- 2 * icc /m)<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"></div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">Regarding my initial question, my hunch was that for SMD, the SMD estimate and its sampling variance are (non-linearly) related to one another. Therefore, correcting the sampling variance for a design issue will necessitate correcting the SDM estimate as well.</div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"><br></div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline">On the other hand, the LRR estimate and its sampling variance are not as much related to one another. Therefore, correcting the sampling variance for a design issue will not necessitate correcting the LRR estimate as well.</div><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"><br></div></div></blockquote><div><br></div><div>No, the issue you've described here is pretty much unrelated to the bias correction problem.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="color:inherit;margin:0px;padding:0px;border:0px;font:inherit;vertical-align:baseline"></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 8:41 AM James Pustejovsky <<a href="mailto:jepusto@gmail.com" target="_blank">jepusto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">One other thought on this question, for the extra-nerdy. <div><br></div><div>The formulas for the Hedges' g SMD estimator involve what statisticians would call "second-order" bias corrections, meaning corrections arising from having a limited sample size. In contrast, the usual estimator of the LRR is just a "plug-in" estimator that works for large sample sizes but can have small biases with limited sample sizes. Lajeunesse (2015; <a href="https://doi.org/10.1890/14-2402.1" target="_blank">https://doi.org/10.1890/14-2402.1</a>) provides formulas for the second-order bias correction of the LRR estimator with independent samples. These bias correction formulas actually *would* need to be different if you have clustered observations. So, the two effect size metrics are maybe more similar than it initially seemed:<div>- Both metrics have plug-in estimators that are not really affected by the dependence structure of the sample, but whose variance estimators do need to take into account the dependence structure</div><div>- Both metrics have second-order corrected estimators, the exact form for which does need to take into account the dependence structure.</div><div><br></div><div>James</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 2, 2023 at 8:14 AM James Pustejovsky <<a href="mailto:jepusto@gmail.com" target="_blank">jepusto@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Wolfgang is correct. The WWC correction factor arises because the sample variance is not quite unbiased as an estimator for the total population variance in a design with clusters of dependent observations, which leads to a small bias in the SMD. <br>
<br>
The thing is, though, this correction factor is usually negligible. Say you’ve got a clustered design with n = 21 kids per cluster and 20 clusters, and an ICC of 0.2. Then the correction factor is going to be about 0.99 and so will make very little difference for the effect size estimate. It only starts to matter if you’re looking at studies with very few clusters and non-trivial ICCs.<br>
<br>
James<br>
<br>
> On Nov 2, 2023, at 3:04 AM, Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis <<a href="mailto:r-sig-meta-analysis@r-project.org" target="_blank">r-sig-meta-analysis@r-project.org</a>> wrote:<br>
> Dear Yuhang,<br>
> <br>
> I haven't looked deeply into this, but an immediate thought I have is that for SMDs, you divide by some measure of variability within the groups. If that measure of variability is affected by your study design, then this will also affect the SMD value. On the other hand, this doesn't have any impact on LRRs since they are only the (log-transformed) ratio of the means.<br>
> <br>
> Best,<br>
> Wolfgang<br>
> <br>
>> -----Original Message-----<br>
>> From: R-sig-meta-analysis <<a href="mailto:r-sig-meta-analysis-bounces@r-project.org" target="_blank">r-sig-meta-analysis-bounces@r-project.org</a>> On Behalf<br>
>> Of Yuhang Hu via R-sig-meta-analysis<br>
>> Sent: Thursday, November 2, 2023 05:42<br>
>> To: R meta <<a href="mailto:r-sig-meta-analysis@r-project.org" target="_blank">r-sig-meta-analysis@r-project.org</a>><br>
>> Cc: Yuhang Hu <<a href="mailto:yh342@nau.edu" target="_blank">yh342@nau.edu</a>><br>
>> Subject: [R-meta] Correcting Hedges' g vs. Log response ratio in nested studies<br>
>> <br>
>> Hello All,<br>
>> <br>
>> I know that when correcting Hedges' g (i.e., bias-corrected SMD, aka "g")<br>
>> in nested studies, we have to **BOTH** adjust our initial "g" and its<br>
>> sampling variance "vi_g"<br>
>> (<a href="https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-" rel="noreferrer" target="_blank">https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-</a><br>
>> 508_09212020.pdf).<br>
>> <br>
>> But when correcting Log Response Ratios (LRR) in nested studies, we have to<br>
>> **ONLY** adjust its initial sampling variance "vi_LRR" but not "LRR" itself<br>
>> (<a href="https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html" rel="noreferrer" target="_blank">https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-October/003486.html</a>).<br>
>> <br>
>> I wonder why the two methods of correction differ for Hedge's g and LRR?<br>
>> <br>
>> Thanks,<br>
>> Yuhang<br>
> <br>
> _______________________________________________<br>
> R-sig-meta-analysis mailing list @ <a href="mailto:R-sig-meta-analysis@r-project.org" target="_blank">R-sig-meta-analysis@r-project.org</a><br>
> To manage your subscription to this mailing list, go to:<br>
> <a href="https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis" rel="noreferrer" target="_blank">https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis</a><br>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</div></blockquote></body></html>