[R-meta] Meta-analyzing studies that failed to account for their nested data

Fri Oct 1 22:47:45 CEST 2021

No it isn't. I mainly have it from James and his blog (see link below), and
then I added the correction from WWC.
https://www.jepusto.com/alternative-formulas-for-the-smd/

James, please let me/us know if this approach is viable. :)

Mikkel

Den fre. 1. okt. 2021 kl. 22.27 skrev Timothy MacKenzie <fswfswt using gmail.com>:

> Mikkel,
>
> Thank you. Is this new formula (with F-value) for the sampling variance of
> g in the WWC procedures handbook? Would you please provide a reference?
>
> Best regards,
> Tim M
>
> On Fri, Oct 1, 2021 at 3:22 PM Mikkel Vembye <mikkel.vembye using gmail.com>
> wrote:
>
>> When I have relevant F-values from ANCOVA and related models, I calculate
>> vg_corrected = omega^2 * [(g_corrected^2/F-value) * eta +
>> g_corrected^2/(2*h)]
>>
>> Have a nice weekend.
>>
>> Best,
>> Mikkel
>>
>> Den fre. 1. okt. 2021 kl. 21.41 skrev Mikkel Vembye <
>> mikkel.vembye using gmail.com>:
>>
>>> Sorry. I referred to an older version of the Appendix. I usually just
>>> follow WWC's recommendation when I cannot obtain R2. This is
>>>
>>> "if R2 is not available, then WWC will take a cautious approach to
>>> calculating the
>>> standard error and assume a value of zero for R2. This cautious
>>> approach will overestimate the
>>> magnitude of the standard error but protects against type I error." (See
>>> the WWC Procedure Handbook, p. E-5)
>>>
>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf
>>>
>>>
>>> Whether this approach is better, is more a James question.
>>>
>>> Mikkel
>>>
>>> Den fre. 1. okt. 2021 kl. 19.44 skrev Timothy MacKenzie <
>>> fswfswt using gmail.com>:
>>>
>>>> Much appreciated, Mikkel. I saw that. BTW, there is no Table 5, it's a
>>>> typo in the WWC document (I found other typos as well).
>>>>
>>>> But I have both ANOVAs and a few ANCOVAs from primary studies that did
>>>> cluster assignment but ignored nesting structure, with barely any R^2
>>>> reported in them.
>>>>
>>>> My understanding is that I should find a more general SE[g] that only
>>>> requires icc, am I correct in thinking this way?
>>>>
>>>> Thanks,
>>>> Tim M
>>>>
>>>> On Fri, Oct 1, 2021 at 12:32 PM Mikkel Vembye <mikkel.vembye using gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Tim,
>>>>>
>>>>> Glad that I/we can help. You find the ANCOVA examples (both
>>>>> uncorrected and corrected) in Table 3.
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> I forgot to mention that you also can find some corrections to Hedges
>>>>> (2007) in Table 5.
>>>>>
>>>>> All the best,
>>>>> Mikkel
>>>>>
>>>>> Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <
>>>>> fswfswt using gmail.com>:
>>>>>
>>>>>> Dear James and Mikkel,
>>>>>>
>>>>>> Thank you both. In my case, the primary studies have used AN(C)OVAs
>>>>>> with (non-)random assignment of classes to conditions. I have the
>>>>>> Means and SDs of student-level data for conditions.
>>>>>>
>>>>>> Based on:
>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>> , I should use the unbiased version of equation E.5.1 to compute the
>>>>>> SMD effect size (g):
>>>>>>
>>>>>> g = wb / s * sqrt(  1 - (2*(n-1)*icc / N - 2)  ) ; s = pooled standard
>>>>>> deviation; n = ave. cluster size; N = n_t + n_c;  w = hedges'
>>>>>> correction factor
>>>>>>
>>>>>> Based on the same document, the standard error of g (SE[g]) for
>>>>>> cluster-assignment studies is (equation E.7.1 under "Cluster
>>>>>> assignment"):
>>>>>>
>>>>>> SE[g] = w * sqrt(  (SE_uc / s )^2 * eta + (g^2 / (2*h))  ); eta =  1 +
>>>>>> (n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
>>>>>> 2*(N-2*n)*icc*(1-icc)  )
>>>>>>
>>>>>> where SE_uc = regression coefficient standard errors uncorrected for
>>>>>> clustering in the primary studies.
>>>>>>
>>>>>> Am I pointing to the correct formulas? If yes, I don't have SE_uc in
>>>>>> my primary studies, what should I do?
>>>>>>
>>>>>> Thanks,
>>>>>> Tim M
>>>>>>
>>>>>>
>>>>>> ------ Forwarded Message ------
>>>>>> From: Mikkel Vembye <mikkel.vembye using gmail.com>
>>>>>> Date: Fri, Oct 1, 2021 at 6:54 AM
>>>>>> Subject: Re-re: [R-meta] Meta-analyzing studies that failed to account
>>>>>> for their nested data
>>>>>> To: <fswfswt using gmail.com>, <r-sig-meta-analysis using r-project.org>
>>>>>>
>>>>>>
>>>>>> Hi Tim,
>>>>>>
>>>>>> Just to follow up on James, WWC do also have a nice description of how
>>>>>> they handle cluster trials and quasi-experiments:
>>>>>>
>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>>
>>>>>> Mikkel
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <jepusto using gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > For studies that claim to find negligible ICCs, I would guess that
>>>>>> they base this judgement either on a) failing to reject a test of ICC = 0
>>>>>> or b) a rule of thumb. a) is not a good justification because with few
>>>>>> classes, the test will have little power. b) is arbitrary and even small
>>>>>> ICCs (of say 0.02 or 0.04) can be consequential for estimating the variance
>>>>>> of the effect size estimate. I would use the ICC adjustment regardless.
>>>>>> >
>>>>>> > To your second question, yes these adjustments are also important
>>>>>> for quasi-experiments.
>>>>>> >
>>>>>> > James
>>>>>> >
>>>>>> > On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <
>>>>>> fswfswt using gmail.com> wrote:
>>>>>> >>
>>>>>> >> Dear James,
>>>>>> >>
>>>>>> >> Many thanks for this information. Certainly this is serious.
>>>>>> >>
>>>>>> >> I should add that a few of the (newer) studies in my pool say that
>>>>>> >> they found their ICCs to be negligible and opted for the
>>>>>> single-level
>>>>>> >> analyses (maybe I should not adjust the sampling variances in these
>>>>>> >> cases, correct?).
>>>>>> >>
>>>>>> >> Also, I'm assuming that I can use these sampling variance
>>>>>> adjustments
>>>>>> >> for quasi-experiments where schools/centers themselves haven't been
>>>>>> >> randomly recruited as well?
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> Tim M
>>>>>> >>
>>>>>> >> On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <
>>>>>> jepusto using gmail.com> wrote:
>>>>>> >> >
>>>>>> >> > Hi Tim,
>>>>>> >> >
>>>>>> >> > One important issue here is that the sampling variance of the
>>>>>> effect size estimate calculated from such a study will be
>>>>>> inaccurate---possibly even an order of magnitude smaller than it should be.
>>>>>> If you ignore this, the consequence will be to make the effect size
>>>>>> estimates appear far more precise than they actually are.
>>>>>> >> >
>>>>>> >> > To properly correct the sampling variance estimate, you would
>>>>>> need to know the intra-class correlation describing the proportion of the
>>>>>> total variation in the outcome that is at the cluster level (in this case,
>>>>>> what fraction of the total variance is between classes?). If this isn't
>>>>>> reported, then it may be possible to develop a reasonable estimate based on
>>>>>> external information. The Cochrane Handbook describes how to correct the
>>>>>> sampling variance based on an imputed intra-class correlation:
>>>>>> >> >
>>>>>> https://training.cochrane.org/handbook/current/chapter-23#section-23-1
>>>>>> >> > Hedges (2007; https://doi.org/10.3102/1076998606298043) and
>>>>>> (2011; https://doi.org/10.3102/1076998610376617) provides slightly
>>>>>> more elaborate methods that can be used if you have more details about the
>>>>>> study designs. Hedges and Hedberg's Variance Almanac (
>>>>>> http://stateva.ci.northwestern.edu/) is a helpful source for
>>>>>> developing estimates of ICCs for educational outcomes.
>>>>>> >> >
>>>>>> >> > James
>>>>>> >> >
>>>>>> >> > On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <
>>>>>> fswfswt using gmail.com> wrote:
>>>>>> >> >>
>>>>>> >> >> Hello All,
>>>>>> >> >>
>>>>>> >> >> I've noticed almost all the studies I have selected for
>>>>>> meta-analysis
>>>>>> >> >> have ignored the nested structure of their data (subjects
>>>>>> nested in
>>>>>> >> >> classrooms) and have conducted only single-level analyses.
>>>>>> >> >>
>>>>>> >> >> I've extracted the condition-level summaries from those studies
>>>>>> (i.e.,
>>>>>> >> >> Means and SDs for C vs. T groups).
>>>>>> >> >>
>>>>>> >> >> But I'm wondering if I can/should make any adjustment to my
>>>>>> >> >> meta-regression model to account for the nested structure of
>>>>>> the data
>>>>>> >> >> in those studies AND if not, whether such a situation poses a
>>>>>> >> >> limitation to my meta-analysis?
>>>>>> >> >>
>>>>>> >> >> Thank you very much for your assistance,
>>>>>> >> >> Tim M
>>>>>> >> >>
>>>>>> >> >> _______________________________________________
>>>>>> >> >> R-sig-meta-analysis mailing list
>>>>>> >> >> R-sig-meta-analysis using r-project.org
>>>>>> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>>
>>>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/2ce936ec/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19281 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/2ce936ec/attachment-0001.png>