[R-meta] Meta-analyzing studies that failed to account for their nested data

Fri Oct 8 01:17:07 CEST 2021

Hi Tim,

My initial response was addressing the context you've described. Yes, the
main concern here is adjusting the SEs (using WWC or Hedges approach) to
account for the increased uncertainty from clustered assignment.

James

On Tue, Oct 5, 2021 at 5:20 PM Timothy MacKenzie <fswfswt using gmail.com> wrote:

> To provide an example, one study has randomly assigned 3 classrooms to 3
> conditions (T1, T2, C).
>
> (T1 <-- class 1, n = 15); (T2 <-- class 2, n = 13); (C <-- class 3, n =
> 15)
>
> So for this study, I have 3 "means" and 3 "sds" for T1, T2, and T3. So,
> should I adjust the SEs of the resulting SMDs using WWC's or Hedges' (
> https://doi.org/10.3102/1076998606298043) formulas here?
>
> Thanks, again
> Tim M
>
>
>
>
>
> On Tue, Oct 5, 2021 at 4:51 PM Timothy MacKenzie <fswfswt using gmail.com>
> wrote:
>
>> Dear All,
>>
>> I just read a paper (
>> https://link.springer.com/content/pdf/10.3758/s13428-011-0153-1.pdf)
>> that made me think whether I understood your advice correctly or not.
>>
>> My question was what to do with studies that provide "means" and "sds"
>> for "student-level" data while they ignore their students being nested in
>> classrooms.
>>
>> The article linked above says that only if we calculate an SMD based on
>> "cluster-level" data (d_cluster), then d_cluster is upwardly biased.
>>
>> But I want to calculate an SMD for studies only based on "student-level"
>> data (d_individual), so do I need to adjust "d_individual" just like
>> "d_cluster"?
>>
>> Thank you for clarifying this confusion for me,
>> Tim M
>>
>>
>> On Fri, Oct 1, 2021 at 3:47 PM Mikkel Vembye <mikkel.vembye using gmail.com>
>> wrote:
>>
>>> No it isn't. I mainly have it from James and his blog (see link below),
>>> and then I added the correction from WWC.
>>> https://www.jepusto.com/alternative-formulas-for-the-smd/
>>>
>>> James, please let me/us know if this approach is viable. :)
>>>
>>> Mikkel
>>>
>>> Den fre. 1. okt. 2021 kl. 22.27 skrev Timothy MacKenzie <
>>> fswfswt using gmail.com>:
>>>
>>>> Mikkel,
>>>>
>>>> Thank you. Is this new formula (with F-value) for the sampling variance
>>>> of g in the WWC procedures handbook? Would you please provide a reference?
>>>>
>>>> Best regards,
>>>> Tim M
>>>>
>>>> On Fri, Oct 1, 2021 at 3:22 PM Mikkel Vembye <mikkel.vembye using gmail.com>
>>>> wrote:
>>>>
>>>>> When I have relevant F-values from ANCOVA and related models, I
>>>>> calculate vg_corrected = omega^2 * [(g_corrected^2/F-value) * eta +
>>>>> g_corrected^2/(2*h)]
>>>>>
>>>>> Have a nice weekend.
>>>>>
>>>>> Best,
>>>>> Mikkel
>>>>>
>>>>> Den fre. 1. okt. 2021 kl. 21.41 skrev Mikkel Vembye <
>>>>> mikkel.vembye using gmail.com>:
>>>>>
>>>>>> Sorry. I referred to an older version of the Appendix. I usually just
>>>>>> follow WWC's recommendation when I cannot obtain R2. This is
>>>>>>
>>>>>> "if R2 is not available, then WWC will take a cautious approach to
>>>>>> calculating the
>>>>>> standard error and assume a value of zero for R2. This cautious
>>>>>> approach will overestimate the
>>>>>> magnitude of the standard error but protects against type I error."
>>>>>> (See the WWC Procedure Handbook, p. E-5)
>>>>>>
>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf
>>>>>>
>>>>>>
>>>>>> Whether this approach is better, is more a James question.
>>>>>>
>>>>>> Mikkel
>>>>>>
>>>>>> Den fre. 1. okt. 2021 kl. 19.44 skrev Timothy MacKenzie <
>>>>>> fswfswt using gmail.com>:
>>>>>>
>>>>>>> Much appreciated, Mikkel. I saw that. BTW, there is no Table 5, it's
>>>>>>> a typo in the WWC document (I found other typos as well).
>>>>>>>
>>>>>>> But I have both ANOVAs and a few ANCOVAs from primary studies that
>>>>>>> did cluster assignment but ignored nesting structure, with barely any R^2
>>>>>>> reported in them.
>>>>>>>
>>>>>>> My understanding is that I should find a more general SE[g] that
>>>>>>> only requires icc, am I correct in thinking this way?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tim M
>>>>>>>
>>>>>>> On Fri, Oct 1, 2021 at 12:32 PM Mikkel Vembye <
>>>>>>> mikkel.vembye using gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Tim,
>>>>>>>>
>>>>>>>> Glad that I/we can help. You find the ANCOVA examples (both
>>>>>>>> uncorrected and corrected) in Table 3.
>>>>>>>>
>>>>>>>> [image: image.png]
>>>>>>>>
>>>>>>>> I forgot to mention that you also can find some corrections to
>>>>>>>> Hedges (2007) in Table 5.
>>>>>>>>
>>>>>>>> All the best,
>>>>>>>> Mikkel
>>>>>>>>
>>>>>>>> Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <
>>>>>>>> fswfswt using gmail.com>:
>>>>>>>>
>>>>>>>>> Dear James and Mikkel,
>>>>>>>>>
>>>>>>>>> Thank you both. In my case, the primary studies have used AN(C)OVAs
>>>>>>>>> with (non-)random assignment of classes to conditions. I have the
>>>>>>>>> Means and SDs of student-level data for conditions.
>>>>>>>>>
>>>>>>>>> Based on:
>>>>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>>>>> , I should use the unbiased version of equation E.5.1 to compute
>>>>>>>>> the
>>>>>>>>> SMD effect size (g):
>>>>>>>>>
>>>>>>>>> g = wb / s * sqrt(  1 - (2*(n-1)*icc / N - 2)  ) ; s = pooled
>>>>>>>>> standard
>>>>>>>>> deviation; n = ave. cluster size; N = n_t + n_c;  w = hedges'
>>>>>>>>> correction factor
>>>>>>>>>
>>>>>>>>> Based on the same document, the standard error of g (SE[g]) for
>>>>>>>>> cluster-assignment studies is (equation E.7.1 under "Cluster
>>>>>>>>> assignment"):
>>>>>>>>>
>>>>>>>>> SE[g] = w * sqrt(  (SE_uc / s )^2 * eta + (g^2 / (2*h))  ); eta =
>>>>>>>>> 1 +
>>>>>>>>> (n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
>>>>>>>>> 2*(N-2*n)*icc*(1-icc)  )
>>>>>>>>>
>>>>>>>>> where SE_uc = regression coefficient standard errors uncorrected
>>>>>>>>> for
>>>>>>>>> clustering in the primary studies.
>>>>>>>>>
>>>>>>>>> Am I pointing to the correct formulas? If yes, I don't have SE_uc
>>>>>>>>> in
>>>>>>>>> my primary studies, what should I do?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Tim M
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------ Forwarded Message ------
>>>>>>>>> From: Mikkel Vembye <mikkel.vembye using gmail.com>
>>>>>>>>> Date: Fri, Oct 1, 2021 at 6:54 AM
>>>>>>>>> Subject: Re-re: [R-meta] Meta-analyzing studies that failed to
>>>>>>>>> account
>>>>>>>>> for their nested data
>>>>>>>>> To: <fswfswt using gmail.com>, <r-sig-meta-analysis using r-project.org>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Tim,
>>>>>>>>>
>>>>>>>>> Just to follow up on James, WWC do also have a nice description of
>>>>>>>>> how
>>>>>>>>> they handle cluster trials and quasi-experiments:
>>>>>>>>>
>>>>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>>>>>
>>>>>>>>> Mikkel
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <
>>>>>>>>> jepusto using gmail.com> wrote:
>>>>>>>>> >
>>>>>>>>> > For studies that claim to find negligible ICCs, I would guess
>>>>>>>>> that they base this judgement either on a) failing to reject a test of ICC
>>>>>>>>> = 0 or b) a rule of thumb. a) is not a good justification because with few
>>>>>>>>> classes, the test will have little power. b) is arbitrary and even small
>>>>>>>>> ICCs (of say 0.02 or 0.04) can be consequential for estimating the variance
>>>>>>>>> of the effect size estimate. I would use the ICC adjustment regardless.
>>>>>>>>> >
>>>>>>>>> > To your second question, yes these adjustments are also
>>>>>>>>> important for quasi-experiments.
>>>>>>>>> >
>>>>>>>>> > James
>>>>>>>>> >
>>>>>>>>> > On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <
>>>>>>>>> fswfswt using gmail.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Dear James,
>>>>>>>>> >>
>>>>>>>>> >> Many thanks for this information. Certainly this is serious.
>>>>>>>>> >>
>>>>>>>>> >> I should add that a few of the (newer) studies in my pool say
>>>>>>>>> that
>>>>>>>>> >> they found their ICCs to be negligible and opted for the
>>>>>>>>> single-level
>>>>>>>>> >> analyses (maybe I should not adjust the sampling variances in
>>>>>>>>> these
>>>>>>>>> >> cases, correct?).
>>>>>>>>> >>
>>>>>>>>> >> Also, I'm assuming that I can use these sampling variance
>>>>>>>>> adjustments
>>>>>>>>> >> for quasi-experiments where schools/centers themselves haven't
>>>>>>>>> been
>>>>>>>>> >> randomly recruited as well?
>>>>>>>>> >>
>>>>>>>>> >> Thanks,
>>>>>>>>> >> Tim M
>>>>>>>>> >>
>>>>>>>>> >> On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <
>>>>>>>>> jepusto using gmail.com> wrote:
>>>>>>>>> >> >
>>>>>>>>> >> > Hi Tim,
>>>>>>>>> >> >
>>>>>>>>> >> > One important issue here is that the sampling variance of the
>>>>>>>>> effect size estimate calculated from such a study will be
>>>>>>>>> inaccurate---possibly even an order of magnitude smaller than it should be.
>>>>>>>>> If you ignore this, the consequence will be to make the effect size
>>>>>>>>> estimates appear far more precise than they actually are.
>>>>>>>>> >> >
>>>>>>>>> >> > To properly correct the sampling variance estimate, you would
>>>>>>>>> need to know the intra-class correlation describing the proportion of the
>>>>>>>>> total variation in the outcome that is at the cluster level (in this case,
>>>>>>>>> what fraction of the total variance is between classes?). If this isn't
>>>>>>>>> reported, then it may be possible to develop a reasonable estimate based on
>>>>>>>>> external information. The Cochrane Handbook describes how to correct the
>>>>>>>>> sampling variance based on an imputed intra-class correlation:
>>>>>>>>> >> >
>>>>>>>>> https://training.cochrane.org/handbook/current/chapter-23#section-23-1
>>>>>>>>> >> > Hedges (2007; https://doi.org/10.3102/1076998606298043) and
>>>>>>>>> (2011; https://doi.org/10.3102/1076998610376617) provides
>>>>>>>>> slightly more elaborate methods that can be used if you have more details
>>>>>>>>> about the study designs. Hedges and Hedberg's Variance Almanac (
>>>>>>>>> http://stateva.ci.northwestern.edu/) is a helpful source for
>>>>>>>>> developing estimates of ICCs for educational outcomes.
>>>>>>>>> >> >
>>>>>>>>> >> > James
>>>>>>>>> >> >
>>>>>>>>> >> > On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <
>>>>>>>>> fswfswt using gmail.com> wrote:
>>>>>>>>> >> >>
>>>>>>>>> >> >> Hello All,
>>>>>>>>> >> >>
>>>>>>>>> >> >> I've noticed almost all the studies I have selected for
>>>>>>>>> meta-analysis
>>>>>>>>> >> >> have ignored the nested structure of their data (subjects
>>>>>>>>> nested in
>>>>>>>>> >> >> classrooms) and have conducted only single-level analyses.
>>>>>>>>> >> >>
>>>>>>>>> >> >> I've extracted the condition-level summaries from those
>>>>>>>>> studies (i.e.,
>>>>>>>>> >> >> Means and SDs for C vs. T groups).
>>>>>>>>> >> >>
>>>>>>>>> >> >> But I'm wondering if I can/should make any adjustment to my
>>>>>>>>> >> >> meta-regression model to account for the nested structure of
>>>>>>>>> the data
>>>>>>>>> >> >> in those studies AND if not, whether such a situation poses a
>>>>>>>>> >> >> limitation to my meta-analysis?
>>>>>>>>> >> >>
>>>>>>>>> >> >> Thank you very much for your assistance,
>>>>>>>>> >> >> Tim M
>>>>>>>>> >> >>
>>>>>>>>> >> >> _______________________________________________
>>>>>>>>> >> >> R-sig-meta-analysis mailing list
>>>>>>>>> >> >> R-sig-meta-analysis using r-project.org
>>>>>>>>> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>>>>>
>>>>>>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211007/37a90570/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19281 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211007/37a90570/attachment-0001.png>