[R-meta] Meta-analyzing studies that failed to account for their nested data

Wed Oct 6 00:20:42 CEST 2021

To provide an example, one study has randomly assigned 3 classrooms to 3
conditions (T1, T2, C).

(T1 <-- class 1, n = 15); (T2 <-- class 2, n = 13); (C <-- class 3, n = 15)

So for this study, I have 3 "means" and 3 "sds" for T1, T2, and T3. So,
should I adjust the SEs of the resulting SMDs using WWC's or Hedges' (
https://doi.org/10.3102/1076998606298043) formulas here?

Thanks, again
Tim M

On Tue, Oct 5, 2021 at 4:51 PM Timothy MacKenzie <fswfswt using gmail.com> wrote:

> Dear All,
>
> I just read a paper (
> https://link.springer.com/content/pdf/10.3758/s13428-011-0153-1.pdf) that
> made me think whether I understood your advice correctly or not.
>
> My question was what to do with studies that provide "means" and "sds" for
> "student-level" data while they ignore their students being nested in
> classrooms.
>
> The article linked above says that only if we calculate an SMD based on
> "cluster-level" data (d_cluster), then d_cluster is upwardly biased.
>
> But I want to calculate an SMD for studies only based on "student-level"
> data (d_individual), so do I need to adjust "d_individual" just like
> "d_cluster"?
>
> Thank you for clarifying this confusion for me,
> Tim M
>
>
> On Fri, Oct 1, 2021 at 3:47 PM Mikkel Vembye <mikkel.vembye using gmail.com>
> wrote:
>
>> No it isn't. I mainly have it from James and his blog (see link below),
>> and then I added the correction from WWC.
>> https://www.jepusto.com/alternative-formulas-for-the-smd/
>>
>> James, please let me/us know if this approach is viable. :)
>>
>> Mikkel
>>
>> Den fre. 1. okt. 2021 kl. 22.27 skrev Timothy MacKenzie <
>> fswfswt using gmail.com>:
>>
>>> Mikkel,
>>>
>>> Thank you. Is this new formula (with F-value) for the sampling variance
>>> of g in the WWC procedures handbook? Would you please provide a reference?
>>>
>>> Best regards,
>>> Tim M
>>>
>>> On Fri, Oct 1, 2021 at 3:22 PM Mikkel Vembye <mikkel.vembye using gmail.com>
>>> wrote:
>>>
>>>> When I have relevant F-values from ANCOVA and related models, I
>>>> calculate vg_corrected = omega^2 * [(g_corrected^2/F-value) * eta +
>>>> g_corrected^2/(2*h)]
>>>>
>>>> Have a nice weekend.
>>>>
>>>> Best,
>>>> Mikkel
>>>>
>>>> Den fre. 1. okt. 2021 kl. 21.41 skrev Mikkel Vembye <
>>>> mikkel.vembye using gmail.com>:
>>>>
>>>>> Sorry. I referred to an older version of the Appendix. I usually just
>>>>> follow WWC's recommendation when I cannot obtain R2. This is
>>>>>
>>>>> "if R2 is not available, then WWC will take a cautious approach to
>>>>> calculating the
>>>>> standard error and assume a value of zero for R2. This cautious
>>>>> approach will overestimate the
>>>>> magnitude of the standard error but protects against type I error."
>>>>> (See the WWC Procedure Handbook, p. E-5)
>>>>>
>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf
>>>>>
>>>>>
>>>>> Whether this approach is better, is more a James question.
>>>>>
>>>>> Mikkel
>>>>>
>>>>> Den fre. 1. okt. 2021 kl. 19.44 skrev Timothy MacKenzie <
>>>>> fswfswt using gmail.com>:
>>>>>
>>>>>> Much appreciated, Mikkel. I saw that. BTW, there is no Table 5, it's
>>>>>> a typo in the WWC document (I found other typos as well).
>>>>>>
>>>>>> But I have both ANOVAs and a few ANCOVAs from primary studies that
>>>>>> did cluster assignment but ignored nesting structure, with barely any R^2
>>>>>> reported in them.
>>>>>>
>>>>>> My understanding is that I should find a more general SE[g] that only
>>>>>> requires icc, am I correct in thinking this way?
>>>>>>
>>>>>> Thanks,
>>>>>> Tim M
>>>>>>
>>>>>> On Fri, Oct 1, 2021 at 12:32 PM Mikkel Vembye <
>>>>>> mikkel.vembye using gmail.com> wrote:
>>>>>>
>>>>>>> Hi Tim,
>>>>>>>
>>>>>>> Glad that I/we can help. You find the ANCOVA examples (both
>>>>>>> uncorrected and corrected) in Table 3.
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> I forgot to mention that you also can find some corrections to
>>>>>>> Hedges (2007) in Table 5.
>>>>>>>
>>>>>>> All the best,
>>>>>>> Mikkel
>>>>>>>
>>>>>>> Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <
>>>>>>> fswfswt using gmail.com>:
>>>>>>>
>>>>>>>> Dear James and Mikkel,
>>>>>>>>
>>>>>>>> Thank you both. In my case, the primary studies have used AN(C)OVAs
>>>>>>>> with (non-)random assignment of classes to conditions. I have the
>>>>>>>> Means and SDs of student-level data for conditions.
>>>>>>>>
>>>>>>>> Based on:
>>>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>>>> , I should use the unbiased version of equation E.5.1 to compute the
>>>>>>>> SMD effect size (g):
>>>>>>>>
>>>>>>>> g = wb / s * sqrt(  1 - (2*(n-1)*icc / N - 2)  ) ; s = pooled
>>>>>>>> standard
>>>>>>>> deviation; n = ave. cluster size; N = n_t + n_c;  w = hedges'
>>>>>>>> correction factor
>>>>>>>>
>>>>>>>> Based on the same document, the standard error of g (SE[g]) for
>>>>>>>> cluster-assignment studies is (equation E.7.1 under "Cluster
>>>>>>>> assignment"):
>>>>>>>>
>>>>>>>> SE[g] = w * sqrt(  (SE_uc / s )^2 * eta + (g^2 / (2*h))  ); eta =
>>>>>>>> 1 +
>>>>>>>> (n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
>>>>>>>> 2*(N-2*n)*icc*(1-icc)  )
>>>>>>>>
>>>>>>>> where SE_uc = regression coefficient standard errors uncorrected for
>>>>>>>> clustering in the primary studies.
>>>>>>>>
>>>>>>>> Am I pointing to the correct formulas? If yes, I don't have SE_uc in
>>>>>>>> my primary studies, what should I do?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tim M
>>>>>>>>
>>>>>>>>
>>>>>>>> ------ Forwarded Message ------
>>>>>>>> From: Mikkel Vembye <mikkel.vembye using gmail.com>
>>>>>>>> Date: Fri, Oct 1, 2021 at 6:54 AM
>>>>>>>> Subject: Re-re: [R-meta] Meta-analyzing studies that failed to
>>>>>>>> account
>>>>>>>> for their nested data
>>>>>>>> To: <fswfswt using gmail.com>, <r-sig-meta-analysis using r-project.org>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Tim,
>>>>>>>>
>>>>>>>> Just to follow up on James, WWC do also have a nice description of
>>>>>>>> how
>>>>>>>> they handle cluster trials and quasi-experiments:
>>>>>>>>
>>>>>>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>>>>>>
>>>>>>>> Mikkel
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <
>>>>>>>> jepusto using gmail.com> wrote:
>>>>>>>> >
>>>>>>>> > For studies that claim to find negligible ICCs, I would guess
>>>>>>>> that they base this judgement either on a) failing to reject a test of ICC
>>>>>>>> = 0 or b) a rule of thumb. a) is not a good justification because with few
>>>>>>>> classes, the test will have little power. b) is arbitrary and even small
>>>>>>>> ICCs (of say 0.02 or 0.04) can be consequential for estimating the variance
>>>>>>>> of the effect size estimate. I would use the ICC adjustment regardless.
>>>>>>>> >
>>>>>>>> > To your second question, yes these adjustments are also important
>>>>>>>> for quasi-experiments.
>>>>>>>> >
>>>>>>>> > James
>>>>>>>> >
>>>>>>>> > On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <
>>>>>>>> fswfswt using gmail.com> wrote:
>>>>>>>> >>
>>>>>>>> >> Dear James,
>>>>>>>> >>
>>>>>>>> >> Many thanks for this information. Certainly this is serious.
>>>>>>>> >>
>>>>>>>> >> I should add that a few of the (newer) studies in my pool say
>>>>>>>> that
>>>>>>>> >> they found their ICCs to be negligible and opted for the
>>>>>>>> single-level
>>>>>>>> >> analyses (maybe I should not adjust the sampling variances in
>>>>>>>> these
>>>>>>>> >> cases, correct?).
>>>>>>>> >>
>>>>>>>> >> Also, I'm assuming that I can use these sampling variance
>>>>>>>> adjustments
>>>>>>>> >> for quasi-experiments where schools/centers themselves haven't
>>>>>>>> been
>>>>>>>> >> randomly recruited as well?
>>>>>>>> >>
>>>>>>>> >> Thanks,
>>>>>>>> >> Tim M
>>>>>>>> >>
>>>>>>>> >> On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <
>>>>>>>> jepusto using gmail.com> wrote:
>>>>>>>> >> >
>>>>>>>> >> > Hi Tim,
>>>>>>>> >> >
>>>>>>>> >> > One important issue here is that the sampling variance of the
>>>>>>>> effect size estimate calculated from such a study will be
>>>>>>>> inaccurate---possibly even an order of magnitude smaller than it should be.
>>>>>>>> If you ignore this, the consequence will be to make the effect size
>>>>>>>> estimates appear far more precise than they actually are.
>>>>>>>> >> >
>>>>>>>> >> > To properly correct the sampling variance estimate, you would
>>>>>>>> need to know the intra-class correlation describing the proportion of the
>>>>>>>> total variation in the outcome that is at the cluster level (in this case,
>>>>>>>> what fraction of the total variance is between classes?). If this isn't
>>>>>>>> reported, then it may be possible to develop a reasonable estimate based on
>>>>>>>> external information. The Cochrane Handbook describes how to correct the
>>>>>>>> sampling variance based on an imputed intra-class correlation:
>>>>>>>> >> >
>>>>>>>> https://training.cochrane.org/handbook/current/chapter-23#section-23-1
>>>>>>>> >> > Hedges (2007; https://doi.org/10.3102/1076998606298043) and
>>>>>>>> (2011; https://doi.org/10.3102/1076998610376617) provides slightly
>>>>>>>> more elaborate methods that can be used if you have more details about the
>>>>>>>> study designs. Hedges and Hedberg's Variance Almanac (
>>>>>>>> http://stateva.ci.northwestern.edu/) is a helpful source for
>>>>>>>> developing estimates of ICCs for educational outcomes.
>>>>>>>> >> >
>>>>>>>> >> > James
>>>>>>>> >> >
>>>>>>>> >> > On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <
>>>>>>>> fswfswt using gmail.com> wrote:
>>>>>>>> >> >>
>>>>>>>> >> >> Hello All,
>>>>>>>> >> >>
>>>>>>>> >> >> I've noticed almost all the studies I have selected for
>>>>>>>> meta-analysis
>>>>>>>> >> >> have ignored the nested structure of their data (subjects
>>>>>>>> nested in
>>>>>>>> >> >> classrooms) and have conducted only single-level analyses.
>>>>>>>> >> >>
>>>>>>>> >> >> I've extracted the condition-level summaries from those
>>>>>>>> studies (i.e.,
>>>>>>>> >> >> Means and SDs for C vs. T groups).
>>>>>>>> >> >>
>>>>>>>> >> >> But I'm wondering if I can/should make any adjustment to my
>>>>>>>> >> >> meta-regression model to account for the nested structure of
>>>>>>>> the data
>>>>>>>> >> >> in those studies AND if not, whether such a situation poses a
>>>>>>>> >> >> limitation to my meta-analysis?
>>>>>>>> >> >>
>>>>>>>> >> >> Thank you very much for your assistance,
>>>>>>>> >> >> Tim M
>>>>>>>> >> >>
>>>>>>>> >> >> _______________________________________________
>>>>>>>> >> >> R-sig-meta-analysis mailing list
>>>>>>>> >> >> R-sig-meta-analysis using r-project.org
>>>>>>>> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>>>>
>>>>>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211005/5006e710/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19281 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211005/5006e710/attachment-0001.png>