[R-meta] Meta-analyzing studies that failed to account for their nested data

[Mikkel Vembye]

Sorry. I referred to an older version of the Appendix. I usually just
follow WWC's recommendation when I cannot obtain R2. This is

"if R2 is not available, then WWC will take a cautious approach to
calculating the
standard error and assume a value of zero for R2. This cautious approach
will overestimate the
magnitude of the standard error but protects against type I error." (See
the WWC Procedure Handbook, p. E-5)

https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-508.pdf


Whether this approach is better, is more a James question.

Mikkel

Den fre. 1. okt. 2021 kl. 19.44 skrev Timothy MacKenzie <fswfswt using gmail.com>:

> Much appreciated, Mikkel. I saw that. BTW, there is no Table 5, it's a
> typo in the WWC document (I found other typos as well).
>
> But I have both ANOVAs and a few ANCOVAs from primary studies that did
> cluster assignment but ignored nesting structure, with barely any R^2
> reported in them.
>
> My understanding is that I should find a more general SE[g] that only
> requires icc, am I correct in thinking this way?
>
> Thanks,
> Tim M
>
> On Fri, Oct 1, 2021 at 12:32 PM Mikkel Vembye <mikkel.vembye using gmail.com>
> wrote:
>
>> Hi Tim,
>>
>> Glad that I/we can help. You find the ANCOVA examples (both uncorrected
>> and corrected) in Table 3.
>>
>> [image: image.png]
>>
>> I forgot to mention that you also can find some corrections to Hedges
>> (2007) in Table 5.
>>
>> All the best,
>> Mikkel
>>
>> Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <
>> fswfswt using gmail.com>:
>>
>>> Dear James and Mikkel,
>>>
>>> Thank you both. In my case, the primary studies have used AN(C)OVAs
>>> with (non-)random assignment of classes to conditions. I have the
>>> Means and SDs of student-level data for conditions.
>>>
>>> Based on:
>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>> , I should use the unbiased version of equation E.5.1 to compute the
>>> SMD effect size (g):
>>>
>>> g = wb / s * sqrt(  1 - (2*(n-1)*icc / N - 2)  ) ; s = pooled standard
>>> deviation; n = ave. cluster size; N = n_t + n_c;  w = hedges'
>>> correction factor
>>>
>>> Based on the same document, the standard error of g (SE[g]) for
>>> cluster-assignment studies is (equation E.7.1 under "Cluster
>>> assignment"):
>>>
>>> SE[g] = w * sqrt(  (SE_uc / s )^2 * eta + (g^2 / (2*h))  ); eta =  1 +
>>> (n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
>>> 2*(N-2*n)*icc*(1-icc)  )
>>>
>>> where SE_uc = regression coefficient standard errors uncorrected for
>>> clustering in the primary studies.
>>>
>>> Am I pointing to the correct formulas? If yes, I don't have SE_uc in
>>> my primary studies, what should I do?
>>>
>>> Thanks,
>>> Tim M
>>>
>>>
>>> ------ Forwarded Message ------
>>> From: Mikkel Vembye <mikkel.vembye using gmail.com>
>>> Date: Fri, Oct 1, 2021 at 6:54 AM
>>> Subject: Re-re: [R-meta] Meta-analyzing studies that failed to account
>>> for their nested data
>>> To: <fswfswt using gmail.com>, <r-sig-meta-analysis using r-project.org>
>>>
>>>
>>> Hi Tim,
>>>
>>> Just to follow up on James, WWC do also have a nice description of how
>>> they handle cluster trials and quasi-experiments:
>>>
>>> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>>>
>>> Mikkel
>>>
>>>
>>> On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>> >
>>> > For studies that claim to find negligible ICCs, I would guess that
>>> they base this judgement either on a) failing to reject a test of ICC = 0
>>> or b) a rule of thumb. a) is not a good justification because with few
>>> classes, the test will have little power. b) is arbitrary and even small
>>> ICCs (of say 0.02 or 0.04) can be consequential for estimating the variance
>>> of the effect size estimate. I would use the ICC adjustment regardless.
>>> >
>>> > To your second question, yes these adjustments are also important for
>>> quasi-experiments.
>>> >
>>> > James
>>> >
>>> > On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <fswfswt using gmail.com>
>>> wrote:
>>> >>
>>> >> Dear James,
>>> >>
>>> >> Many thanks for this information. Certainly this is serious.
>>> >>
>>> >> I should add that a few of the (newer) studies in my pool say that
>>> >> they found their ICCs to be negligible and opted for the single-level
>>> >> analyses (maybe I should not adjust the sampling variances in these
>>> >> cases, correct?).
>>> >>
>>> >> Also, I'm assuming that I can use these sampling variance adjustments
>>> >> for quasi-experiments where schools/centers themselves haven't been
>>> >> randomly recruited as well?
>>> >>
>>> >> Thanks,
>>> >> Tim M
>>> >>
>>> >> On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>> >> >
>>> >> > Hi Tim,
>>> >> >
>>> >> > One important issue here is that the sampling variance of the
>>> effect size estimate calculated from such a study will be
>>> inaccurate---possibly even an order of magnitude smaller than it should be.
>>> If you ignore this, the consequence will be to make the effect size
>>> estimates appear far more precise than they actually are.
>>> >> >
>>> >> > To properly correct the sampling variance estimate, you would need
>>> to know the intra-class correlation describing the proportion of the total
>>> variation in the outcome that is at the cluster level (in this case, what
>>> fraction of the total variance is between classes?). If this isn't
>>> reported, then it may be possible to develop a reasonable estimate based on
>>> external information. The Cochrane Handbook describes how to correct the
>>> sampling variance based on an imputed intra-class correlation:
>>> >> >
>>> https://training.cochrane.org/handbook/current/chapter-23#section-23-1
>>> >> > Hedges (2007; https://doi.org/10.3102/1076998606298043) and (2011;
>>> https://doi.org/10.3102/1076998610376617) provides slightly more
>>> elaborate methods that can be used if you have more details about the study
>>> designs. Hedges and Hedberg's Variance Almanac (
>>> http://stateva.ci.northwestern.edu/) is a helpful source for developing
>>> estimates of ICCs for educational outcomes.
>>> >> >
>>> >> > James
>>> >> >
>>> >> > On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <
>>> fswfswt using gmail.com> wrote:
>>> >> >>
>>> >> >> Hello All,
>>> >> >>
>>> >> >> I've noticed almost all the studies I have selected for
>>> meta-analysis
>>> >> >> have ignored the nested structure of their data (subjects nested in
>>> >> >> classrooms) and have conducted only single-level analyses.
>>> >> >>
>>> >> >> I've extracted the condition-level summaries from those studies
>>> (i.e.,
>>> >> >> Means and SDs for C vs. T groups).
>>> >> >>
>>> >> >> But I'm wondering if I can/should make any adjustment to my
>>> >> >> meta-regression model to account for the nested structure of the
>>> data
>>> >> >> in those studies AND if not, whether such a situation poses a
>>> >> >> limitation to my meta-analysis?
>>> >> >>
>>> >> >> Thank you very much for your assistance,
>>> >> >> Tim M
>>> >> >>
>>> >> >> _______________________________________________
>>> >> >> R-sig-meta-analysis mailing list
>>> >> >> R-sig-meta-analysis using r-project.org
>>> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/325469f4/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19281 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/325469f4/attachment-0001.png>