[R-meta] Meta-analyzing studies that failed to account for their nested data

[Mikkel Vembye]

Hi Tim,

Glad that I/we can help. You find the ANCOVA examples (both uncorrected and
corrected) in Table 3.

[image: image.png]

I forgot to mention that you also can find some corrections to Hedges
(2007) in Table 5.

All the best,
Mikkel

Den fre. 1. okt. 2021 kl. 19.21 skrev Timothy MacKenzie <fswfswt using gmail.com>:

> Dear James and Mikkel,
>
> Thank you both. In my case, the primary studies have used AN(C)OVAs
> with (non-)random assignment of classes to conditions. I have the
> Means and SDs of student-level data for conditions.
>
> Based on:
> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
> , I should use the unbiased version of equation E.5.1 to compute the
> SMD effect size (g):
>
> g = wb / s * sqrt(  1 - (2*(n-1)*icc / N - 2)  ) ; s = pooled standard
> deviation; n = ave. cluster size; N = n_t + n_c;  w = hedges'
> correction factor
>
> Based on the same document, the standard error of g (SE[g]) for
> cluster-assignment studies is (equation E.7.1 under "Cluster
> assignment"):
>
> SE[g] = w * sqrt(  (SE_uc / s )^2 * eta + (g^2 / (2*h))  ); eta =  1 +
> (n - 1)*icc; h = ( [(N-2)-2*(n-1)*icc ]^2 ) / ((N-2)*(1-icc)^2 +
> 2*(N-2*n)*icc*(1-icc)  )
>
> where SE_uc = regression coefficient standard errors uncorrected for
> clustering in the primary studies.
>
> Am I pointing to the correct formulas? If yes, I don't have SE_uc in
> my primary studies, what should I do?
>
> Thanks,
> Tim M
>
>
> ------ Forwarded Message ------
> From: Mikkel Vembye <mikkel.vembye using gmail.com>
> Date: Fri, Oct 1, 2021 at 6:54 AM
> Subject: Re-re: [R-meta] Meta-analyzing studies that failed to account
> for their nested data
> To: <fswfswt using gmail.com>, <r-sig-meta-analysis using r-project.org>
>
>
> Hi Tim,
>
> Just to follow up on James, WWC do also have a nice description of how
> they handle cluster trials and quasi-experiments:
>
> https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-41-Supplement-508_09212020.pdf
>
> Mikkel
>
>
> On Thu, Sep 30, 2021 at 11:09 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
> >
> > For studies that claim to find negligible ICCs, I would guess that they
> base this judgement either on a) failing to reject a test of ICC = 0 or b)
> a rule of thumb. a) is not a good justification because with few classes,
> the test will have little power. b) is arbitrary and even small ICCs (of
> say 0.02 or 0.04) can be consequential for estimating the variance of the
> effect size estimate. I would use the ICC adjustment regardless.
> >
> > To your second question, yes these adjustments are also important for
> quasi-experiments.
> >
> > James
> >
> > On Thu, Sep 30, 2021 at 10:53 PM Timothy MacKenzie <fswfswt using gmail.com>
> wrote:
> >>
> >> Dear James,
> >>
> >> Many thanks for this information. Certainly this is serious.
> >>
> >> I should add that a few of the (newer) studies in my pool say that
> >> they found their ICCs to be negligible and opted for the single-level
> >> analyses (maybe I should not adjust the sampling variances in these
> >> cases, correct?).
> >>
> >> Also, I'm assuming that I can use these sampling variance adjustments
> >> for quasi-experiments where schools/centers themselves haven't been
> >> randomly recruited as well?
> >>
> >> Thanks,
> >> Tim M
> >>
> >> On Thu, Sep 30, 2021 at 9:40 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
> >> >
> >> > Hi Tim,
> >> >
> >> > One important issue here is that the sampling variance of the effect
> size estimate calculated from such a study will be inaccurate---possibly
> even an order of magnitude smaller than it should be. If you ignore this,
> the consequence will be to make the effect size estimates appear far more
> precise than they actually are.
> >> >
> >> > To properly correct the sampling variance estimate, you would need to
> know the intra-class correlation describing the proportion of the total
> variation in the outcome that is at the cluster level (in this case, what
> fraction of the total variance is between classes?). If this isn't
> reported, then it may be possible to develop a reasonable estimate based on
> external information. The Cochrane Handbook describes how to correct the
> sampling variance based on an imputed intra-class correlation:
> >> >
> https://training.cochrane.org/handbook/current/chapter-23#section-23-1
> >> > Hedges (2007; https://doi.org/10.3102/1076998606298043) and (2011;
> https://doi.org/10.3102/1076998610376617) provides slightly more
> elaborate methods that can be used if you have more details about the study
> designs. Hedges and Hedberg's Variance Almanac (
> http://stateva.ci.northwestern.edu/) is a helpful source for developing
> estimates of ICCs for educational outcomes.
> >> >
> >> > James
> >> >
> >> > On Thu, Sep 30, 2021 at 4:58 PM Timothy MacKenzie <fswfswt using gmail.com>
> wrote:
> >> >>
> >> >> Hello All,
> >> >>
> >> >> I've noticed almost all the studies I have selected for meta-analysis
> >> >> have ignored the nested structure of their data (subjects nested in
> >> >> classrooms) and have conducted only single-level analyses.
> >> >>
> >> >> I've extracted the condition-level summaries from those studies
> (i.e.,
> >> >> Means and SDs for C vs. T groups).
> >> >>
> >> >> But I'm wondering if I can/should make any adjustment to my
> >> >> meta-regression model to account for the nested structure of the data
> >> >> in those studies AND if not, whether such a situation poses a
> >> >> limitation to my meta-analysis?
> >> >>
> >> >> Thank you very much for your assistance,
> >> >> Tim M
> >> >>
> >> >> _______________________________________________
> >> >> R-sig-meta-analysis mailing list
> >> >> R-sig-meta-analysis using r-project.org
> >> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/a6f25e5e/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 19281 bytes
Desc: not available
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20211001/a6f25e5e/attachment-0001.png>