[R-meta] Confidence Intervals after aggregating estimates

[Viechtbauer, Wolfgang (NP)]

Dear Max,

Simply put, the variance components also affect studies with a single estimate. For example, for study 4, the new sampling variance in 'agg' is simply the original sampling variance plus the sum of the two variance components:

agg[agg$study == 4,]
dat$vi[dat$study == 4] + sum(res$sigma2)

Best,
Wolfgang

> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
> Of Maximilian Steininger via R-sig-meta-analysis
> Sent: Friday, May 17, 2024 15:09
> To: Maximilian Steininger via R-sig-meta-analysis <r-sig-meta-analysis using r-
> project.org>
> Cc: Maximilian Steininger <maximilian.steininger using univie.ac.at>
> Subject: [R-meta] Confidence Intervals after aggregating estimates
>
> Dear all,
>
> I want to create a forest plot for a multilevel meta-analysis. I ran a three-
> level metaanalysis that accounts for different dependencies in the data (as
> suggested here https://wviechtb.github.io/metafor/reference/misc-
> recs.html#general-workflow-for-meta-analyses-involving-complex-dependency-
> structures). I am under the impression that it „is more informative“ to create a
> forest plot for this analysis on the level of studies (using aggregated effect
> sizes if a study has more than one) rather than on the level of individual
> effects. I followed the example code from the metafor-project, which can be
> found here:
>
> https://www.metafor-project.org/doku.php/tips:forest_plot_with_aggregated_values
>
> However, I noticed some peculiarities regarding the confidence intervals of the
> effect sizes in the resulting forest plot, which I can’t make sense of.
>
> Referring to the dataset and the plots presented in the link above, I noticed
> that after aggregating the data, the confidence intervals in the plot get very
> large. Even so, for studies that only have one effect size estimate. E.g. before
> aggregating the dataset studies 4, 5, and 8 in the example have an effect size
> and CIs of:
>
> Study 4: -0.04 [-0.40, 0.31]
> Study 5: 1.55 [0.82, 2.28]
> Study 8: 0.37 [0.09, 0.65]
>
> Applying the aggregate() function to the dataset, recalculating the model -
> which indeed results in the same pooled estimate - and creating the forest plot
> with the aggregated data works, but the studies have rather large CIs. Studies
> 4, 5, and 8 - which only have one effect estimate - now have a considerably
> larger CI e.g.:
>
> Study 4: -0.04 [-1.06, 0.97]
> Study 5: 1.55 [0.35, 2.75]
> Study 8: 0.37 [-0.62, 1.36]
>
> I understand that aggregating the data of studies with several effect sizes can
> lead to larger CIs, but why is this also happening for studies with single
> effect estimates?
>
> Many thanks!
> Max
>
> ——
>
> Mag. Maximilian Steininger
>   PhD candidate
>
>   Social, Cognitive and Affective Neuroscience Unit
>   Faculty of Psychology
>   University of Vienna
>
>   Liebiggasse 5
>   1010 Vienna, Austria
>
>   e: maximilian.steininger using univie.ac.at
>   w: http://scan.psy.univie.ac.at