[R-meta] Notable difference between treditional and bootstrap 95% CI for sigma2: which one is preffered?

[James Pustejovsky]

On question (3) regarding I-squared with RVE, I agree with Wolfgang's
initial assessment that RVE and I-squared pertain to different aspects of
the model. RVE is about assessing uncertainty in the average effect size or
(more generally) meta-regression coefficients of the model (the "fixed"
part of the model). It is robust in the sense that it does not require
correct specification of the random effects structure or sampling
variances/covariances (the "random" part of the model).

I-squared is a description of the proportion of variation in effect size
estimates that can be attributed to true heterogeneity in effect sizes.
Its definition is contingent on the assumptions about the random effects in
the model, which are not "robust" in the RVE sense.

On Tue, Mar 29, 2022 at 6:45 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Ali,
>
> 1) I would say not much. confint() gives you a profile likelihood CI,
> bootstrapping a different type of CI. I wouldn't expect them to be similar
> in the first place - maybe asymptotically, but not even sure about that.
>
> I examined profile likelihood versus bootstrap (versus a bunch of other)
> CIs for the simpler standard RE model in this paper:
>
> Viechtbauer, W. (2007). Confidence intervals for the amount of
> heterogeneity in meta-analysis. Statistics in Medicine, 26(1), 37-52.
> https://doi.org/10.1002/sim.2514
>
> At least in this case, the bootstrap CIs didn't fare so well. The profile
> likelihood CIs did better although they are based on large-sample theory,
> so if k is small, then not so great either (and with log odds ratios - as
> examined in the paper above - things go really bad when the within-study
> sample sizes are small, since the estimated sampling variances can then be
> really off).
>
> 2) For the moment, I would go with the profile ll CIs.
>
> 3) Hmmm, that's a tricky one. In principle, the I^2 calculation and RVE
> are about different things. I^2 is asking how much of the total variance is
> due to heterogeneity (or particular variance components in the model),
> while RVE is about making inferences about the model coefficients. But RVE
> is also in some sense about the variance -- it uses the product of the
> residuals to get a (very rough!) approximation to the marginal var-cov
> matrix of the effect size estimates and then squishes this together into
> the var-cov matrix of the model coefficients (which then ends up being a
> really good approximation to the var-cov matrix of the model coefficients).
> Maybe one could compute a sort of robust version of the P matrix that is
> used in the calculation of I^2 - which might again be a very rough
> approximation, but since I^2 in essence takes the average of the trace of
> P, this 'cluster-robust version of P' might again be acceptable to use in
> the calculation of I^2. But all of this is just mere brainstorming. At the
> moment, I would just report the I^2 from the model before applying RVE.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of towhidi
> >Sent: Monday, 28 March, 2022 12:54
> >To: r sig meta-analysis list
> >Subject: [R-meta] Notable difference between treditional and bootstrap
> 95% CI for
> >sigma2: which one is preffered?
> >
> >Dear all,
> >
> >I am working on a dataset with a multilevel structure: 185 SMDs, nested
> >in 108 outcomes, nested in 41 comparisons (to address multiarmed trials)
> >nested in 34 studies (random = ~1 |
> >stud_id/cont_id/outcome_id/occasion).
> >
> >For some of the sigma^2 values, the CI from confint() is largely
> >different from the bootstrap CI, e.g., for a sigma^2 = .04, the upper
> >limit from confint() is .38, while the boot CI upper limit is .21.
> >
> >(1) What does this difference imply?
> >
> >(2) When such differences exist between traditional and boot CIs, Which
> >one is more reliable?
> >
> >For calculating boot CI I used the following:
> >
> >sim <- simulate(res, nsim=300)
> >sav <- lapply(sim, function(x) {
> >tmp <- try(rma.mv(x, vi, data = dat, random = res$random), silent=TRUE)
> >if (inherits(tmp, "try-error")) {
> >next
> >} else {
> >tmp
> >}})
> >
> >sigma2.l4 <- sapply(sav, function(x) x$sigma2[2])
> >
> >quantile(sigma2.l4, c(0.025, .975))
> >
> >Of note, I have checked the profile plot and there seemed to be no
> >convergence problem.
> >
> >I also have another related question:
> >(3) Is the general formula for I^2 for multilevel models
> >(https://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate
> )
> >can be applied to RVE without any modifications?
> >
> >Thank you.
> >
> >--
> >Ali Zia-Tohidi MSc
> >Clinical Psychology
> >University of Tehran
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]