[R-meta] Notable difference between treditional and bootstrap 95% CI for sigma2: which one is preffered?

[towhidi]

On 2022-03-29 19:06, James Pustejovsky wrote:

> On question (3) regarding I-squared with RVE, I agree with Wolfgang's 
> initial assessment that RVE and I-squared pertain to different aspects 
> of the model. RVE is about assessing uncertainty in the average effect 
> size or (more generally) meta-regression coefficients of the model (the 
> "fixed" part of the model). It is robust in the sense that it does not 
> require correct specification of the random effects structure or 
> sampling variances/covariances (the "random" part of the model).
> 
> I-squared is a description of the proportion of variation in effect 
> size estimates that can be attributed to true heterogeneity in effect 
> sizes. Its definition is contingent on the assumptions about the random 
> effects in the model, which are not "robust" in the RVE sense.
> 
> On Tue, Mar 29, 2022 at 6:45 AM Viechtbauer, Wolfgang (SP) 
> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> 
>> Dear Ali,
>> 
>> 1) I would say not much. confint() gives you a profile likelihood CI, 
>> bootstrapping a different type of CI. I wouldn't expect them to be 
>> similar in the first place - maybe asymptotically, but not even sure 
>> about that.
>> 
>> I examined profile likelihood versus bootstrap (versus a bunch of 
>> other) CIs for the simpler standard RE model in this paper:
>> 
>> Viechtbauer, W. (2007). Confidence intervals for the amount of 
>> heterogeneity in meta-analysis. Statistics in Medicine, 26(1), 37-52. 
>> https://doi.org/10.1002/sim.2514
>> 
>> At least in this case, the bootstrap CIs didn't fare so well. The 
>> profile likelihood CIs did better although they are based on 
>> large-sample theory, so if k is small, then not so great either (and 
>> with log odds ratios - as examined in the paper above - things go 
>> really bad when the within-study sample sizes are small, since the 
>> estimated sampling variances can then be really off).
>> 
>> 2) For the moment, I would go with the profile ll CIs.
>> 
>> 3) Hmmm, that's a tricky one. In principle, the I^2 calculation and 
>> RVE are about different things. I^2 is asking how much of the total 
>> variance is due to heterogeneity (or particular variance components in 
>> the model), while RVE is about making inferences about the model 
>> coefficients. But RVE is also in some sense about the variance -- it 
>> uses the product of the residuals to get a (very rough!) approximation 
>> to the marginal var-cov matrix of the effect size estimates and then 
>> squishes this together into the var-cov matrix of the model 
>> coefficients (which then ends up being a really good approximation to 
>> the var-cov matrix of the model coefficients). Maybe one could compute 
>> a sort of robust version of the P matrix that is used in the 
>> calculation of I^2 - which might again be a very rough approximation, 
>> but since I^2 in essence takes the average of the trace of P, this 
>> 'cluster-robust version of P' might again be acceptable to use in the 
>> calculation of I^2. But all of this is just mere brainstorming. At the 
>> moment, I would just report the I^2 from the model before applying 
>> RVE.
>> 
>> Best,
>> Wolfgang
>> 
>>> -----Original Message-----
>>> From: R-sig-meta-analysis 
>>> [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>> Behalf Of towhidi
>>> Sent: Monday, 28 March, 2022 12:54
>>> To: r sig meta-analysis list
>>> Subject: [R-meta] Notable difference between treditional and 
>>> bootstrap 95% CI for
>>> sigma2: which one is preffered?
>>> 
>>> Dear all,
>>> 
>>> I am working on a dataset with a multilevel structure: 185 SMDs, 
>>> nested
>>> in 108 outcomes, nested in 41 comparisons (to address multiarmed 
>>> trials)
>>> nested in 34 studies (random = ~1 |
>>> stud_id/cont_id/outcome_id/occasion).
>>> 
>>> For some of the sigma^2 values, the CI from confint() is largely
>>> different from the bootstrap CI, e.g., for a sigma^2 = .04, the upper
>>> limit from confint() is .38, while the boot CI upper limit is .21.
>>> 
>>> (1) What does this difference imply?
>>> 
>>> (2) When such differences exist between traditional and boot CIs, 
>>> Which
>>> one is more reliable?
>>> 
>>> For calculating boot CI I used the following:
>>> 
>>> sim <- simulate(res, nsim=300)
>>> sav <- lapply(sim, function(x) {
>>> tmp <- try(rma.mv(x, vi, data = dat, random = res$random), 
>>> silent=TRUE)
>>> if (inherits(tmp, "try-error")) {
>>> next
>>> } else {
>>> tmp
>>> }})
>>> 
>>> sigma2.l4 <- sapply(sav, function(x) x$sigma2[2])
>>> 
>>> quantile(sigma2.l4, c(0.025, .975))
>>> 
>>> Of note, I have checked the profile plot and there seemed to be no
>>> convergence problem.
>>> 
>>> I also have another related question:
>>> (3) Is the general formula for I^2 for multilevel models
>>> (https://www.metafor-project.org/doku.php/tips:i2_multilevel_multivariate)
>>> can be applied to RVE without any modifications?
>>> 
>>> Thank you.
>>> 
>>> --
>>> Ali Zia-Tohidi MSc
>>> Clinical Psychology
>>> University of Tehran
>> 
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

Dear Wolfgang, and dear James,

Thank you for elaborating on the issue, and thanks for the reference on 
CI for heterogeneity.

Best,
Ali