[R-meta] Interpreting variance components in rma.mv

[James Pustejovsky]

Hi Yuhang,

Your calculations look correct to me and I think this is a useful aid for
interpreting the results of a multi-level meta-regression. That said, I can
see one potential critique of your calculations. The probabilities you've
calculated treat the model parameter estimates (the average ES and variance
components) as fixed, known values. But of course, they're actually
*estimates* with some degree of uncertainty attached to them. It might be
good to acknowledge this issue by qualifying your statement as "Based on
these model parameter estimates..." or "Treating these model parameter
estimates as known values, the probability that effects with ..."

If you want to better account for the uncertainty in the model parameter
estimates, you could try bootstrapping the probability calculation, similar
to what Mathur and Vanderweele propose in this paper:
Mathur, M. B., & VanderWeele, T. J. (2019). New metrics for meta‐analyses
of heterogeneous effects. *Statistics in Medicine*, *38*(8), 1336-1342.
For a multi-level meta-regression, you would need to do a cluster-level
bootstrap, I think. Another alternative is to move to a fully Bayesian
meta-regression, which would allow for fully integrating the uncertainty
from the model parameter estimates into the probability calculations by
integrating over the joint posterior.

Neither of these proposals is trivial, nor are they something that I would
require if I were reviewing a manuscript with probability calculations like
yours. Like I said, I think they're very helpful as is, especially if their
limitations can be qualified appropriately.

James

On Fri, Aug 19, 2022 at 11:20 AM Yuhang Hu <yh342 using nau.edu> wrote:

> Hello All,
>
> I have a 3-level meta-regression model with a categorical moderator (3
> levels) plus some covariates fit as:
>
> rma.mv(yi ~ 0 + cat_mod + covariates, random = ~ 1 | study/effect)
>
> The means for cat_mod are estimated as: cat1: .27;  cat2 = .33, cat3 = .37
>
> The between-study standard deviation is .548, the within-study standard
> deviation is .434, and the total variation in sd unit is .699.
>
> ***Question: I wonder whether I can say the following or not?***
>
> "The probability that effects with cat1 characteristic are 0 or larger is:
> pnorm(0,.27, .699, lower.tail = FALSE)
> > [1] 0.650
>
> with cat2 characteristic are 0 or larger is:
> pnorm(0,.33, .699, lower.tail = FALSE)
> > [1] 0.68
>
> with cat3 characteristic are 0 or larger is:
> pnorm(0,.37, .699, lower.tail = FALSE)
> > [1] 0.70
> "
> Thanks,
> Yuhang
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]