[R-meta] Interpreting variance components in rma.mv

[Yuhang Hu]

Dear James,

I appreciate your confirmation. If my models becomes (which I'm considering
at the moment):

rma.mv(yi ~ 0 + cat_mod * time + covariates, random = ~ 1 | study/effect)

and, assuming my variance components are the same (total sd = 0.699), and

At time 0, the estimated means for cat_mod are: cat1: 0.12; cat2: 0.13;
cat3: 0.13
At time 1, the estimated means for cat_mod are: cat1: 0.11; cat2: 0.17;
cat3: 0.18

Then, I will end up using that total sd of 0.699 for the interpretation of
both time 0 mean effects and time 1 mean effects. But is it appropriate to
assume that true effects' dispersion at time 0 and time 1 is exactly the
same (equality of variances across time points)?

Thank you for your time.
Yuhang

On Sat, Aug 20, 2022 at 7:32 AM James Pustejovsky <jepusto using gmail.com> wrote:

> Hi Yuhang,
>
> Your calculations look correct to me and I think this is a useful aid for
> interpreting the results of a multi-level meta-regression. That said, I can
> see one potential critique of your calculations. The probabilities you've
> calculated treat the model parameter estimates (the average ES and variance
> components) as fixed, known values. But of course, they're actually
> *estimates* with some degree of uncertainty attached to them. It might be
> good to acknowledge this issue by qualifying your statement as "Based on
> these model parameter estimates..." or "Treating these model parameter
> estimates as known values, the probability that effects with ..."
>
> If you want to better account for the uncertainty in the model parameter
> estimates, you could try bootstrapping the probability calculation, similar
> to what Mathur and Vanderweele propose in this paper:
> Mathur, M. B., & VanderWeele, T. J. (2019). New metrics for meta‐analyses
> of heterogeneous effects. *Statistics in Medicine*, *38*(8), 1336-1342.
> For a multi-level meta-regression, you would need to do a cluster-level
> bootstrap, I think. Another alternative is to move to a fully Bayesian
> meta-regression, which would allow for fully integrating the uncertainty
> from the model parameter estimates into the probability calculations by
> integrating over the joint posterior.
>
> Neither of these proposals is trivial, nor are they something that I would
> require if I were reviewing a manuscript with probability calculations like
> yours. Like I said, I think they're very helpful as is, especially if their
> limitations can be qualified appropriately.
>
> James
>
> On Fri, Aug 19, 2022 at 11:20 AM Yuhang Hu <yh342 using nau.edu> wrote:
>
>> Hello All,
>>
>> I have a 3-level meta-regression model with a categorical moderator (3
>> levels) plus some covariates fit as:
>>
>> rma.mv(yi ~ 0 + cat_mod + covariates, random = ~ 1 | study/effect)
>>
>> The means for cat_mod are estimated as: cat1: .27;  cat2 = .33, cat3 = .37
>>
>> The between-study standard deviation is .548, the within-study standard
>> deviation is .434, and the total variation in sd unit is .699.
>>
>> ***Question: I wonder whether I can say the following or not?***
>>
>> "The probability that effects with cat1 characteristic are 0 or larger is:
>> pnorm(0,.27, .699, lower.tail = FALSE)
>> > [1] 0.650
>>
>> with cat2 characteristic are 0 or larger is:
>> pnorm(0,.33, .699, lower.tail = FALSE)
>> > [1] 0.68
>>
>> with cat3 characteristic are 0 or larger is:
>> pnorm(0,.37, .699, lower.tail = FALSE)
>> > [1] 0.70
>> "
>> Thanks,
>> Yuhang
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

-- 
Yuhang Hu (She/Her/Hers)
Ph.D. Student in Applied Linguistics
Department of English
Northern Arizona University

	[[alternative HTML version deleted]]