[R-meta] Follow-up: Interpreting variance components in rma.mv

[Yuhang Hu]

Dear James,

Thank you very much for your clear response.

As another alternative, if I define my model as:

rma.mv(yi ~ 0 + cat_mod * time + covariates, random = list(~ time | study,
~1| effect), struc = "UN")

Then, let's say we get the following var-covar matrix for the true effects
at our time points in each study averaged across all studies:

         time0 time1
time0    0.6   0.2
time1    0.2   1.2

with rho(time0, time1) = 1.00 (I know 1.00 is odd but in my case this is
fully identified)

In addition, a typical study's own heterogeneity, lets say, is estimated to
be .13.

If Gain (cat1) = 0.27; Gain (cat2) = 0.33, then Var(Gain) from time0 to
time1 at the study level will be:

Var(Gain) = sqrt( Var(time0) + Var(time1) - 2 * rho(time0, time1) *
(sqrt(Var[time0]) * sqrt(Var[time1]))  )

***Question: So now should we add Var(effects) i.e., omega in your
notation, to the above equation or it needs to be further modified?

Thank you,
Yuhang

On Fri, Aug 26, 2022 at 7:38 AM James Pustejovsky <jepusto using gmail.com> wrote:

> Hi Yuhang,
>
> The probability calculations are not correct here because the SD you're
> using does not apply to gains. For the model you've specified:
> y_ij = b1 * Cat1_ij + b2 * Cat2_ij + b3 * Cat1_ij x Time1_ij + b4 *
> Cat2_ij x Time2_ij + u_j + v_ij + e_ij,
> where Var(u_j) = tau^2, Var(v_ij) = omega^2, and Var(e_ij) = V_ij (the
> known sampling variance).
>
> Now consider a new study j* that reports effects of type Cat1 at both
> time0 (i = 1) and time1 (i = 2), the true effect size parameters would be:
> y_1j* = b1 + u_j* + v_1j*
> y_2j* = b1 + b3 + u_j* + v_2j*
> and therefore the gain score would be
> y_2j* - y_1j* = b3 + v_2j* - v_1j*.
>
> Under the assumptions of your model,
> E(y_2j* - y_1j*) = b3
> Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) = 2 * omega^2,
> So you would need to calculate the prediction using an SD of sqrt(2) *
> omega.
>
> One thing to emphasize here is that these calculations hinge on the model
> being appropriately specified. If you've got the random effect structure
> wrong, then the probability calculation could be completely off.
>
> Another way to approach this prediction would be to do a meta-analysis of
> the gain scores directly (i.e., take the effect sizes from time-1 minus
> those from time-0 and use that in a basic random effects meta-analysis).
> You could then do the probability calculation in the usual way (as you did
> in your earlier post).
>
> James
>
> On Wed, Aug 24, 2022 at 10:50 PM Yuhang Hu <yh342 using nau.edu> wrote:
>
>> Hello All,
>>
>> I wanted to ask a follow-up on my previous post (
>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-August/004139.html
>> ).
>>
>> I'm currently fitting the following model (cat_mod = categorical mod):
>>
>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = ~ 1 | study/effect)
>>
>> with a total heterogeneity in sd unit = 0.699.
>>
>> "cat_mod" levels' means at time0 are very different from each other. As
>> such, I have computed the gains (i.e., time1 - time0) for each level of
>> cat_mod:
>>
>> Gain (cat1) = 0.27
>> Gain (cat2) = 0.33
>>
>> ***Question: I wonder whether I can say the following or not?***
>>
>> "The probability that a gain from time0 to time1 in cat1 is 0 or larger
>> is:
>> pnorm(0,.27, .699, lower.tail = FALSE)
>> > [1] 0.650
>>
>> "The probability that a gain from time0 to time1 in cat2 is 0 or larger
>> is:
>> pnorm(0,.33, .699, lower.tail = FALSE)
>> > [1] 0.68
>>
>> Thank you for your attention.
>>
>> Yuhang Hu
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

-- 
Yuhang Hu (She/Her/Hers)
Ph.D. Student in Applied Linguistics
Department of English
Northern Arizona University

	[[alternative HTML version deleted]]