[R-meta] Follow-up: Interpreting variance components in rma.mv

[James Pustejovsky]

Hi Yuhang,
Yes, according to the model that you've specified, you would need to add 2
* omega^2 to calculate the variance of the gain from a new study.
James

On Fri, Aug 26, 2022 at 10:39 PM Yuhang Hu <yh342 using nau.edu> wrote:

> Dear James,
>
> Thank you very much for your clear response.
>
> As another alternative, if I define my model as:
>
> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = list(~ time |
> study, ~1| effect), struc = "UN")
>
> Then, let's say we get the following var-covar matrix for the true effects
> at our time points in each study averaged across all studies:
>
>          time0 time1
> time0    0.6   0.2
> time1    0.2   1.2
>
> with rho(time0, time1) = 1.00 (I know 1.00 is odd but in my case this is
> fully identified)
>
> In addition, a typical study's own heterogeneity, lets say, is estimated
> to be .13.
>
> If Gain (cat1) = 0.27; Gain (cat2) = 0.33, then Var(Gain) from time0 to
> time1 at the study level will be:
>
> Var(Gain) = sqrt( Var(time0) + Var(time1) - 2 * rho(time0, time1) *
> (sqrt(Var[time0]) * sqrt(Var[time1]))  )
>
> ***Question: So now should we add Var(effects) i.e., omega in your
> notation, to the above equation or it needs to be further modified?
>
> Thank you,
> Yuhang
>
> On Fri, Aug 26, 2022 at 7:38 AM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Yuhang,
>>
>> The probability calculations are not correct here because the SD you're
>> using does not apply to gains. For the model you've specified:
>> y_ij = b1 * Cat1_ij + b2 * Cat2_ij + b3 * Cat1_ij x Time1_ij + b4 *
>> Cat2_ij x Time2_ij + u_j + v_ij + e_ij,
>> where Var(u_j) = tau^2, Var(v_ij) = omega^2, and Var(e_ij) = V_ij (the
>> known sampling variance).
>>
>> Now consider a new study j* that reports effects of type Cat1 at both
>> time0 (i = 1) and time1 (i = 2), the true effect size parameters would be:
>> y_1j* = b1 + u_j* + v_1j*
>> y_2j* = b1 + b3 + u_j* + v_2j*
>> and therefore the gain score would be
>> y_2j* - y_1j* = b3 + v_2j* - v_1j*.
>>
>> Under the assumptions of your model,
>> E(y_2j* - y_1j*) = b3
>> Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) = 2 * omega^2,
>> So you would need to calculate the prediction using an SD of sqrt(2) *
>> omega.
>>
>> One thing to emphasize here is that these calculations hinge on the model
>> being appropriately specified. If you've got the random effect structure
>> wrong, then the probability calculation could be completely off.
>>
>> Another way to approach this prediction would be to do a meta-analysis of
>> the gain scores directly (i.e., take the effect sizes from time-1 minus
>> those from time-0 and use that in a basic random effects meta-analysis).
>> You could then do the probability calculation in the usual way (as you did
>> in your earlier post).
>>
>> James
>>
>> On Wed, Aug 24, 2022 at 10:50 PM Yuhang Hu <yh342 using nau.edu> wrote:
>>
>>> Hello All,
>>>
>>> I wanted to ask a follow-up on my previous post (
>>>
>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-August/004139.html
>>> ).
>>>
>>> I'm currently fitting the following model (cat_mod = categorical mod):
>>>
>>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = ~ 1 |
>>> study/effect)
>>>
>>> with a total heterogeneity in sd unit = 0.699.
>>>
>>> "cat_mod" levels' means at time0 are very different from each other. As
>>> such, I have computed the gains (i.e., time1 - time0) for each level of
>>> cat_mod:
>>>
>>> Gain (cat1) = 0.27
>>> Gain (cat2) = 0.33
>>>
>>> ***Question: I wonder whether I can say the following or not?***
>>>
>>> "The probability that a gain from time0 to time1 in cat1 is 0 or larger
>>> is:
>>> pnorm(0,.27, .699, lower.tail = FALSE)
>>> > [1] 0.650
>>>
>>> "The probability that a gain from time0 to time1 in cat2 is 0 or larger
>>> is:
>>> pnorm(0,.33, .699, lower.tail = FALSE)
>>> > [1] 0.68
>>>
>>> Thank you for your attention.
>>>
>>> Yuhang Hu
>>>
>>>         [[alternative HTML version deleted]]
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>> To manage your subscription to this mailing list, go to:
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>
>>
>
> --
> Yuhang Hu (She/Her/Hers)
> Ph.D. Student in Applied Linguistics
> Department of English
> Northern Arizona University
>

	[[alternative HTML version deleted]]