[R-meta] Follow-up: Interpreting variance components in rma.mv

[Yuhang Hu]

Hello James,

Thank you very much for your confirmation.

I want to conclude from our discussion that for any additional level, in
fully uncorrelated random-effects models, we require adding 2 *
Var(additional level) to obtain the Var(gain) across any two time points.

For instance, if "random = ~ 1 | paper / study / effect", where Var(paper)
= zeta^2, then Var(gain) will be:

Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) + Var(z_2j*) + Var(z_1j*) = 2
* (omega^2 +  zeta^2)

Thanks again,
Yuhang

On Sat, Aug 27, 2022 at 12:35 PM James Pustejovsky <jepusto using gmail.com>
wrote:

> Hi Yuhang,
> Yes, according to the model that you've specified, you would need to add 2
> * omega^2 to calculate the variance of the gain from a new study.
> James
>
> On Fri, Aug 26, 2022 at 10:39 PM Yuhang Hu <yh342 using nau.edu> wrote:
>
>> Dear James,
>>
>> Thank you very much for your clear response.
>>
>> As another alternative, if I define my model as:
>>
>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = list(~ time |
>> study, ~1| effect), struc = "UN")
>>
>> Then, let's say we get the following var-covar matrix for the true
>> effects at our time points in each study averaged across all studies:
>>
>>          time0 time1
>> time0    0.6   0.2
>> time1    0.2   1.2
>>
>> with rho(time0, time1) = 1.00 (I know 1.00 is odd but in my case this is
>> fully identified)
>>
>> In addition, a typical study's own heterogeneity, lets say, is estimated
>> to be .13.
>>
>> If Gain (cat1) = 0.27; Gain (cat2) = 0.33, then Var(Gain) from time0 to
>> time1 at the study level will be:
>>
>> Var(Gain) = sqrt( Var(time0) + Var(time1) - 2 * rho(time0, time1) *
>> (sqrt(Var[time0]) * sqrt(Var[time1]))  )
>>
>> ***Question: So now should we add Var(effects) i.e., omega in your
>> notation, to the above equation or it needs to be further modified?
>>
>> Thank you,
>> Yuhang
>>
>> On Fri, Aug 26, 2022 at 7:38 AM James Pustejovsky <jepusto using gmail.com>
>> wrote:
>>
>>> Hi Yuhang,
>>>
>>> The probability calculations are not correct here because the SD you're
>>> using does not apply to gains. For the model you've specified:
>>> y_ij = b1 * Cat1_ij + b2 * Cat2_ij + b3 * Cat1_ij x Time1_ij + b4 *
>>> Cat2_ij x Time2_ij + u_j + v_ij + e_ij,
>>> where Var(u_j) = tau^2, Var(v_ij) = omega^2, and Var(e_ij) = V_ij (the
>>> known sampling variance).
>>>
>>> Now consider a new study j* that reports effects of type Cat1 at both
>>> time0 (i = 1) and time1 (i = 2), the true effect size parameters would be:
>>> y_1j* = b1 + u_j* + v_1j*
>>> y_2j* = b1 + b3 + u_j* + v_2j*
>>> and therefore the gain score would be
>>> y_2j* - y_1j* = b3 + v_2j* - v_1j*.
>>>
>>> Under the assumptions of your model,
>>> E(y_2j* - y_1j*) = b3
>>> Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) = 2 * omega^2,
>>> So you would need to calculate the prediction using an SD of sqrt(2) *
>>> omega.
>>>
>>> One thing to emphasize here is that these calculations hinge on the
>>> model being appropriately specified. If you've got the random effect
>>> structure wrong, then the probability calculation could be completely off.
>>>
>>> Another way to approach this prediction would be to do a meta-analysis
>>> of the gain scores directly (i.e., take the effect sizes from time-1 minus
>>> those from time-0 and use that in a basic random effects meta-analysis).
>>> You could then do the probability calculation in the usual way (as you did
>>> in your earlier post).
>>>
>>> James
>>>
>>> On Wed, Aug 24, 2022 at 10:50 PM Yuhang Hu <yh342 using nau.edu> wrote:
>>>
>>>> Hello All,
>>>>
>>>> I wanted to ask a follow-up on my previous post (
>>>>
>>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-August/004139.html
>>>> ).
>>>>
>>>> I'm currently fitting the following model (cat_mod = categorical mod):
>>>>
>>>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = ~ 1 |
>>>> study/effect)
>>>>
>>>> with a total heterogeneity in sd unit = 0.699.
>>>>
>>>> "cat_mod" levels' means at time0 are very different from each other. As
>>>> such, I have computed the gains (i.e., time1 - time0) for each level of
>>>> cat_mod:
>>>>
>>>> Gain (cat1) = 0.27
>>>> Gain (cat2) = 0.33
>>>>
>>>> ***Question: I wonder whether I can say the following or not?***
>>>>
>>>> "The probability that a gain from time0 to time1 in cat1 is 0 or larger
>>>> is:
>>>> pnorm(0,.27, .699, lower.tail = FALSE)
>>>> > [1] 0.650
>>>>
>>>> "The probability that a gain from time0 to time1 in cat2 is 0 or larger
>>>> is:
>>>> pnorm(0,.33, .699, lower.tail = FALSE)
>>>> > [1] 0.68
>>>>
>>>> Thank you for your attention.
>>>>
>>>> Yuhang Hu
>>>>
>>>>         [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>>> To manage your subscription to this mailing list, go to:
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>
>>>
>>
>> --
>> Yuhang Hu (She/Her/Hers)
>> Ph.D. Student in Applied Linguistics
>> Department of English
>> Northern Arizona University
>>
>

-- 
Yuhang Hu (She/Her/Hers)
Ph.D. Student in Applied Linguistics
Department of English
Northern Arizona University
-- 
Yuhang Hu (She/Her/Hers)
Ph.D. Student in Applied Linguistics
Department of English
Northern Arizona University

	[[alternative HTML version deleted]]