[R-meta] Follow-up: Interpreting variance components in rma.mv

[James Pustejovsky]

Hi Yuhang,

No, this isn't correct. Based on the model that you've specified, the
random effects at higher levels would drop out of the gain score
calculation. Only the effect-level variance is included.

But again, this is based on very strong modeling assumptions so you need to
be very careful that you've selected an appropriate random effects
specification.

James

On Sun, Aug 28, 2022 at 2:32 AM Yuhang Hu <yh342 using nau.edu> wrote:

> Hello James,
>
> Thank you very much for your confirmation.
>
> I want to conclude from our discussion that for any additional level, in
> fully uncorrelated random-effects models, we require adding 2 *
> Var(additional level) to obtain the Var(gain) across any two time points.
>
> For instance, if "random = ~ 1 | paper / study / effect", where Var(paper)
> = zeta^2, then Var(gain) will be:
>
> Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) + Var(z_2j*) + Var(z_1j*) = 2
> * (omega^2 +  zeta^2)
>
> Thanks again,
> Yuhang
>
> On Sat, Aug 27, 2022 at 12:35 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
>
>> Hi Yuhang,
>> Yes, according to the model that you've specified, you would need to add
>> 2 * omega^2 to calculate the variance of the gain from a new study.
>> James
>>
>> On Fri, Aug 26, 2022 at 10:39 PM Yuhang Hu <yh342 using nau.edu> wrote:
>>
>>> Dear James,
>>>
>>> Thank you very much for your clear response.
>>>
>>> As another alternative, if I define my model as:
>>>
>>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = list(~ time |
>>> study, ~1| effect), struc = "UN")
>>>
>>> Then, let's say we get the following var-covar matrix for the true
>>> effects at our time points in each study averaged across all studies:
>>>
>>>          time0 time1
>>> time0    0.6   0.2
>>> time1    0.2   1.2
>>>
>>> with rho(time0, time1) = 1.00 (I know 1.00 is odd but in my case this is
>>> fully identified)
>>>
>>> In addition, a typical study's own heterogeneity, lets say, is estimated
>>> to be .13.
>>>
>>> If Gain (cat1) = 0.27; Gain (cat2) = 0.33, then Var(Gain) from time0 to
>>> time1 at the study level will be:
>>>
>>> Var(Gain) = sqrt( Var(time0) + Var(time1) - 2 * rho(time0, time1) *
>>> (sqrt(Var[time0]) * sqrt(Var[time1]))  )
>>>
>>> ***Question: So now should we add Var(effects) i.e., omega in your
>>> notation, to the above equation or it needs to be further modified?
>>>
>>> Thank you,
>>> Yuhang
>>>
>>> On Fri, Aug 26, 2022 at 7:38 AM James Pustejovsky <jepusto using gmail.com>
>>> wrote:
>>>
>>>> Hi Yuhang,
>>>>
>>>> The probability calculations are not correct here because the SD you're
>>>> using does not apply to gains. For the model you've specified:
>>>> y_ij = b1 * Cat1_ij + b2 * Cat2_ij + b3 * Cat1_ij x Time1_ij + b4 *
>>>> Cat2_ij x Time2_ij + u_j + v_ij + e_ij,
>>>> where Var(u_j) = tau^2, Var(v_ij) = omega^2, and Var(e_ij) = V_ij (the
>>>> known sampling variance).
>>>>
>>>> Now consider a new study j* that reports effects of type Cat1 at both
>>>> time0 (i = 1) and time1 (i = 2), the true effect size parameters would be:
>>>> y_1j* = b1 + u_j* + v_1j*
>>>> y_2j* = b1 + b3 + u_j* + v_2j*
>>>> and therefore the gain score would be
>>>> y_2j* - y_1j* = b3 + v_2j* - v_1j*.
>>>>
>>>> Under the assumptions of your model,
>>>> E(y_2j* - y_1j*) = b3
>>>> Var(y_2j* - y_1j*) = Var(v_2j*) + Var(v_1j*) = 2 * omega^2,
>>>> So you would need to calculate the prediction using an SD of sqrt(2) *
>>>> omega.
>>>>
>>>> One thing to emphasize here is that these calculations hinge on the
>>>> model being appropriately specified. If you've got the random effect
>>>> structure wrong, then the probability calculation could be completely off.
>>>>
>>>> Another way to approach this prediction would be to do a meta-analysis
>>>> of the gain scores directly (i.e., take the effect sizes from time-1 minus
>>>> those from time-0 and use that in a basic random effects meta-analysis).
>>>> You could then do the probability calculation in the usual way (as you did
>>>> in your earlier post).
>>>>
>>>> James
>>>>
>>>> On Wed, Aug 24, 2022 at 10:50 PM Yuhang Hu <yh342 using nau.edu> wrote:
>>>>
>>>>> Hello All,
>>>>>
>>>>> I wanted to ask a follow-up on my previous post (
>>>>>
>>>>> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-August/004139.html
>>>>> ).
>>>>>
>>>>> I'm currently fitting the following model (cat_mod = categorical mod):
>>>>>
>>>>> rma.mv(yi ~ 0 + cat_mod * time + covariates, random = ~ 1 |
>>>>> study/effect)
>>>>>
>>>>> with a total heterogeneity in sd unit = 0.699.
>>>>>
>>>>> "cat_mod" levels' means at time0 are very different from each other. As
>>>>> such, I have computed the gains (i.e., time1 - time0) for each level of
>>>>> cat_mod:
>>>>>
>>>>> Gain (cat1) = 0.27
>>>>> Gain (cat2) = 0.33
>>>>>
>>>>> ***Question: I wonder whether I can say the following or not?***
>>>>>
>>>>> "The probability that a gain from time0 to time1 in cat1 is 0 or
>>>>> larger is:
>>>>> pnorm(0,.27, .699, lower.tail = FALSE)
>>>>> > [1] 0.650
>>>>>
>>>>> "The probability that a gain from time0 to time1 in cat2 is 0 or
>>>>> larger is:
>>>>> pnorm(0,.33, .699, lower.tail = FALSE)
>>>>> > [1] 0.68
>>>>>
>>>>> Thank you for your attention.
>>>>>
>>>>> Yuhang Hu
>>>>>
>>>>>         [[alternative HTML version deleted]]
>>>>>
>>>>> _______________________________________________
>>>>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>>>>> To manage your subscription to this mailing list, go to:
>>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>>
>>>>
>>>
>>> --
>>> Yuhang Hu (She/Her/Hers)
>>> Ph.D. Student in Applied Linguistics
>>> Department of English
>>> Northern Arizona University
>>>
>>
>
> --
> Yuhang Hu (She/Her/Hers)
> Ph.D. Student in Applied Linguistics
> Department of English
> Northern Arizona University
> --
> Yuhang Hu (She/Her/Hers)
> Ph.D. Student in Applied Linguistics
> Department of English
> Northern Arizona University
>

	[[alternative HTML version deleted]]