[R-meta] Confusion about how to use UN structure

Thu Sep 2 04:33:11 CEST 2021

Dear James,

Thank you very much. This now makes logical sense. But this brings up
another issue. Given that HCS and CS both allow for one common
correlation among categories in each study (assuming UN_MOD | study,
struct = "HCS"|"CS"), then, how "the borrowing of strength" in HCS and
CS differs from that in the UN?

For example, it seems to me that in HCS and CS, borrowing of strength
is kind of malleable (a lump "the borrowing of strength") as opposed
to the directional "the borrowing of strength" in UN.

If so, then in my paper, (assuming I'm not allowed to use an
overparameterized UN structure but only focus on its reliable rho
estimates), I will probably describe/interpret the the common rho
estimate given by HCS or CS as some kind of "assumptional" quantity,
one that in the case of theoretically correlated categories honors the
theory. But one that can't allow examining the exact degree of
correlations among average effects associated with each category
(i.e., can't substantiate the theory in its full form).

I remember asking this a while back that does random effect
specification have any bearing on the fixed effects in multilevel
meta-regression models? It seems I am indirectly formulating an answer
to that question as well!

Thank you,
Simon

On Wed, Sep 1, 2021 at 8:45 PM James Pustejovsky <jepusto using gmail.com> wrote:
>
> Hi Simon,
>
> I'm sorry, my earlier email had an error in it. Revised version below, followed by comments on your cases.
>
> James
>
> CORRECTION:
>
> The choice of whether to fix a rho value to zero has implications for how the average effects for a given moderator category are estimated. Consider a scenario with a three-level moderator and with the slightly simpler meta-regression specification where you just have intercept terms for each moderator category (but no other predictors):
>  rma.mv(eff_size ~ UN_MOD -1, V = V, struct = "UN", random = ~ UN_MOD | study, data = data)
> If you set rho_AC and rho_BC to zero, then the average effect for category C is going to get estimated using only the effect size estimates from category C. The estimated average effect should be equivalent (up to numerical error) to the effect you get from a random effects meta-analysis using only the category C effects. The effect size estimates from the other two categories don't influence the results at all.
>
> On the other hand, if you set rho_AC to a non-zero value (or make the assumption that it is equivalent to a different correlation, such as rho_AB), then the model will "borrow strength" from the effect size estimates in category A for any study that has both A and C results, and so the average effect for category C will be influenced (at least a little bit) by the ES estimates in category A. The degree of influence depends on how many studies include categories A and C together, and on the size of the assumed rho_AC. If you set *all* of the correlations to zero (or equivalently, use struct = "DIAG"), then the average effect estimates are based only on the ES estimates for that category---that is, they're based only on the direct evidence, with no borrowing of strength across categories. If you were to estimate rho_AB but set rho_AC and rho_BC to zero, then you would get some borrowing of strength between categories A and B but none with category C. In my "expanding the range of working models" paper, we call the model where all correlations are zero the "subgroup correlated effects" working model because it is equivalent to running separate models on each category of the moderator.
>
> On Wed, Sep 1, 2021 at 8:15 PM Simon Harmel <sim.harmel using gmail.com> wrote:
>>
>> Dear James,
>>
>> I also want to make sure I can correctly understand the directions in
>> your borrowing of strength description (see my summary below).
>>
>> 1- In CASE 1 (setting all pairwise correlations to 0 except for A ~
>> B), why "A" can't get strength from "B" even though A & B are
>> correlated?
>>
>> 2- In CASE 2, are you assuming "B" ~ "C" == 0 or not? How come studies
>> with "A & C" can get strength from "C"?
>>
>> Many thanks,
>> Simon
>>
>> #---CASE 1: No borrowing of strength for cat A, pure A effect
>> #[1,] "A" ~ "B"
>> #[2,] "A" ~ "C" == 0
>> #[3,] "B" ~ "C" == 0
>>
> Here, there will be borrowing of strength between categories A and B, but category C will be estimated based on direct evidence only.
>
>>
>> #---CASE 2: borrowing of strength from C for studies with A & C
>> #[1,] "A" ~ "B"
>> #[2,] "A" ~ "C" != 0
>> #[3,] "B" ~ "C" == 0
>>
>
> Here, there will also be borrowing of strength between categories A and C.
>
>>
>> #---CASE 3: borrowing of strength between cat A & B, but none with C
>> #[1,] "A" ~ "B" != 0
>> #[2,] "A" ~ "C" == 0
>> #[3,] "B" ~ "C" == 0
>>
> I think this is the same as CASE 1.
>
>>
>> On Wed, Sep 1, 2021 at 6:19 PM Simon Harmel <sim.harmel using gmail.com> wrote:
>> >
>> > Dear James,
>> >
>> > Thank you very much for sharing your expertise with me. I actually
>> > found much of what you kindly mentioned to be in line with this
>> > article (https://pubmed.ncbi.nlm.nih.gov/21268052/) which I read in
>> > preparation for my meta-analysis.
>> >
>> > As an alternative to HCS or CS, can I keep the UN structure but only
>> > focus on/interpret the correlation estimates that are based on
>> > sufficient pairs of levels? Or as Reza said (and I also confirmed that
>> > with my own data), because some rho estimates under the UN structure
>> > give flat curves for their profile likelihood function, then, my whole
>> > model is overparameterized and should be replaced with an
>> > appropriately parameterized model (e.g., using HCS or CS)?
>> >
>> > Once again, thank you for assistance,
>> > Simon
>> >
>> >
>> > On Wed, Sep 1, 2021 at 4:13 PM James Pustejovsky <jepusto using gmail.com> wrote:
>> > >
>> > > Hi Simon,
>> > >
>> > > Like Wolfgang, I don't know of any simulation studies that look at this situation, so it's hard to say what the best approach would be. But I'll offer a few (somewhat speculative) observations:
>> > >
>> > > First, to your question about the meaning of the single rho returned when using "HCS" or "CS", I would *not* characterize it as an average correlation among all pairs of levels. It's the (restricted) maximum likelihood estimate of the correlation between random effects for different moderator categories, under whatever structure is assumed. Suppose there's adequate data to estimate a correlation between categories A and B, but little or no data about the correlations between categories A and C or between B and C. Then we can't really say anything empirical about the AC or BC correlations. All we can do is make assumptions. But perhaps HCS or CS is reasonable as a very rough "working" model, which at least acknowledges that there is dependence between effects drawn from different moderator categories within the same study.
>> > >
>> > > Second, the choice of whether to fix a rho value to zero has implications for how the average effects for a given moderator category are estimated. Consider a scenario with a three-level moderator and with the slightly simpler meta-regression specification where you just have intercept terms for each moderator category (but no other predictors):
>> > >  rma.mv(eff_size ~ UN_MOD -1, V = V, struct = "UN", random = ~ UN_MOD | study, data = data)
>> > > If you set rho_AC and rho_BC to zero, then the average effect for category A is going to get estimated using only the effect size estimates from category A. The estimated average effect should be equivalent (up to numerical error) to the effect you get from a random effects meta-analysis using only the category A effects. The effect size estimates from the other two categories don't influence the results at all.
>> > >
>> > > On the other hand, if you set rho_AC to a non-zero value (or make the assumption that it is equivalent to a different correlation, such as rho_AB), then the model will "borrow strength" from the effect size estimates in category C for any study that has both A and C results, and so the average effect for category A will be influenced (at least a little bit) by the ES estimates in category C. The degree of influence depends on how many studies include categories A and C together, and on the size of the assumed rho_AC. If you set *all* of the correlations to zero (or equivalently, use struct = "DIAG"), then the average effect estimates are based only on the ES estimates for that category---that is, they're based only on the direct evidence, with no borrowing of strength across categories. If you were to estimate rho_AB but set rho_AC and rho_BC to zero, then you would get some borrowing of strength between categories A and B but none with category C. In my "expanding the range of working models" paper, we call the model where all correlations are zero the "subgroup correlated effects" working model because it is equivalent to running separate models on each category of the moderator.
>> > >
>> > > So to figure out how to proceed, you might consider whether the idea of borrowing of strength across categories of the moderator makes practical sense. If it does, then I would probably go with HCS or CS (paired with RVE for insurance against mis-specification of the random effects structure). If it seems like a weird thing to do, then I would go with the subgroup correlated effects model that uses only the direct evidence. Of course, you could also report both, which would have the benefit of allowing others to judge the extent to which the findings are influenced by borrowing of strength across categories.
>> > >
>> > > James
>> > >
>> > >
>> > > On Wed, Sep 1, 2021 at 3:41 PM Simon Harmel <sim.harmel using gmail.com> wrote:
>> > >>
>> > >> Dear Wolfgang,
>> > >>
>> > >> Sure, thank you. So, if I alternatively use "HCS" or "CS", what can I
>> > >> say about the meaning of the single rho that I get, the average
>> > >> correlation among all pairs of the levels or something else?
>> > >>
>> > >> Also this single rho won't be negatively affected by the lack of
>> > >> co-occurrences on some pairs of the levels as was the case with "UN"?
>> > >>
>> > >> These will be very helpful to know, thanks a lot,
>> > >> Simon
>> > >>
>> > >> On Wed, Sep 1, 2021 at 2:48 PM Viechtbauer, Wolfgang (SP)
>> > >> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> > >> >
>> > >> > Dear Simon,
>> > >> >
>> > >> > I don't know and I am not aware of any simulation studies that address this. Therefore, I cannot really make any recommendations.
>> > >> >
>> > >> > Best,
>> > >> > Wolfgang
>> > >> >
>> > >> > >-----Original Message-----
>> > >> > >From: Simon Harmel [mailto:sim.harmel using gmail.com]
>> > >> > >Sent: Wednesday, 01 September, 2021 21:19
>> > >> > >To: Viechtbauer, Wolfgang (SP)
>> > >> > >Cc: Reza Norouzian; R meta
>> > >> > >Subject: Re: [R-meta] Confusion about how to use UN structure
>> > >> > >
>> > >> > >Dear Wolfgang,
>> > >> > >
>> > >> > >To correct my question, is it acceptable to not estimate the
>> > >> > >correlations (rhos) that are based on too few pairs (i.e., fixing them
>> > >> > >to 0) in a "UN" structure but at least estimate the correlations that
>> > >> > >are based on a large number of pairs?
>> > >> > >
>> > >> > >Or in such a situation, one has to use a simpler structure like "CS"
>> > >> > >or "HCS" and obtain a single rho estimate representing the correlation
>> > >> > >among all the pairs of the levels? (Maybe in this case, such a rho
>> > >> > >will represent the average correlation among all pairs of the levels,
>> > >> > >right?).
>> > >> > >
>> > >> > >This is exactly the situation I'm in now.
>> > >> > >
>> > >> > >Thank you very much,
>> > >> > >Simon
>> > >>
>> > >> _______________________________________________
>> > >> R-sig-meta-analysis mailing list
>> > >> R-sig-meta-analysis using r-project.org
>> > >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis