[R-meta] Confusion about how to use UN structure

[Luke Martinez]

Dear James,

I also got a bit confused about the second case (below). You mentioned
that in this case we have borrowing of strength from category A for
studies that have the A-C pairs. Could you please elaborate on how
this logic works?

I thought that each category can influence another category with which
it is correlated. For example, for the below case, (1): A and B can
influence each other, and also (2): A and C can influence one another.

By influence, I mean, for example, if A has occurred more frequently
than B in "the same studies" that B has also occurred, then, A can
influence B's average effect (and A's estimate of heterogeneity).

I appreciate it if you could clarify how we can have borrowing of
strength from category A for studies that have the A-C pairs in the
case below?

Thank you,
Luke

#[1,] "A" ~ "B"
#[2,] "A" ~ "C"
#[3,] "B" ~ "C" == 0

On Wed, Sep 1, 2021 at 8:45 PM James Pustejovsky <jepusto using gmail.com> wrote:
>
> Hi Simon,
>
> I'm sorry, my earlier email had an error in it. Revised version below,
> followed by comments on your cases.
>
> James
>
> CORRECTION:
>
> The choice of whether to fix a rho value to zero has implications for how
> the average effects for a given moderator category are estimated. Consider
> a scenario with a three-level moderator and with the slightly simpler
> meta-regression specification where you just have intercept terms for each
> moderator category (but no other predictors):
>  rma.mv(eff_size ~ UN_MOD -1, V = V, struct = "UN", random = ~ UN_MOD |
> study, data = data)
> If you set rho_AC and rho_BC to zero, then the average effect for category
> C is going to get estimated using only the effect size estimates from
> category C. The estimated average effect should be equivalent (up to
> numerical error) to the effect you get from a random effects meta-analysis
> using only the category C effects. The effect size estimates from the other
> two categories don't influence the results at all.
>
> On the other hand, if you set rho_AC to a non-zero value (or make the
> assumption that it is equivalent to a different correlation, such as
> rho_AB), then the model will "borrow strength" from the effect size
> estimates in category A for any study that has both A and C results, and so
> the average effect for category C will be influenced (at least a little
> bit) by the ES estimates in category A. The degree of influence depends on
> how many studies include categories A and C together, and on the size of
> the assumed rho_AC. If you set *all* of the correlations to zero (or
> equivalently, use struct = "DIAG"), then the average effect estimates are
> based only on the ES estimates for that category---that is, they're based
> only on the direct evidence, with no borrowing of strength across
> categories. If you were to estimate rho_AB but set rho_AC and rho_BC to
> zero, then you would get some borrowing of strength between categories A
> and B but none with category C. In my "expanding the range of working
> models" paper, we call the model where all correlations are zero the
> "subgroup correlated effects" working model because it is equivalent to
> running separate models on each category of the moderator.
>
> On Wed, Sep 1, 2021 at 8:15 PM Simon Harmel <sim.harmel using gmail.com> wrote:
>
> > Dear James,
> >
> > I also want to make sure I can correctly understand the directions in
> > your borrowing of strength description (see my summary below).
> >
> > 1- In CASE 1 (setting all pairwise correlations to 0 except for A ~
> > B), why "A" can't get strength from "B" even though A & B are
> > correlated?
> >
> > 2- In CASE 2, are you assuming "B" ~ "C" == 0 or not? How come studies
> > with "A & C" can get strength from "C"?
> >
> > Many thanks,
> > Simon
> >
> > #---CASE 1: No borrowing of strength for cat A, pure A effect
> > #[1,] "A" ~ "B"
> > #[2,] "A" ~ "C" == 0
> > #[3,] "B" ~ "C" == 0
> >
> > Here, there will be borrowing of strength between categories A and B, but
> category C will be estimated based on direct evidence only.
>
>
> > #---CASE 2: borrowing of strength from C for studies with A & C
> > #[1,] "A" ~ "B"
> > #[2,] "A" ~ "C" != 0
> > #[3,] "B" ~ "C" == 0
> >
> >
> Here, there will also be borrowing of strength between categories A and C.
>
>
> > #---CASE 3: borrowing of strength between cat A & B, but none with C
> > #[1,] "A" ~ "B" != 0
> > #[2,] "A" ~ "C" == 0
> > #[3,] "B" ~ "C" == 0
> >
> > I think this is the same as CASE 1.
>
>
> > On Wed, Sep 1, 2021 at 6:19 PM Simon Harmel <sim.harmel using gmail.com> wrote:
> > >
> > > Dear James,
> > >
> > > Thank you very much for sharing your expertise with me. I actually
> > > found much of what you kindly mentioned to be in line with this
> > > article (https://pubmed.ncbi.nlm.nih.gov/21268052/) which I read in
> > > preparation for my meta-analysis.
> > >
> > > As an alternative to HCS or CS, can I keep the UN structure but only
> > > focus on/interpret the correlation estimates that are based on
> > > sufficient pairs of levels? Or as Reza said (and I also confirmed that
> > > with my own data), because some rho estimates under the UN structure
> > > give flat curves for their profile likelihood function, then, my whole
> > > model is overparameterized and should be replaced with an
> > > appropriately parameterized model (e.g., using HCS or CS)?
> > >
> > > Once again, thank you for assistance,
> > > Simon
> > >
> > >
> > > On Wed, Sep 1, 2021 at 4:13 PM James Pustejovsky <jepusto using gmail.com>
> > wrote:
> > > >
> > > > Hi Simon,
> > > >
> > > > Like Wolfgang, I don't know of any simulation studies that look at
> > this situation, so it's hard to say what the best approach would be. But
> > I'll offer a few (somewhat speculative) observations:
> > > >
> > > > First, to your question about the meaning of the single rho returned
> > when using "HCS" or "CS", I would *not* characterize it as an average
> > correlation among all pairs of levels. It's the (restricted) maximum
> > likelihood estimate of the correlation between random effects for different
> > moderator categories, under whatever structure is assumed. Suppose there's
> > adequate data to estimate a correlation between categories A and B, but
> > little or no data about the correlations between categories A and C or
> > between B and C. Then we can't really say anything empirical about the AC
> > or BC correlations. All we can do is make assumptions. But perhaps HCS or
> > CS is reasonable as a very rough "working" model, which at least
> > acknowledges that there is dependence between effects drawn from different
> > moderator categories within the same study.
> > > >
> > > > Second, the choice of whether to fix a rho value to zero has
> > implications for how the average effects for a given moderator category are
> > estimated. Consider a scenario with a three-level moderator and with the
> > slightly simpler meta-regression specification where you just have
> > intercept terms for each moderator category (but no other predictors):
> > > >  rma.mv(eff_size ~ UN_MOD -1, V = V, struct = "UN", random = ~ UN_MOD
> > | study, data = data)
> > > > If you set rho_AC and rho_BC to zero, then the average effect for
> > category A is going to get estimated using only the effect size estimates
> > from category A. The estimated average effect should be equivalent (up to
> > numerical error) to the effect you get from a random effects meta-analysis
> > using only the category A effects. The effect size estimates from the other
> > two categories don't influence the results at all.
> > > >
> > > > On the other hand, if you set rho_AC to a non-zero value (or make the
> > assumption that it is equivalent to a different correlation, such as
> > rho_AB), then the model will "borrow strength" from the effect size
> > estimates in category C for any study that has both A and C results, and so
> > the average effect for category A will be influenced (at least a little
> > bit) by the ES estimates in category C. The degree of influence depends on
> > how many studies include categories A and C together, and on the size of
> > the assumed rho_AC. If you set *all* of the correlations to zero (or
> > equivalently, use struct = "DIAG"), then the average effect estimates are
> > based only on the ES estimates for that category---that is, they're based
> > only on the direct evidence, with no borrowing of strength across
> > categories. If you were to estimate rho_AB but set rho_AC and rho_BC to
> > zero, then you would get some borrowing of strength between categories A
> > and B but none with category C. In my "expanding the range of working
> > models" paper, we call the model where all correlations are zero the
> > "subgroup correlated effects" working model because it is equivalent to
> > running separate models on each category of the moderator.
> > > >
> > > > So to figure out how to proceed, you might consider whether the idea
> > of borrowing of strength across categories of the moderator makes practical
> > sense. If it does, then I would probably go with HCS or CS (paired with RVE
> > for insurance against mis-specification of the random effects structure).
> > If it seems like a weird thing to do, then I would go with the subgroup
> > correlated effects model that uses only the direct evidence. Of course, you
> > could also report both, which would have the benefit of allowing others to
> > judge the extent to which the findings are influenced by borrowing of
> > strength across categories.
> > > >
> > > > James
> > > >
> > > >
> > > > On Wed, Sep 1, 2021 at 3:41 PM Simon Harmel <sim.harmel using gmail.com>
> > wrote:
> > > >>
> > > >> Dear Wolfgang,
> > > >>
> > > >> Sure, thank you. So, if I alternatively use "HCS" or "CS", what can I
> > > >> say about the meaning of the single rho that I get, the average
> > > >> correlation among all pairs of the levels or something else?
> > > >>
> > > >> Also this single rho won't be negatively affected by the lack of
> > > >> co-occurrences on some pairs of the levels as was the case with "UN"?
> > > >>
> > > >> These will be very helpful to know, thanks a lot,
> > > >> Simon
> > > >>
> > > >> On Wed, Sep 1, 2021 at 2:48 PM Viechtbauer, Wolfgang (SP)
> > > >> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > > >> >
> > > >> > Dear Simon,
> > > >> >
> > > >> > I don't know and I am not aware of any simulation studies that
> > address this. Therefore, I cannot really make any recommendations.
> > > >> >
> > > >> > Best,
> > > >> > Wolfgang
> > > >> >
> > > >> > >-----Original Message-----
> > > >> > >From: Simon Harmel [mailto:sim.harmel using gmail.com]
> > > >> > >Sent: Wednesday, 01 September, 2021 21:19
> > > >> > >To: Viechtbauer, Wolfgang (SP)
> > > >> > >Cc: Reza Norouzian; R meta
> > > >> > >Subject: Re: [R-meta] Confusion about how to use UN structure
> > > >> > >
> > > >> > >Dear Wolfgang,
> > > >> > >
> > > >> > >To correct my question, is it acceptable to not estimate the
> > > >> > >correlations (rhos) that are based on too few pairs (i.e., fixing
> > them
> > > >> > >to 0) in a "UN" structure but at least estimate the correlations
> > that
> > > >> > >are based on a large number of pairs?
> > > >> > >
> > > >> > >Or in such a situation, one has to use a simpler structure like
> > "CS"
> > > >> > >or "HCS" and obtain a single rho estimate representing the
> > correlation
> > > >> > >among all the pairs of the levels? (Maybe in this case, such a rho
> > > >> > >will represent the average correlation among all pairs of the
> > levels,
> > > >> > >right?).
> > > >> > >
> > > >> > >This is exactly the situation I'm in now.
> > > >> > >
> > > >> > >Thank you very much,
> > > >> > >Simon
> > > >>
> > > >> _______________________________________________
> > > >> R-sig-meta-analysis mailing list
> > > >> R-sig-meta-analysis using r-project.org
> > > >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis