[R-meta] AICc or variance components, which one matters more?

[Luke Martinez]

correction:

You proposed g3 to solve this impasse, but g3 gives a higher AICc than
that of g2.

On Mon, Nov 15, 2021 at 2:06 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
>
> Hi Wolfgang,
>
> Thank you! If we go by your ICC principle, then since almost all
> studies uniquely belong to one lab (except one exception), then ~ 1 |
> lab/study (as in g1) should prevail over list(~1|lab, ~1|study) [as in
> g2].
>
> But the thing is that AICc doesn't agree with this. That's exactly
> where I get stuck in preferring one model over the other. Model g1
> matches the data structure better, but g2 has a smaller AICc?
>
> You proposed g3 to solve this impasse, but g3 gives the same AICc as that of g2.
>
> Moreover, we have not yet added any moderators. If we do, then using
> g2 or g3 with already 0 variance components would mean that such
> zero-variance components really don't do much in the model. And that
> was why I thought specifying random effects as in g1 which gives
> non-zero variance components seems like a better use of the random
> effects especially as we add moderators.
>
>
> (g1=rma.mv(yi, vi, random = ~1|lab/study, data = dd))
> (g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))
> (g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))
>
> fitstats(g1,g2,g3)[5,]
>
>              g1            g2           g3
> AICc: 30.85992 29.73897 35.73897
>
> On Mon, Nov 15, 2021 at 1:14 PM Viechtbauer, Wolfgang (SP)
> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >
> > The various models allow for different correlation structures for the underlying true effects. For example, the model with only ~1|lab/study implies an ICC of sigma^2_lab / (sigma^2_lab + sigma^2_study) for true effects belonging to different studies within the same lab and does not allow true effects to be correlated across labs even if these labs were involved in the same study. If one wants to account for the latter, one can add ~ 1 | study and this doesn't automatically have to replace the /study part in ~1|lab/study.
> >
> > In the original data structure you showed, there were at times multiple rows for the same study within the same lab. One could then even go further and use ~1|lab/study/es_id because without this, the ~1|lab/study model implies an ICC of 1 for true effects belonging to the same study within the same lab. So one could even entertain the model:
> >
> > (g5=rma.mv(yi, vi, random = ~1|lab/study/es_id, data = dd))
> >
> > and then again:
> >
> > (g6=rma.mv(yi, vi, random = list(~1|lab/study/es_id, ~ 1 | study), data = dd))
> >
> > Of course this is all silly with the toy dataset, but even there all variance components are identifiable.
> >
> > Ultimately, fully understanding these models requires writing out what they imply about the ICC for different combinations of lab, study, and es_id (e.g., same lab and same study and different es_id, same lab and different studies, different labs and same study, and so on). As any good stats book would say at this point: I leave this as an exercise to the reader.
> >
> > Best,
> > Wolfgang
> >
> > >-----Original Message-----
> > >From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> > >Sent: Monday, 15 November, 2021 19:32
> > >To: Viechtbauer, Wolfgang (SP)
> > >Cc: R meta
> > >Subject: Re: [R-meta] AICc or variance components, which one matters more?
> > >
> > >ATTACHMENT(S) REMOVED: g4.PNG
> > >
> > >Ok, I see what you mean regarding g4. "lab" is just repetitive
> > >(profile curves for two repetitive lab var. components are mainly
> > >flat). Ok g4 is out.
> > >
> > > But I still wonder about the thinking behind g3 where 'study' can be
> > >both nested and at the same time crossed?
> > >
> > >(g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))
> > >
> > >On Mon, Nov 15, 2021 at 12:07 PM Viechtbauer, Wolfgang (SP)
> > ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >>
> > >> Please post the profile likelihood plots for the three variance components of
> > >model g4.
> > >>
> > >> >-----Original Message-----
> > >> >From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> > >> >Sent: Monday, 15 November, 2021 18:37
> > >> >To: Viechtbauer, Wolfgang (SP)
> > >> >Cc: R meta
> > >> >Subject: Re: [R-meta] AICc or variance components, which one matters more?
> > >> >
> > >> >In g4, also, all variance components are identifiable (and larger in
> > >magnitude).
> > >> >
> > >> >But what's the thinking behind either g3 or g4 where the same grouping
> > >variable
> > >> >is both nested and crossed?
> > >> >
> > >> >On Mon, Nov 15, 2021, 11:33 AM Viechtbauer, Wolfgang (SP)
> > >> ><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >> >In g3, all variance components are identifiable.
> > >> >
> > >> >What have you found out about g4?
> > >> >
> > >> >>-----Original Message-----
> > >> >>From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> > >> >>Sent: Monday, 15 November, 2021 18:28
> > >> >>To: Viechtbauer, Wolfgang (SP)
> > >> >>Cc: R meta
> > >> >>Subject: Re: [R-meta] AICc or variance components, which one matters more?
> > >> >>
> > >> >>Hi Wolfgang,
> > >> >>
> > >> >>Could you possibly elaborate a bit on "in this case, yes"?
> > >> >>
> > >> >>This allows me to better justify g3 or g4 models to my co-authors.
> > >> >>
> > >> >>Thanks again,
> > >> >>Luke
> > >> >>
> > >> >>On Mon, Nov 15, 2021, 11:19 AM Viechtbauer, Wolfgang (SP)
> > >> >><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >> >>>-----Original Message-----
> > >> >>>From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> > >> >>>Sent: Saturday, 13 November, 2021 21:10
> > >> >>>To: Viechtbauer, Wolfgang (SP)
> > >> >>>Cc: R meta
> > >> >>>Subject: Re: [R-meta] AICc or variance components, which one matters more?
> > >> >>>
> > >> >>>Interesting! To make sure I'm following you, your suggested g3 model
> > >> >>>both considers 'study' to be nested in the 'lab', and at the same time
> > >> >>>it considers 'study' to have its own independent crossed effect. Can
> > >> >>>we consider the same variable (e.g., study) to be both nested and
> > >> >>>crossed at the same time?
> > >> >>
> > >> >>In this case, yes.
> > >> >>
> > >> >>>If so, I can then suggest the following model as well:
> > >> >>>
> > >> >>>(g4=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | lab), data = dd))
> > >> >>
> > >> >>I suggest you profile the variance components from that model and draw
> > >> >>appropriate conclusions.
> > >> >>
> > >> >>>Doesn't this denote that one is uncertain about whether to take a
> > >> >>>variable as nested or crossed or there are other justifications?
> > >> >>>
> > >> >>>Thank you,
> > >> >>>Luke
> > >> >>>
> > >> >>>On Sat, Nov 13, 2021 at 1:11 PM Viechtbauer, Wolfgang (SP)
> > >> >>><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> > >> >>>>
> > >> >>>> >-----Original Message-----
> > >> >>>> >From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> > >> >>>> >Sent: Saturday, 13 November, 2021 18:44
> > >> >>>> >To: Viechtbauer, Wolfgang (SP)
> > >> >>>> >Cc: Philippe Tadger; R meta
> > >> >>>> >Subject: Re: [R-meta] AICc or variance components, which one matters
> > >more?
> > >> >>>> >
> > >> >>>> >Hi Wolfgang,
> > >> >>>> >
> > >> >>>> >I'm fully with you, however, in my data only once 2 labs (labs 1 and
> > >> >>>> >2) have collaborated on study 2. Specifically, part of study 2 has
> > >> >>>> >been carried out by lab 1 (one row) and part of it by lab 2 (one row).
> > >> >>>> >Except in this case, no such between-lab collaborations have ever
> > >> >>>> >occurred in the data.
> > >> >>>> >
> > >> >>>> >If such a between-lab collaboration didn't exist, I could directly go
> > >> >>>> >for g1 (hierarchical model). But with this collaboration, there is
> > >> >>>> >just a tiny possibility for g2 (crossed model) as well.
> > >> >>>> >
> > >> >>>> >So, do you think AICc should be the basis of the comparison between g1
> > >> >>>> >vs. g2 or the dominant data structure (ignoring the one exception)?
> > >> >>>>
> > >> >>>> Using information criteria *could* be the basis. But I might be inclined
> > >to
> > >> >>>just ignore the issue you describe above though if this only affects one
> > >study.
> > >> >>>>
> > >> >>>> Just as a note: It's not necessarily an either-or choice. This model is
> > >also
> > >> >>>possible:
> > >> >>>>
> > >> >>>> (g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))
> > >> >>>>
> > >> >>>> and profile(g3) suggests that all variance components are identifiable -
> > >> >>>although of course this is quite overfitted with so little data.
> > >> >>>>
> > >> >>>> >(g1=rma.mv(yi, vi, random = ~1|lab/study, data = dd))
> > >> >>>> >(g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))
> > >> >>>> >
> > >> >>>> >Thanks,
> > >> >>>> >Luke