[R-meta] AICc or variance components, which one matters more?

[Luke Martinez]

Wolfgang, the results of the toy data happened to resemble what I saw
in the full data.

I appreciate very much your highly valuable input,
Luke

On Mon, Nov 15, 2021 at 2:44 PM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> I thought this toy dataset was just for discussing principles. I would not consider most, if any, of these models and would not use ICs to compare them *for this toy dataset*.
>
> My goal in this discussion was to point out a general principle, namely that g1 vs g2 is a false dichotomy and that one could also use g3 (and in the end point out that one could even entertain more complex models given the general data structure).
>
> But as I mentioned a number of posts ago: If there really is only a single case where two labs were involved in the same study, then I might be inclined to just ignore the issue. Knowing nothing more of these data and assuming that there really is a larger dataset and that the dataset you posted is a small subset thereof, I would go with ~1|lab/study/es_id and call it a day.
>
> I think at this point the discussion is going a bit in circles and I will bow out of it.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Luke Martinez [mailto:martinezlukerm using gmail.com]
> >Sent: Monday, 15 November, 2021 21:10
> >To: Viechtbauer, Wolfgang (SP)
> >Cc: R meta
> >Subject: Re: [R-meta] AICc or variance components, which one matters more?
> >
> >correction:
> >
> >You proposed g3 to solve this impasse, but g3 gives a higher AICc than
> >that of g2.
> >
> >On Mon, Nov 15, 2021 at 2:06 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
> >>
> >> Hi Wolfgang,
> >>
> >> Thank you! If we go by your ICC principle, then since almost all
> >> studies uniquely belong to one lab (except one exception), then ~ 1 |
> >> lab/study (as in g1) should prevail over list(~1|lab, ~1|study) [as in
> >> g2].
> >>
> >> But the thing is that AICc doesn't agree with this. That's exactly
> >> where I get stuck in preferring one model over the other. Model g1
> >> matches the data structure better, but g2 has a smaller AICc?
> >>
> >> You proposed g3 to solve this impasse, but g3 gives the same AICc as that of
> >g2.
> >>
> >> Moreover, we have not yet added any moderators. If we do, then using
> >> g2 or g3 with already 0 variance components would mean that such
> >> zero-variance components really don't do much in the model. And that
> >> was why I thought specifying random effects as in g1 which gives
> >> non-zero variance components seems like a better use of the random
> >> effects especially as we add moderators.
> >>
> >>
> >> (g1=rma.mvyi, vi, random = ~1|lab/study, data = dd))
> >> (g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))
> >> (g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))
> >>
> >> fitstats(g1,g2,g3)[5,]
> >>
> >>              g1            g2           g3
> >> AICc: 30.85992 29.73897 35.73897
> >>
> >> On Mon, Nov 15, 2021 at 1:14 PM Viechtbauer, Wolfgang (SP)
> >> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
> >> >
> >> > The various models allow for different correlation structures for the
> >underlying true effects. For example, the model with only ~1|lab/study implies an
> >ICC of sigma^2_lab / (sigma^2_lab + sigma^2_study) for true effects belonging to
> >different studies within the same lab and does not allow true effects to be
> >correlated across labs even if these labs were involved in the same study. If one
> >wants to account for the latter, one can add ~ 1 | study and this doesn't
> >automatically have to replace the /study part in ~1|lab/study.
> >> >
> >> > In the original data structure you showed, there were at times multiple rows
> >for the same study within the same lab. One could then even go further and use
> >~1|lab/study/es_id because without this, the ~1|lab/study model implies an ICC of
> >1 for true effects belonging to the same study within the same lab. So one could
> >even entertain the model:
> >> >
> >> > (g5=rma.mv(yi, vi, random = ~1|lab/study/es_id, data = dd))
> >> >
> >> > and then again:
> >> >
> >> > (g6=rma.mv(yi, vi, random = list(~1|lab/study/es_id, ~ 1 | study), data =
> >dd))
> >> >
> >> > Of course this is all silly with the toy dataset, but even there all variance
> >components are identifiable.
> >> >
> >> > Ultimately, fully understanding these models requires writing out what they
> >imply about the ICC for different combinations of lab, study, and es_id (e.g.,
> >same lab and same study and different es_id, same lab and different studies,
> >different labs and same study, and so on). As any good stats book would say at
> >this point: I leave this as an exercise to the reader.
> >> >
> >> > Best,
> >> > Wolfgang