[R-meta] AICc or variance components, which one matters more?

[Viechtbauer, Wolfgang (SP)]

I thought this toy dataset was just for discussing principles. I would not consider most, if any, of these models and would not use ICs to compare them *for this toy dataset*.

My goal in this discussion was to point out a general principle, namely that g1 vs g2 is a false dichotomy and that one could also use g3 (and in the end point out that one could even entertain more complex models given the general data structure).

But as I mentioned a number of posts ago: If there really is only a single case where two labs were involved in the same study, then I might be inclined to just ignore the issue. Knowing nothing more of these data and assuming that there really is a larger dataset and that the dataset you posted is a small subset thereof, I would go with ~1|lab/study/es_id and call it a day.

I think at this point the discussion is going a bit in circles and I will bow out of it.

Best,
Wolfgang

>-----Original Message-----
>From: Luke Martinez [mailto:martinezlukerm using gmail.com]
>Sent: Monday, 15 November, 2021 21:10
>To: Viechtbauer, Wolfgang (SP)
>Cc: R meta
>Subject: Re: [R-meta] AICc or variance components, which one matters more?
>
>correction:
>
>You proposed g3 to solve this impasse, but g3 gives a higher AICc than
>that of g2.
>
>On Mon, Nov 15, 2021 at 2:06 PM Luke Martinez <martinezlukerm using gmail.com> wrote:
>>
>> Hi Wolfgang,
>>
>> Thank you! If we go by your ICC principle, then since almost all
>> studies uniquely belong to one lab (except one exception), then ~ 1 |
>> lab/study (as in g1) should prevail over list(~1|lab, ~1|study) [as in
>> g2].
>>
>> But the thing is that AICc doesn't agree with this. That's exactly
>> where I get stuck in preferring one model over the other. Model g1
>> matches the data structure better, but g2 has a smaller AICc?
>>
>> You proposed g3 to solve this impasse, but g3 gives the same AICc as that of
>g2.
>>
>> Moreover, we have not yet added any moderators. If we do, then using
>> g2 or g3 with already 0 variance components would mean that such
>> zero-variance components really don't do much in the model. And that
>> was why I thought specifying random effects as in g1 which gives
>> non-zero variance components seems like a better use of the random
>> effects especially as we add moderators.
>>
>>
>> (g1=rma.mvyi, vi, random = ~1|lab/study, data = dd))
>> (g2=rma.mv(yi, vi, random = list(~1|lab, ~1|study), data = dd))
>> (g3=rma.mv(yi, vi, random = list(~1|lab/study, ~ 1 | study), data = dd))
>>
>> fitstats(g1,g2,g3)[5,]
>>
>>              g1            g2           g3
>> AICc: 30.85992 29.73897 35.73897
>>
>> On Mon, Nov 15, 2021 at 1:14 PM Viechtbauer, Wolfgang (SP)
>> <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >
>> > The various models allow for different correlation structures for the
>underlying true effects. For example, the model with only ~1|lab/study implies an
>ICC of sigma^2_lab / (sigma^2_lab + sigma^2_study) for true effects belonging to
>different studies within the same lab and does not allow true effects to be
>correlated across labs even if these labs were involved in the same study. If one
>wants to account for the latter, one can add ~ 1 | study and this doesn't
>automatically have to replace the /study part in ~1|lab/study.
>> >
>> > In the original data structure you showed, there were at times multiple rows
>for the same study within the same lab. One could then even go further and use
>~1|lab/study/es_id because without this, the ~1|lab/study model implies an ICC of
>1 for true effects belonging to the same study within the same lab. So one could
>even entertain the model:
>> >
>> > (g5=rma.mv(yi, vi, random = ~1|lab/study/es_id, data = dd))
>> >
>> > and then again:
>> >
>> > (g6=rma.mv(yi, vi, random = list(~1|lab/study/es_id, ~ 1 | study), data =
>dd))
>> >
>> > Of course this is all silly with the toy dataset, but even there all variance
>components are identifiable.
>> >
>> > Ultimately, fully understanding these models requires writing out what they
>imply about the ICC for different combinations of lab, study, and es_id (e.g.,
>same lab and same study and different es_id, same lab and different studies,
>different labs and same study, and so on). As any good stats book would say at
>this point: I leave this as an exercise to the reader.
>> >
>> > Best,
>> > Wolfgang