[R-meta] When to skip an extra level?

[Timothy MacKenzie]

Thank you Wolfgang. In the first linked post
[https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-July/000896.html]
you mention that:

random = list(~ 1 | study, ~ 1 | outcome)

would mean that different studies with the same outcome, and different
outcomes from the same study are "correlated" making 'outcome' and
'study' crossed random-effects. So, two questions.

First, if we had a factor moderator such as:    random = list(~
fac_mod | study, ~ fac_mod | outcome)

now, do we have two "further" correlated crossed random-effects?

Second, if we had multiple rows with the same outcome value in each
study, then can we still add some random effect for that like:

random = list(~ fac_mod | study, ~ fac_mod | outcome, ~ 1 | id)

Much appreciated,
Tim

On Thu, Sep 16, 2021 at 6:09 AM Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Dear Philippe,
>
> There are two issues here:
>
> 1) Model identifiability: Yes, that should always be checked and profile() can help with that. However, even if the number of units at two levels just differs by one, the two components should, in principle, be identifiable. Here is an example:
>
> library(metafor)
>
> dat <- dat.bornmann2007
> dat <- escalc(measure="OR", ai=waward, n1i=wtotal, ci=maward, n2i=mtotal, data=dat)
> res1 <- rma.mv(yi, vi, random = ~ 1 | study/obs, data=dat)
> res1
>
> par(mfrow=c(2,1))
> profile(res1) # both profile likelihood plots show a peak
>
> # now suppose that studies 1 and 2 come from a single 'paper'
> dat$paper <- as.numeric(factor(dat$study))
> dat$paper[dat$paper <= 2] <- 1
>
> res2 <- rma.mv(yi, vi, random = ~ 1 | paper/study/obs, data=dat)
> res2
>
> par(mfrow=c(3,1))
> profile(res2) # all three profiles show a peak
>
> 2) Using likelihood ratio tests to compare nested models: This is something that one can also do. res1 and res2 are nested, so we can do:
>
> anova(res1, res2)
>
> and we find that res2 does not provide a significantly better fit than res1 (p=.43).
>
> However, then we are back to potentially making changes to the model (if we had started with res2) based on the data and I would personally try to avoid this.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Philippe Tadger [mailto:philippetadger using gmail.com]
> >Sent: Thursday, 16 September, 2021 12:54
> >To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> >Subject: Re: [R-meta] When to skip an extra level?
> >
> >Dear Wolfgang, Tim
> >Thank you for bringing such interesting topic. What do you think in using the
> >profile likelihood (PL) exploration of the variance-covariance parameters to see
> >if exist any difference between a) full model with 3 levels,  b) model with 2
> >levels (sample error and paper level) c) model with 2 levels (sample error and
> >outcome level). The PL have been propose as a way to check the assumption that the
> >likelihood is indeed identifiable, additionally it is implemented in metafor.
> >
> >On 15/09/2021 20:06, Viechtbauer, Wolfgang (SP) wrote:
> >Dear Tim,
> >
> >The question generally is when it makes sense to leave out a level if the data
> >could be regarded as having a hierarchical structure (which is modeled in terms of
> >nested random effects along the lines of '~ 1 | var1/var2/var3/...') and if so,
> >which level(s) to leave out.
> >
> >I don't think there is any general consensus on this or even much empirical
> >evidence to back up any particular approach. However, in general, I would say that
> >if the number of units at a particular level is very similar to the number of
> >units at one level below it (e.g., there are 199 papers and 200 studies - so one
> >paper describes two studies while the remaining 198 papers describe one study  --
> >to make the example from the second link even more extreme), then it becomes very
> >difficult to distinguish the variances at those two levels and I would consider
> >dropping one of the two levels. I don't have any super strong feelings on whether
> >to then drop the upper (paper) or lower (study) level -- in the extreme scenario
> >above, it is unlikely to matter. Dropping the paper level would treat the two
> >studies from that one paper as independent. Dropping the study level would assume
> >that the average true effects (averaged over whatever lower levels there are in
> >the hierarchy below 'studies') in those two studies from that one paper are
> >homogeneous. Neither is (probably) correct.
> >
> >I cannot tell you where the exact point is (in terms of # of papers versus # of
> >studies) where I would start to consider dropping a level.
> >
> >Best,
> >Wolfgang
> >
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Timothy MacKenzie
> >Sent: Wednesday, 15 September, 2021 2:31
> >To: R meta
> >Subject: [R-meta] When to skip an extra level?
> >
> >Dear Meta-analysis Community Members,
> >
> >I want to get some clarity regarding when not to add an additional
> >level. I have found two posts and was wondering how they agree with
> >one another? (It seems the first one says is at odds with the second
> >one)
> >
> >***This post: https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-
> >July/000896.html
> >suggests that we should avoid adding an extra level (row id) in:
> >
> >random = ~ 1 | study/outcome/id
> >
> >if not so many "studies" have repeatedly used the same "outcome".
> >
> >***This post: https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2019-
> >March/001479.html
> >(second message from the top) suggests that we should avoid adding an
> >extra level (study_id) in:
> >
> >random = ~ 1 | paper_id/study_id/row_id
> >
> >Arguing that "One can probably skip a level if the number of units at
> >a particular level is not much higher than the number of units at the
> >next level (the two variance components are then hard to distinguish).
> >So, for example, 200 "studies" in 180 "papers" is quite similar, so
> >one could probably *leave out the studies* level and only add random
> >effects for papers (the two variance components are then hard to
> >distinguish)."
> >
> >Sincerely,
> >Tim
> >_______________________________________________
> >R-sig-meta-analysis mailing list
> >R-sig-meta-analysis using r-project.org
> >https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >--
> >Kind regards/Saludos cordiales
> >Philippe Tadger
> >ORCID, Reseach Gate
> >Phone/WhatsApp: +32498774742
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis