[R-meta] Advice on whether to add nested study group effects to longitudinal meta analysis

[Reza Norouzian]

Dear Mike,

Prior to responding to your queries, the syntax you have provided
seems to need a minor modification (likely a typo) to be corrected as:

random = list(~group | study, ~ time | interaction(study, group))

This set-up allows the group-level true effects to be correlated in
each study across all your studies.

In addition, this set-up allows the time-level true effects to be
correlated in each group in a study across all your studies.

Taken together, this set up indicates that group-level true effects
(co)vary in the studies, and in turn, time-level true effects (co)vary
in the groups in each study (a fully nested random-effects structure).

Please see my comments inline:

>>>> Quite a few of the studies (71) use multiple independent groups (but same population of interest for us), just split slightly differently across studies. The way these have been split is of less interest to us / the outcome, and the grouping is not consistent between studies. (I.e. group A in study 1 is not the same grouping as group Ain study 2).

For the correlations I mentioned above, in regards to your syntax, to
be meaningful, you would need for the instances of *group* to
represent the same thing across your studies. (This would have not
been necessary, if *group* had been used solely as an *id* variable on
the right side of `|`, see below).

>>>> Do you think it would be better to keep these groups separate and nest their effects within studies? My hesitancy doing this/reason for seeking advice about this is whether this will add too much complexity, and make it difficult to distinguish between different levels of variance.

Without data and context, it's often difficult to offer meaningful
advice on how to structure the random part of your model (arguably
even with data and context, it's still not as easy).

Oftentimes, there are quite a few options to consider. However, each
of those options are to be empirically tested to see which one(s)
provide a relatively good match to your data and help you achieve your
study goals.

>>>> Alternatively should I combine these groups into one before running the meta-analysis, to simplify the structure (so one effect per study)? Obviously I’m aware of the downsides and cautions of such data reduction too

Again, it's the matter of data and context. You can try different
models and compare their fit to the data. For instance, to use *group*
solely as an *id* variable, one of the available options is to simply
re-parametrize the syntax above to:

random = list(~1 | study/group, ~ time | interaction(study, group))

In theory, this set-up allows a distinction to be made between the
variation in true effects at the study and group levels while allowing
the time-level true effects to (co)vary in each group in a study.

Conversely, if you end up dropping the *group* variable from the
analysis, then an option might be:

random = list(~time | study, ~ 1| row_id)

This set-up allows the time-level true effects to be correlated in
each study across all your studies while allowing the variation
possibly owing to all other uniquenesses and/or uncoded features of
the studies (like the *group* variable we dropped) to be somewhat
captured at the individual true effect size level (row_id).

Once again, without data and context, it's best to think of these
suggestions simply as a few of the *potential* possibilities. rma.mv()
is fairly flexible in accommodating various situations you might be
encountering in your data.

>>>> Also of note is that unfortunately as is unsurprising we don’t have exact data on correlation between timepoints or correlation within study groups, so we will have to estimate these to construct approx covariance matrices. To help overcome these limitations I’m planning on using clubSandwhich and robust methods as appropriate.

This is good thinking. However, please don't think of RVE as some kind
of panacea that magically relieves the burden of thinking about the
assumptions encoded into a model, especially so, if you plan to
meaningfully use the estimates of true-effects' variation in the model
as part of your research study.

Kind regards,
Reza
Reza Norouzian (he/him/his)
Assistant Professor of TESOL | Anaheim University | Homepage
Senior Researcher for Multilingual Learners | Oregon Department of Education


Reza Norouzian (he/him/his)
Assistant Professor of TESOL | Anaheim University | Homepage
Senior Researcher for Multilingual Learners | Oregon Department of Education


On Sun, Jan 22, 2023 at 7:17 PM Mick Girdwood <M.Girdwood using latrobe.edu.au> wrote:
>
> Thank you again for all the valuable information on this message board.
>
> I am planning a longitudinal meta-analysis and am looking for advice on the best approach for my data structure. This is an example for one of the outcomes we are investigating. We currently have k=120 studies measuring this outcome with a variety of different timepoints and follow ups (i.e. some cross-sectional, some longitudinal). I have spent a lot of time reading about the different ways to set up a so called longitudinal MA including different correlation structures etc, so for now I am ok there. My questions is around adding additional levels to the analysis.
>
> Quite a few of the studies (71) use multiple independent groups (but same population of interest for us), just split slightly differently across studies. The way these have been split is of less interest to us / the outcome, and the grouping is not consistent between studies. (I.e. group A in study 1 is not the same grouping as group Ain study 2)
>
> ## Do you think it would be better to keep these groups separate and nest their effects within studies? My hesitancy doing this/reason for seeking advice about this is whether this will add too much complexity, and make it difficult to distinguish between different levels of variance
>
> Confirming then to specify my random effects I would use
>
> random = list(~study | group), ~ time | interaction(study, group)
>
> ## Alternatively should I combine these groups into one before running the meta-analysis, to simplify the structure (so one effect per study)? Obviously I’m aware of the downsides and cautions of such data reduction too
>
> I appreciate there is no black and white best answer, and answer is very context dependant, but your advice is greatly appreciated! For reference these two previous questions were helpful and similar in nature https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-April/002794.html, https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-July/000896.html. Also of note is that unfortunately as is unsurprising we don’t have exact data on correlation between timepoints or correlation within study groups, so we will have to estimate these to construct approx covariance matrices. To help overcome these limitations I’m planning on using clubSandwhich and robust methods as appropriate.
>
> Thank you,
>
> Mick Girdwood
> La Trobe University | Australia
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis