[R-meta] Multilevel meta-analysis, mutually exclusive levels

[Viechtbauer Wolfgang (SP)]

Dear Erik,

Random effects do not have to be nested -- there can also be crossed random effects. See, for example, chapter 12 (Models for cross-classified random effects) in Raudenbush and Bryk (2002).

In your case, you might consider adding random effects for studies, random effects for the effects within studies, and random effects for the outcomes (not nested within studies!). So, assuming the data are coded like this:

study effect outcome  lag  yi  vi
---------------------------------
1     1      A
1     2      D
2     1      A
2     2      B
2     3      C
3     1      B
4     1      B
4     2      D

then this would be something along the lines of (if you are using metafor):

rma.mv(yi, vi, mods = ~ lag, random = list(~ 1 | study/effect, ~ 1 | outcome), data=dat)

Not sure what you mean by 'participant level' -- unless you have the raw data, then there is no participant level.

Also, in studies examining multiple outcomes, I assume the same participants were measured with respect to the various outcomes. In that case, the sampling errors of the effects (yi values) are not independent, so you would need to account for this, either by computing the non-diagonal V matrix (which would require knowing or obtaining 'guestimates' of the correlations between the outcomes) or using cluster-robust inference methods. See the archives for lots of discussions on this.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of E. van der Meulen
Sent: Tuesday, 19 December, 2017 11:38
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Multilevel meta-analysis, mutually exclusive levels

Dear all,

I have been struggling with a specific problem in conducting a multilevel meta-analysis. I will discuss my problem in more detail below. However, in essence the question is the following: in a multilevel structure for meta-analysis can a single study fall under multiple branches of a level that is higher than the study level.

In more detail:

The study context:  I am conducting a review on the predictive validity of a personality concept of soldiers (concept X) n the development of a range of mental health and functioning problems. In short the research question is: too which degree can the concept X predict mental health status of soldiers. Concept X has been longitudinally associated to a wide variety of mental health outcomes: PTSD, depression, alcohol abuse, anxiety problems, etc. The first part of the review is to conduct a meta-analysis for these different concept X and mental health outcomes associations separately without any MULTILEVEL meta-analysis.

A second part is to test a series of moderators using multilevel meta-analysis, and that is where the problems start. I expect to find an effect of for example the time-lag between the measurement of concept X and mental health outcomes (the longer the time-lag, the lower the effect size). However, to make such analyses meaningful, I can only use the aggregate of all effect sizes available, regardless of the type of mental health outcome (otherwise I would have too few effect sizes).

The multilevel moderator analysis problem: as longitudinal associations of concept X and different outcomes can not (and are not) similar, the assessment of the time-lag moderator might yield spurious relationships. For example, when concept X is correlated with PTSD at .10 and with depression on .30, but on the same account the typical time-lag in studies on PTSD is 3 years, and depression is only one week. Then a result could be that the time-lag is a significant moderator, however, due to the huge difference in time-lag and effect size magnitude in PTSD an depression studies, it is not the time-lag but the outcome measure that causes this significant effect. The most tangible manner in overcoming this problem would be to correct for the type of outcome measure (i.e. depression or PTSD as a predictor). However, this would lead to a high number of predictors (75 effect sizes and about 18 predictors). Therefore I made a multilevel model with four levels, from highest to lowest: level
  4) mental health outcome, level 3) individual study, level 2) individual effect sizes and level 1) participant level. (In which the level 1 through 3 are similar to most 'typical' multilevel meta-analysis.) However, a consequence of this multilevel model is that several studies fall under multiple branches of level 4, as they conduct analyses on concept X associations with more than 1 mental health outcome (e.g. PTSD AND depression). In a typical multilevel analogy, consider a multilevel construct of classrooms, schools and school districts. In this a typical multilevel model with include a highest level of school districts (I and II), under which you might have several schools A, B and C, and several classrooms within each school (A1, A2, A3, B1, C1). In my model I would have the problem that my model includes the following:

Schooldistrict level I --> School A and B ---> Classrooms A3 and B1
Schooldistrict Level II --> School A and C ---> Classrooms A1, A2, and C1

As you can see school A (or in my actual problem, the school is a study) falls both under Schooldistrict level I and II. However, the classrooms are only used a single time (or effect sizes only appear once). Is that a problem, that these categories are not mutually exclusive? Or even violating an assumption?

My own model would like something like this:

Mental health outcome (PTSD) ---> Study A and B ---> Effect size A1, A2 and B1 (all effect sizes of associations between PTSD and concept X).
Mental health outcome (depression) --> Study A and C ---> Effect sizes A3 and C1 (all effect sizes of associations between depression and concept X).

I could really use some input on this!

Kind regards,

Erik van der Meulen