[R-meta] Multilevel meta-analysis, mutually exclusive levels

[E. van der Meulen]

Dear all,

I have been struggling with a specific problem in conducting a multilevel meta-analysis. I will discuss my problem in more detail below. However, in essence the question is the following: in a multilevel structure for meta-analysis can a single study fall under multiple branches of a level that is higher than the study level.

In more detail:

The study context:  I am conducting a review on the predictive validity of a personality concept of soldiers (concept X) n the development of a range of mental health and functioning problems. In short the research question is: too which degree can the concept X predict mental health status of soldiers. Concept X has been longitudinally associated to a wide variety of mental health outcomes: PTSD, depression, alcohol abuse, anxiety problems, etc. The first part of the review is to conduct a meta-analysis for these different concept X and mental health outcomes associations separately without any MULTILEVEL meta-analysis.

A second part is to test a series of moderators using multilevel meta-analysis, and that is where the problems start. I expect to find an effect of for example the time-lag between the measurement of concept X and mental health outcomes (the longer the time-lag, the lower the effect size). However, to make such analyses meaningful, I can only use the aggregate of all effect sizes available, regardless of the type of mental health outcome (otherwise I would have too few effect sizes).

The multilevel moderator analysis problem: as longitudinal associations of concept X and different outcomes can not (and are not) similar, the assessment of the time-lag moderator might yield spurious relationships. For example, when concept X is correlated with PTSD at .10 and with depression on .30, but on the same account the typical time-lag in studies on PTSD is 3 years, and depression is only one week. Then a result could be that the time-lag is a significant moderator, however, due to the huge difference in time-lag and effect size magnitude in PTSD an depression studies, it is not the time-lag but the outcome measure that causes this significant effect. The most tangible manner in overcoming this problem would be to correct for the type of outcome measure (i.e. depression or PTSD as a predictor). However, this would lead to a high number of predictors (75 effect sizes and about 18 predictors). Therefore I made a multilevel model with four levels, from highest to lowest: level 4) mental health outcome, level 3) individual study, level 2) individual effect sizes and level 1) participant level. (In which the level 1 through 3 are similar to most 'typical' multilevel meta-analysis.) However, a consequence of this multilevel model is that several studies fall under multiple branches of level 4, as they conduct analyses on concept X associations with more than 1 mental health outcome (e.g. PTSD AND depression). In a typical multilevel analogy, consider a multilevel construct of classrooms, schools and school districts. In this a typical multilevel model with include a highest level of school districts (I and II), under which you might have several schools A, B and C, and several classrooms within each school (A1, A2, A3, B1, C1). In my model I would have the problem that my model includes the following:

Schooldistrict level I --> School A and B ---> Classrooms A3 and B1
Schooldistrict Level II --> School A and C ---> Classrooms A1, A2, and C1

As you can see school A (or in my actual problem, the school is a study) falls both under Schooldistrict level I and II. However, the classrooms are only used a single time (or effect sizes only appear once). Is that a problem, that these categories are not mutually exclusive? Or even violating an assumption?

My own model would like something like this:

Mental health outcome (PTSD) ---> Study A and B ---> Effect size A1, A2 and B1 (all effect sizes of associations between PTSD and concept X).
Mental health outcome (depression) --> Study A and C ---> Effect sizes A3 and C1 (all effect sizes of associations between depression and concept X).

I could really use some input on this!

Kind regards,

Erik van der Meulen

	[[alternative HTML version deleted]]