[R-meta] Multilevel meta-analysis, mutually exclusive levels

Tue Jan 2 20:40:20 CET 2018

Erik,

This is an interesting question. I would like to offer a perspective that
contrasts a bit with the approach you are pursuing. If I understand your
problem correctly, your aim is to investigate whether follow-up time (or
some other predictor) is associated with effect magnitude, while
controlling for the possible confounding effects of the type of outcome. In
this case, I think it is acceptable and even desirable to include dummy
variables for each outcome category, even though this leads to a model with
many predictors. It's not that you'd want to interpret the coefficients on
all of the outcome category dummies, it's just that they need to be in the
model to control for possible spurious association. This approach is the
simplest and I would argue safest way to examine the focal moderator,
because the estimated association between follow-up time and effect size is
based entirely on patterns *within* outcome categories.

The random effects approach that you've described (and that Wolfgang
elaborated) might be valid as well, but it entails a further, potentially
strong assumption that the random outcome effects are independent of the
study's follow-up time (i.e., that the random effects are independent of
the predictors). Based on the description of the problem you're
investigating, it seems like this assumption might not be that reasonable.
If it is wrong, then the focal coefficient estimate may be biased (to an
extent that depends on the strength of association between the random
effects and the predictor, among other factors) because it is estimated
based on a combination of associations both within *and between* outcome
categories, rather than solely within outcome categories.

James

On Tue, Jan 2, 2018 at 4:36 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

> Dear Erik,
>
> Random effects do not have to be nested -- there can also be crossed
> random effects. See, for example, chapter 12 (Models for cross-classified
> random effects) in Raudenbush and Bryk (2002).
>
> In your case, you might consider adding random effects for studies, random
> effects for the effects within studies, and random effects for the outcomes
> (not nested within studies!). So, assuming the data are coded like this:
>
> study effect outcome  lag  yi  vi
> ---------------------------------
> 1     1      A
> 1     2      D
> 2     1      A
> 2     2      B
> 2     3      C
> 3     1      B
> 4     1      B
> 4     2      D
>
> then this would be something along the lines of (if you are using metafor):
>
> rma.mv(yi, vi, mods = ~ lag, random = list(~ 1 | study/effect, ~ 1 |
> outcome), data=dat)
>
> Not sure what you mean by 'participant level' -- unless you have the raw
> data, then there is no participant level.
>
> Also, in studies examining multiple outcomes, I assume the same
> participants were measured with respect to the various outcomes. In that
> case, the sampling errors of the effects (yi values) are not independent,
> so you would need to account for this, either by computing the non-diagonal
> V matrix (which would require knowing or obtaining 'guestimates' of the
> correlations between the outcomes) or using cluster-robust inference
> methods. See the archives for lots of discussions on this.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
> bounces at r-project.org] On Behalf Of E. van der Meulen
> Sent: Tuesday, 19 December, 2017 11:38
> To: r-sig-meta-analysis at r-project.org
> Subject: [R-meta] Multilevel meta-analysis, mutually exclusive levels
>
> Dear all,
>
> I have been struggling with a specific problem in conducting a multilevel
> meta-analysis. I will discuss my problem in more detail below. However, in
> essence the question is the following: in a multilevel structure for
> meta-analysis can a single study fall under multiple branches of a level
> that is higher than the study level.
>
> In more detail:
>
> The study context:  I am conducting a review on the predictive validity of
> a personality concept of soldiers (concept X) n the development of a range
> of mental health and functioning problems. In short the research question
> is: too which degree can the concept X predict mental health status of
> soldiers. Concept X has been longitudinally associated to a wide variety of
> mental health outcomes: PTSD, depression, alcohol abuse, anxiety problems,
> etc. The first part of the review is to conduct a meta-analysis for these
> different concept X and mental health outcomes associations separately
> without any MULTILEVEL meta-analysis.
>
> A second part is to test a series of moderators using multilevel
> meta-analysis, and that is where the problems start. I expect to find an
> effect of for example the time-lag between the measurement of concept X and
> mental health outcomes (the longer the time-lag, the lower the effect
> size). However, to make such analyses meaningful, I can only use the
> aggregate of all effect sizes available, regardless of the type of mental
> health outcome (otherwise I would have too few effect sizes).
>
> The multilevel moderator analysis problem: as longitudinal associations of
> concept X and different outcomes can not (and are not) similar, the
> assessment of the time-lag moderator might yield spurious relationships.
> For example, when concept X is correlated with PTSD at .10 and with
> depression on .30, but on the same account the typical time-lag in studies
> on PTSD is 3 years, and depression is only one week. Then a result could be
> that the time-lag is a significant moderator, however, due to the huge
> difference in time-lag and effect size magnitude in PTSD an depression
> studies, it is not the time-lag but the outcome measure that causes this
> significant effect. The most tangible manner in overcoming this problem
> would be to correct for the type of outcome measure (i.e. depression or
> PTSD as a predictor). However, this would lead to a high number of
> predictors (75 effect sizes and about 18 predictors). Therefore I made a
> multilevel model with four levels, from highest to lowest: level
>   4) mental health outcome, level 3) individual study, level 2) individual
> effect sizes and level 1) participant level. (In which the level 1 through
> 3 are similar to most 'typical' multilevel meta-analysis.) However, a
> consequence of this multilevel model is that several studies fall under
> multiple branches of level 4, as they conduct analyses on concept X
> associations with more than 1 mental health outcome (e.g. PTSD AND
> depression). In a typical multilevel analogy, consider a multilevel
> construct of classrooms, schools and school districts. In this a typical
> multilevel model with include a highest level of school districts (I and
> II), under which you might have several schools A, B and C, and several
> classrooms within each school (A1, A2, A3, B1, C1). In my model I would
> have the problem that my model includes the following:
>
> Schooldistrict level I --> School A and B ---> Classrooms A3 and B1
> Schooldistrict Level II --> School A and C ---> Classrooms A1, A2, and C1
>
> As you can see school A (or in my actual problem, the school is a study)
> falls both under Schooldistrict level I and II. However, the classrooms are
> only used a single time (or effect sizes only appear once). Is that a
> problem, that these categories are not mutually exclusive? Or even
> violating an assumption?
>
> My own model would like something like this:
>
> Mental health outcome (PTSD) ---> Study A and B ---> Effect size A1, A2
> and B1 (all effect sizes of associations between PTSD and concept X).
> Mental health outcome (depression) --> Study A and C ---> Effect sizes A3
> and C1 (all effect sizes of associations between depression and concept X).
>
> I could really use some input on this!
>
> Kind regards,
>
> Erik van der Meulen
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]