[R-meta] Computing var-covariance matrix with correlations of six non-independent outcomes

[Viechtbauer, Wolfgang (SP)]

For a long time, the 'modus operandi' for dealing with dependencies was to avoid them at all costs (by selecting only one of multiple dependent estimates or collapsing them into a single one and/or by running separate analyses). I am glad at least to see that people are slowly exploring more feature-rich models.

As for my course (since you mentioned it): I am actually teaching my meta-analysis course at the end of October and this time fully online (for obvious reasons). If you are interested, see:

http://www.wvbauer.com/doku.php/course_ma

Best,
Wolfgang

>-----Original Message-----
>From: Mika Manninen [mailto:mixu89 using gmail.com]
>Sent: Wednesday, 08 July, 2020 10:33
>To: Viechtbauer, Wolfgang (SP)
>Cc: James Pustejovsky; r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
>six non-independent outcomes
>
>Wolfgang,
>
>Wow! That is about as much detail as one could ever hope for, thank you so
>much again. That clarifies a lot, not least that the recommendation for the
>multilevel models is not always based on a deep understanding of the
>assumptions and data relationships. but rather seems to be almost like a
>trendy word.
>
>Mika
>
>ps hope to take part in your meta-analysis workshops in the future!
>
>ke 8. heinäk. 2020 klo 11.20 Viechtbauer, Wolfgang (SP)
>(wolfgang.viechtbauer using maastrichtuniversity.nl) kirjoitti:
>Hi Mika,
>
>Just for completeness sake, there was one omission and one error in the code
>I posted:
>
>meta <- data.frame(meta) # to turn your matrix into a dataframe
>
>And this line should have been:
>
>corlist <- mapply(function(rmat, mots) rmat[mots,mots], corlist, mots)
>
>As for your questions:
>
>1) Running separate analyses: Nothing inherently wrong with that. But there
>is the potential to obtain slightly more precise estimates of the 6 average
>effects by 'borrowing information' from each other. Using the subset of the
>data you posted, let's compare:
>
># multivariate model
>res <- rma.mv(g, V, mods = ~ factor(motivation) - 1, random = ~
>factor(motivation) | study, struct="UN", data=meta)
>res
>
># separate RE models for each level of motivation
>sep <- lapply(1:6, function(i) rma(g, v, data=meta, subset=motivation==i))
>sep <- do.call(rbind, lapply(sep, function(x) cbind(coef(summary(x)),
>tau2=x$tau2)))
>rownames(sep) <- paste0("motivation", 1:6)
>
># compare results
>res
>round(sep, 4)
>
>As you will see, the SE of several estimates (but not all) is lower in the
>MV model compared to the separate models.
>
>Also, if you want to compare effects, you really need the MV model. For
>example, I can do this to test whether the effect is different for outcomes
>1 and 2:
>
>anova(res, L=c(1,-1,0,0,0,0))
>
>But you cannot do this when you have fitted separate models, because the
>results in sep[[1]] and sep[[2]] are not independent.
>
>2) Why do people propose fitting a multilevel model (with random effects for
>study and estimates within studies)?
>
>I can only speculate about this. I suspect it is a combination of the
>following things.
>
>For one, people find it difficult to assemble the correlations needed to fit
>a proper multivariate model (so that the V matrix can be constructed). Also,
>this step is not something that is easily automated and requires some
>programming.
>
>Also, people may have learned in their multilevel course that one can deal
>with dependencies by adding appropriate random effects to their model. That
>is true to some extent, but there are two levels of dependencies in the
>present (i.e., meta-analytic) context, namely dependencies between the
>sampling errors (which are captured by the covariances in the V matrix) and
>dependencies between the underlying true effects (which we can capture by
>adding correlated random effects within studies to the model). People may
>not be aware of this additional complexity and hence may think that just
>adding random effects is sufficient.
>
>Furthermore, there are some papers that discuss the use of the multilevel
>approach for meta-analyzing correlated outcomes:
>
>Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van den
>Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-
>analysis: A comparison between averaging effect sizes, robust variance
>estimation and multilevel meta-analysis. International Journal of Social
>Research Methodology, 20(6), 559-572.
>
>Van den Noortgate, W., Lopez-Lopez, J. A., Marin-Martinez, F., & Sanchez-
>Meca, J. (2013). Three-level meta-analysis of dependent effect sizes.
>Behavior Research Methods, 45(2), 576-594.
>
>Van den Noortgate, W., Lopez-Lopez, J. A., Marin-Martinez, F., & Sanchez-
>Meca, J. (2015). Meta-analysis of multiple outcomes: A multilevel approach.
>Behavior Research Methods, 47(4), 1274-1294.
>
>Based on the simulations, the multilevel approach may be 'good enough' under
>certain conditions.
>
>In the present example, that would be:
>
>mlm <- rma.mv(g, v, mods = ~ factor(motivation) - 1, random = ~ 1 |
>study/outcome, data=meta)
>mlm
>
>There are three essential differences compared to the multivariate model we
>fitted earlier:
>
>a) We ignore the covariances in V and treat the sampling errors as
>independent.
>b) The three-level model implies that the correlation of the underlying true
>effects within studies is the same for all 6*5/2 = 15 pairs of outcomes.
>c) The model implies that the amount of heterogeneity is the same for all 6
>outcomes.
>
>Since the study-level variance component is estimated to be 0 here, the
>model actually implies that there is no correlation between the true effects
>for the different outcomes within studies.
>
>Therefore, the results would be the same as fitting 6 separate models where
>we constrain the amount of heterogeneity to the outcome-level variance
>component. We can easily check this:
>
>sep <- lapply(1:6, function(i) rma(g, v, data=meta, subset=motivation==i,
>tau2=mlm$sigma2[2]))
>sep <- do.call(rbind, lapply(sep, function(x) cbind(coef(summary(x)),
>tau2=x$tau2)))
>rownames(sep) <- paste0("motivation", 1:6)
>round(sep, 4)
>
>You will see that the results are indeed exactly the same as those from the
>multilevel model.
>
>Nobody can say which result is "correct" here, but the multilevel model
>makes a number of simplifying assumptions that I would want to avoid if
>possible.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Mika Manninen [mailto:mixu89 using gmail.com]
>>Sent: Wednesday, 08 July, 2020 9:25
>>To: Viechtbauer, Wolfgang (SP)
>>Cc: James Pustejovsky; r-sig-meta-analysis using r-project.org
>>Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
>>six non-independent outcomes
>>
>>Dear Wolfgang and James,
>>
>>Thank you so much for your replies and I just want to say that I can only
>>admire your attitude and willingness to help and explain. I was banging my
>>head against the wall for a while there trying to figure out how the matrix
>>should be created. I owe you one!
>>
>>Two quick questions. The multivariate approach with the var-cov matrix
>seems
>>to make the most sense for this analysis, but if one was to run six
>separate
>>analyses for the outcomes (like I did the first time around), how
>>problematic would that be ? Also, can you estimate as to why a 3-level
>>approach was recommended to me (just adding hierarchical study level to the
>>analysis)? It is hard for me to see the rationale behind this.
>>
>>I hope you have a great day,
>>
>>Mika