[R-meta] Computing var-covariance matrix with correlations of six non-independent outcomes

[James Pustejovsky]

Just for fun, here are a couple of further observations to add to
Wolfgang's responses:

First, I agree entirely with Wolfgang's explanation of the benefits and
drawbacks of a multivariate model versus separate models for each outcome.
There is actually a trick, though, that will let you compare the average ES
across outcomes even when using sub-group analysis. The idea is to put all
of the sub-groups into one model, treating the effect sizes as independent,
and allowing separate parameters for each outcome. You can do this by
setting struct = "DIAG" and assuming independent ES in the multivariate
model specification:

# sub-group model
res <- rma.mv(g, V = v, mods = ~ factor(motivation) - 1, random = ~
factor(motivation) | study, struct="DIAG", data=meta)
res

The results of this model should be (nearly) identical to what you get from
fitting separate models for each type of motivation. The problem is that
the standard errors for the average effect estimates, and the covariances
of the average effect estimates, are going to be wrong because this model
assumes all the sampling errors and random effects are independent (which
we know not to be the case). But this can be resolved by using robust
variance estimates for the average effect size estimates:

robust(res, cluster = meta$study) # metafor's robust SEs

or

library(clubSandwich)
conf_int(res, vcov = "CR2") # clubSandwich SEs with small-sample adjustments

Robust variance estimation also provides a way to test contrasts between
types of motivation. This approach is very similar (though not quite
identical) to an approach described by Chen, Hong, & Riley:

Chen, Y., Hong, C., & Riley, R. D. (2015). An alternative pseudolikelihood
method for multivariate random‐effects meta‐analysis. *Statistics in
medicine*, *34*(3), 361-380.

For an in-depth discussion of the benefits of multivariate meta-analysis, I
recommend the following:

Jackson, D., Riley, R., & White, I. R. (2011). Multivariate meta‐analysis:
potential and promise. *Statistics in Medicine*, *30*(20), 2481-2498.

Riley, R. D., Jackson, D., Salanti, G., Burke, D. L., Price, M., Kirkham,
J., & White, I. R. (2017). Multivariate and network meta-analysis of
multiple outcomes and multiple treatments: rationale, concepts, and
examples. *BMJ*, *358*, j3932.


James

On Wed, Jul 8, 2020 at 3:45 AM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> For a long time, the 'modus operandi' for dealing with dependencies was to
> avoid them at all costs (by selecting only one of multiple dependent
> estimates or collapsing them into a single one and/or by running separate
> analyses). I am glad at least to see that people are slowly exploring more
> feature-rich models.
>
> As for my course (since you mentioned it): I am actually teaching my
> meta-analysis course at the end of October and this time fully online (for
> obvious reasons). If you are interested, see:
>
> http://www.wvbauer.com/doku.php/course_ma
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Mika Manninen [mailto:mixu89 using gmail.com]
> >Sent: Wednesday, 08 July, 2020 10:33
> >To: Viechtbauer, Wolfgang (SP)
> >Cc: James Pustejovsky; r-sig-meta-analysis using r-project.org
> >Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
> >six non-independent outcomes
> >
> >Wolfgang,
> >
> >Wow! That is about as much detail as one could ever hope for, thank you so
> >much again. That clarifies a lot, not least that the recommendation for
> the
> >multilevel models is not always based on a deep understanding of the
> >assumptions and data relationships. but rather seems to be almost like a
> >trendy word.
> >
> >Mika
> >
> >ps hope to take part in your meta-analysis workshops in the future!
> >
> >ke 8. heinäk. 2020 klo 11.20 Viechtbauer, Wolfgang (SP)
> >(wolfgang.viechtbauer using maastrichtuniversity.nl) kirjoitti:
> >Hi Mika,
> >
> >Just for completeness sake, there was one omission and one error in the
> code
> >I posted:
> >
> >meta <- data.frame(meta) # to turn your matrix into a dataframe
> >
> >And this line should have been:
> >
> >corlist <- mapply(function(rmat, mots) rmat[mots,mots], corlist, mots)
> >
> >As for your questions:
> >
> >1) Running separate analyses: Nothing inherently wrong with that. But
> there
> >is the potential to obtain slightly more precise estimates of the 6
> average
> >effects by 'borrowing information' from each other. Using the subset of
> the
> >data you posted, let's compare:
> >
> ># multivariate model
> >res <- rma.mv(g, V, mods = ~ factor(motivation) - 1, random = ~
> >factor(motivation) | study, struct="UN", data=meta)
> >res
> >
> ># separate RE models for each level of motivation
> >sep <- lapply(1:6, function(i) rma(g, v, data=meta, subset=motivation==i))
> >sep <- do.call(rbind, lapply(sep, function(x) cbind(coef(summary(x)),
> >tau2=x$tau2)))
> >rownames(sep) <- paste0("motivation", 1:6)
> >
> ># compare results
> >res
> >round(sep, 4)
> >
> >As you will see, the SE of several estimates (but not all) is lower in the
> >MV model compared to the separate models.
> >
> >Also, if you want to compare effects, you really need the MV model. For
> >example, I can do this to test whether the effect is different for
> outcomes
> >1 and 2:
> >
> >anova(res, L=c(1,-1,0,0,0,0))
> >
> >But you cannot do this when you have fitted separate models, because the
> >results in sep[[1]] and sep[[2]] are not independent.
> >
> >2) Why do people propose fitting a multilevel model (with random effects
> for
> >study and estimates within studies)?
> >
> >I can only speculate about this. I suspect it is a combination of the
> >following things.
> >
> >For one, people find it difficult to assemble the correlations needed to
> fit
> >a proper multivariate model (so that the V matrix can be constructed).
> Also,
> >this step is not something that is easily automated and requires some
> >programming.
> >
> >Also, people may have learned in their multilevel course that one can deal
> >with dependencies by adding appropriate random effects to their model.
> That
> >is true to some extent, but there are two levels of dependencies in the
> >present (i.e., meta-analytic) context, namely dependencies between the
> >sampling errors (which are captured by the covariances in the V matrix)
> and
> >dependencies between the underlying true effects (which we can capture by
> >adding correlated random effects within studies to the model). People may
> >not be aware of this additional complexity and hence may think that just
> >adding random effects is sufficient.
> >
> >Furthermore, there are some papers that discuss the use of the multilevel
> >approach for meta-analyzing correlated outcomes:
> >
> >Moeyaert, M., Ugille, M., Beretvas, S. N., Ferron, J., Bunuan, R., & Van
> den
> >Noortgate, W. (2017). Methods for dealing with multiple outcomes in meta-
> >analysis: A comparison between averaging effect sizes, robust variance
> >estimation and multilevel meta-analysis. International Journal of Social
> >Research Methodology, 20(6), 559-572.
> >
> >Van den Noortgate, W., Lopez-Lopez, J. A., Marin-Martinez, F., & Sanchez-
> >Meca, J. (2013). Three-level meta-analysis of dependent effect sizes.
> >Behavior Research Methods, 45(2), 576-594.
> >
> >Van den Noortgate, W., Lopez-Lopez, J. A., Marin-Martinez, F., & Sanchez-
> >Meca, J. (2015). Meta-analysis of multiple outcomes: A multilevel
> approach.
> >Behavior Research Methods, 47(4), 1274-1294.
> >
> >Based on the simulations, the multilevel approach may be 'good enough'
> under
> >certain conditions.
> >
> >In the present example, that would be:
> >
> >mlm <- rma.mv(g, v, mods = ~ factor(motivation) - 1, random = ~ 1 |
> >study/outcome, data=meta)
> >mlm
> >
> >There are three essential differences compared to the multivariate model
> we
> >fitted earlier:
> >
> >a) We ignore the covariances in V and treat the sampling errors as
> >independent.
> >b) The three-level model implies that the correlation of the underlying
> true
> >effects within studies is the same for all 6*5/2 = 15 pairs of outcomes.
> >c) The model implies that the amount of heterogeneity is the same for all
> 6
> >outcomes.
> >
> >Since the study-level variance component is estimated to be 0 here, the
> >model actually implies that there is no correlation between the true
> effects
> >for the different outcomes within studies.
> >
> >Therefore, the results would be the same as fitting 6 separate models
> where
> >we constrain the amount of heterogeneity to the outcome-level variance
> >component. We can easily check this:
> >
> >sep <- lapply(1:6, function(i) rma(g, v, data=meta, subset=motivation==i,
> >tau2=mlm$sigma2[2]))
> >sep <- do.call(rbind, lapply(sep, function(x) cbind(coef(summary(x)),
> >tau2=x$tau2)))
> >rownames(sep) <- paste0("motivation", 1:6)
> >round(sep, 4)
> >
> >You will see that the results are indeed exactly the same as those from
> the
> >multilevel model.
> >
> >Nobody can say which result is "correct" here, but the multilevel model
> >makes a number of simplifying assumptions that I would want to avoid if
> >possible.
> >
> >Best,
> >Wolfgang
> >
> >>-----Original Message-----
> >>From: Mika Manninen [mailto:mixu89 using gmail.com]
> >>Sent: Wednesday, 08 July, 2020 9:25
> >>To: Viechtbauer, Wolfgang (SP)
> >>Cc: James Pustejovsky; r-sig-meta-analysis using r-project.org
> >>Subject: Re: [R-meta] Computing var-covariance matrix with correlations
> of
> >>six non-independent outcomes
> >>
> >>Dear Wolfgang and James,
> >>
> >>Thank you so much for your replies and I just want to say that I can only
> >>admire your attitude and willingness to help and explain. I was banging
> my
> >>head against the wall for a while there trying to figure out how the
> matrix
> >>should be created. I owe you one!
> >>
> >>Two quick questions. The multivariate approach with the var-cov matrix
> >seems
> >>to make the most sense for this analysis, but if one was to run six
> >separate
> >>analyses for the outcomes (like I did the first time around), how
> >>problematic would that be ? Also, can you estimate as to why a 3-level
> >>approach was recommended to me (just adding hierarchical study level to
> the
> >>analysis)? It is hard for me to see the rationale behind this.
> >>
> >>I hope you have a great day,
> >>
> >>Mika
>

	[[alternative HTML version deleted]]