[R-meta] Multivariate meta-analysis with unknown covariances?

Thu Aug 17 17:26:16 CEST 2017

This approach is similar to the robumeta implementation of RVE, but there
are some differences:

   - robumeta uses a specific random effects structure that can’t be
   controlled by the user, whereas metafor can be used to estimate a variety
   of different random effects structures.
   - robumeta uses a moment estimator for the between-study variance,
   whereas metafor provides FML or REML estimation.
   - robumeta uses semi-efficient, diagonal weights when fitting the
   meta-regression, whereas metafor uses weights that are fully efficient
   (exactly inverse-variance) under the working model.

Regarding the last point about using diagonal weighting, the issues are a
bit subtle and there's no guidance in the literature about when to use
which approach. My current understanding (which I'm still working on
developing formally, so please take this with a grain of salt) is that the
robumeta weighting approach is reasonable for meta-regression models in
which the predictor variables are all at the study-level (i.e., no
within-study variation in a given predictor). However, if you are
interested in using predictors that vary within study (such as contrasting
effects between different categories of outcomes), then the metafor
approach should be considered---and particularly if there is wide variation
in the number of effects per study (with some studies contributing just 1-2
effects while others contribute many more).

James

On Thu, Aug 17, 2017 at 10:15 AM, Mark White <markhwhiteii at gmail.com> wrote:

> How does this approach differ from the robust variance estimation (RVE;
> https://pdfs.semanticscholar.org/0f89/ebc4d1822486a21b42c087bd5ded78
> c58e4a.pdf)
> approach, using the `robumeta` package? That is what I have used to model
> unknown covariances between effect sizes.
>
> On Thu, Aug 17, 2017 at 10:11 AM, Viechtbauer Wolfgang (SP) <
> wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:
>
> > Looks good.
> >
> > A few points:
> >
> > About the warning: "Some combinations of the levels of the inner factor
> > never occurred. Corresponding rho value(s) fixed to 0." Indeed, this is a
> > result of some pairs of outcomes never occurring within studies. Hence,
> it
> > is not possible to estimate the correlation for such pairs.
> >
> > With 24 different outcome measures for 28 effect sizes, struct="CS" is
> > indeed the most you can do. Any more complex structure would either be a
> > total overfit or will not converge. Note that struct="CS" implies that
> the
> > amount of heterogeneity is the same for all outcomes and that the
> > correlation between outcomes is the same, regardless of which pair of
> > outcomes you are looking at. Obviously, the results from your model are
> > then an approximation to a more complex reality.
> >
> > In most cases, whether you use metafor or clubSandwich for the 'cluster
> > robust' computations won't make much of a difference. The nice thing
> about
> > clubSandwich is that it provides some refined robust methods (esp.
> > vcov="CR2" and vcov="CR3") that should in principle lead to more accurate
> > results in small samples. You might as well use them.
> >
> > Note that cooks.distance() won't work with an object returned by
> robust().
> > So, you would have to directly apply cooks.distance() to 'MultiMeta'. But
> > that should still be okay as a general diagnostic tool.
> >
> > I would suggest to present the results with and without the outlier.
> >
> > Also, you cannot get a test for (residual) heterogeneity based on
> > (cluster) robust methods. A test for (residual) heterogeneity is based
> on a
> > model that assumes that the only source of variability (and covariation)
> is
> > contained/captured by V. But (cluster) robust methods try to approximate
> > all source of variability (and covariation), not just those due to V.
> Given
> > that V in your case is just a rough approximation, any test for
> (residual)
> > heterogeneity should also be treated that way.
> >
> > Best,
> > Wolfgang
> >
> > -----Original Message-----
> > From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
> > bounces at r-project.org] On Behalf Of schlegei
> > Sent: Thursday, August 17, 2017 15:23
> > To: r-sig-meta-analysis at r-project.org
> > Subject: Re: [R-meta] Multivariate meta-analysis with unknown
> covariances?
> >
> > First of all: Thanks a lot to the two of you for your kind and very
> > helpful answers!
> > What a fortunate coincidence that James Pustejovsky published his blog
> > entry the same day I was asking how to deal with unknown covariances.
> > Does anyone know a published reference, in which the three steps are
> > recommended? Until now, I couldn‘t find one.
> >
> >
> > I want to share my R-Code with the list (some explanations included).
> > Maybe someone more experienced might check if my specification is
> > correct. And maybe it helps other clueless people with the same problem:
> >
> > 1. Calculation of the effect sizes:
> >
> > data <- escalc(measure = "SMD", m1i = m12, sd1i = sd12, n1i = n12, m2i =
> > m22, sd2i = sd22, n2i = n22, data = data, append = TRUE, replace =
> > FALSE)
> > data <- escalc(measure = "PBIT", ai = a1, bi = b1, ci = c1, di = d1,
> > data = data, append = TRUE, replace = FALSE)
> >
> > I calculated the effect sizes with Hedges g („SMD“) and transformed
> > dichotomized data to standardized mean differences with the help of the
> > probit transformed risk difference („PBIT“).
> >
> >
> > 2.  Imputation of the variance-covariance matrix:
> >
> > Vlist <- impute_covariance_matrix(vi = data$vi, cluster = data$study, r
> > = 0.7)
> >
> > Right now, I fixed the correlation between all outcomes in the same
> > study to 0.7. This is quite rough and I want to precise this guestimates
> > (I asked most of the original autors if they can provide me with the
> > correlations between the outcomes and will also precise this guestimate
> > by substituting it with correlations from other studies that used the
> > same outcomes).
> >
> >
> > 3. Conduct the multivariate meta-analysis:
> >
> > MultiMeta <- rma.mv(yi = yi, V = Vlist, mods =  ~ factor(controlgroup)
> > -1, random = ~ factor(outcome)|study, struct = "CS", data = data)
> >
> > I formulated a multivariate meta-analysis with random effects and
> > included the imputed covariance matrix into the model. I had to fix the
> > structure to a compound symmetric structure („CS“), because with less
> > restrictive structures I received the following warning message:
> > Fehler in rma.mv(yi = yi, V = VPostdicho, mods = ~factor(Kontrollgruppe)
> > -  : Optimizer (nlminb) did not achieve convergence (convergence = 1).
> > Zusätzlich: Warnmeldung: In .process.G.afterrmna(mf.g, g.nlevels,
> > g.levels, struct[1], tau2, :Some combinations of the levels of the inner
> > factor never occurred. Corresponding rho value(s) fixed to 0.
> > Probably the warning message is due to the fact that I have 24 different
> > outcome measures for 28 effect sizes – so there is no combination of
> > outcomes that appears several times. In addition to that, I included a
> > categorical moderator (mods =  ~ factor(controlgroup) -1) in the
> > meta-analysis. When I set e.g. struct = „UN“, the optimizer is not
> > converging. With struct = „CS“, it is.
> > So setting struct = „CS“ seems to be the only possiblity here?
> >
> >
> > 4. Compute robust tests and confidence intervals:
> > I tried both options (in metafor as well as in club sandwich) to
> > estimate robust standard errors and p-values:
> >
> > metafor_robust <- robust.rma.mv(MultiMeta, cluster = data$study)
> > ClubSandwich_robust <- coef_test(MultiMeta, vcov = "CR2")
> >
> > Both options result in similar (but not identical) values. Probably, for
> > my research it is interchangeable which option I choose?
> >
> > For further analysis (like Cook‘s distance or Egger‘s test) one uses
> > exclusively the robust estimates, or?
> > In my case, the test for residual heterogenity in the model with the
> > imputed covariance matrix is highly significant. When I exclude one
> > effect size that is an outlier, the residual heterogenity is not
> > significant anymore. May I present this result, although it refers to
> > the model with the imputed covariance matrix? (There is no test for
> > residual heterogenity for robust estimates)
> >
> > Best regards,
> > Isabel Schlegel
> >
> > Am 10.08.2017 21:23, schrieb Viechtbauer Wolfgang (SP):
> > > Indeed, unknown correlations seems to be a 'hot topic' right now.
> > >
> > > Let me first clarify that metafor cannot somehow magically solve the
> > > problem with missing covariances. One has to do some extra work to
> > > deal with this issue. The post that Isabel refers to
> > > (https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html)
> > > discusses some possibilities. A defensible strategy is to:
> > >
> > > 1) Guestimate the unknown correlations and then compute approximate
> > > covariances between the effect size estimates.
> > > 2) Fit a proper multivariate model.
> > > 3) Follow things up with a cluster-robust approach.
> > >
> > > This is basically what James just described in his post and on his
> > > blog (and what Emily refers to):
> > >
> > > http://jepusto.github.io/imputing-covariance-matrices-
> > for-multi-variate-meta-analysis
> > >
> > > And so, yes, I think you can conduct a multivariate meta-analysis
> > > based on the data set described.
> > >
> > > Best,
> > > Wolfgang
> > >
> > > -----Original Message-----
> > > From: R-sig-meta-analysis
> > > [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Emily
> > > Finne
> > > Sent: Thursday, August 10, 2017 20:25
> > > To: r-sig-meta-analysis at r-project.org
> > > Subject: Re: [R-meta] Multivariate meta-analysis with unknown
> > > covariances?
> > >
> > > Dear Isabel,
> > >
> > > I'm not an expert in meta-analysis, so I have to leave your question -
> > > if multivariate MA makes sense at all in your case - open to the
> > > experts
> > > around. If your different outcomes do measure different symptoms
> > > originating in different disorders I would have my doubts if it makes
> > > sense to combine them.
> > >
> > > But it seems to me that the follwing recent post is addressing your
> > > problem with the missing covariances quite well:
> > > https://stat.ethz.ch/pipermail/r-sig-meta-analysis/
> > 2017-August/000094.html
> > >
> > > How to construct the var-cov matrix for multiple endpoint studies in
> > > metafor is illustrated here:
> > > http://www.metafor-project.org/doku.php/analyses:gleser2009. (But you
> > > would have to guess a within-study correlation between different
> > > outcomes).
> > >
> > > Good luck with your thesis!
> > >
> > > Best,
> > > Emily
> > >
> > > Am 10.08.2017 um 18:54 schrieb schlegei:
> > >> for my master thesis, I want to conduct a multivariate meta-analysis
> > >> with the R-package metafor.
> > >> Unfortunately, I‘m not sure if it is possible to conduct this analysis
> > >> based on my data set.
> > >>
> > >> To illustrate my problem, a few words about my research question: I
> > >> investigate the efficacy of gestalt therapy (a psychotherapeutic
> > >> approach) for people with a mental disorder according to DSM-IV/ICD-10
> > >> (my focus is on symptom reduction). My literature research resulted in
> > >> 12 randomized controlled trials (RCTs).
> > >> From this 12 studies, I extracted 28 outcomes and calculated the
> > >> effect sizes and variances (standardized mean differences). I assume
> > >> outcomes from the same study are dependent. Unfortunately, in no study
> > >> correlations/covariances between the sampling errors of the outcomes
> > >> are reported. So the within study covariance structure is totally
> > >> missing.
> > >> Because I included studies about people with different mental
> > >> disorders, my outcomes are pretty heterogeneous: Only one
> > >> questionnaire (Beck‘s Depression Inventory) was used in 4 studies,
> > >> apart from that the outcomes don‘t overlap between the studies.
> > >>
> > >> Summed up, my data set consists of the following variables:
> > >> ES_ID = idenfification number for every effect size (I have 28 effect
> > >> sizes)
> > >> study = every study gets one number (I included 12 studies)
> > >> outcome = every questionnaire/outcome gets one letter (I included 24
> > >> different outcomes)
> > >> yi = effect size
> > >> vi = variance of the effect size
> > >>
> > >> Is it possible to conduct a multivariate meta-analysis based on this
> > >> data set?
> > >>
> > >> My supervisor told me, the missing within study covariances can be
> > >> easily estimated with the R-package metafor. Up until now, I do not
> > >> understand how. Following the discussions on stackexchange and this
> > >> mailing list (e.g.
> > >> https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html
> ),
> > >> it seems to me that estimating the whole covariance structure is not
> > >> that easy and is attended by some disadvantages/assumptions . This
> > >> also coincides with other articles I have read about multivariate
> > >> meta-analysis and in which missing covariances are described as a
> > >> major problem. When I contacted my supervisor, he just told me that
> > >> the literature I read is out-dated and repeated that the problem of
> > >> missing covariances can be solved with metafor (unfortunately, he
> > >> didn‘t recommend up-to-date articles to me).
> > >>
> > >> Right now, I feel a bit locked in a stalemate. Is there a simple,
> > >> up-to-date solution for my problem with the missing covariances that I
> > >> have overseen?
> > >> If yes: I would be extremely happy about any tip!
> > >> If no: Do you think, it makes sense to conduct a multivariate
> > >> meta-analysis based on my data set or is it more appropriate to choose
> > >> one effect size per study (univariate meta-analysis)? If it is
> > >> possible to conduct a multivariate meta-analysis based on my data: Is
> > >> there a strategy – like making a rough guess of the correlations or
> > >> using robust methods – that you would recommend?
> > >>
> > >> I would be really happy to hear a response,
> > >> Isabel Schlegel
> > > _______________________________________________
> > > R-sig-meta-analysis mailing list
> > > R-sig-meta-analysis at r-project.org
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]