[R-meta] Multivariate meta-analysis with unknown covariances?

Thu Aug 17 17:15:17 CEST 2017

How does this approach differ from the robust variance estimation (RVE;
https://pdfs.semanticscholar.org/0f89/ebc4d1822486a21b42c087bd5ded78c58e4a.pdf)
approach, using the `robumeta` package? That is what I have used to model
unknown covariances between effect sizes.

On Thu, Aug 17, 2017 at 10:11 AM, Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl> wrote:

> Looks good.
>
> A few points:
>
> About the warning: "Some combinations of the levels of the inner factor
> never occurred. Corresponding rho value(s) fixed to 0." Indeed, this is a
> result of some pairs of outcomes never occurring within studies. Hence, it
> is not possible to estimate the correlation for such pairs.
>
> With 24 different outcome measures for 28 effect sizes, struct="CS" is
> indeed the most you can do. Any more complex structure would either be a
> total overfit or will not converge. Note that struct="CS" implies that the
> amount of heterogeneity is the same for all outcomes and that the
> correlation between outcomes is the same, regardless of which pair of
> outcomes you are looking at. Obviously, the results from your model are
> then an approximation to a more complex reality.
>
> In most cases, whether you use metafor or clubSandwich for the 'cluster
> robust' computations won't make much of a difference. The nice thing about
> clubSandwich is that it provides some refined robust methods (esp.
> vcov="CR2" and vcov="CR3") that should in principle lead to more accurate
> results in small samples. You might as well use them.
>
> Note that cooks.distance() won't work with an object returned by robust().
> So, you would have to directly apply cooks.distance() to 'MultiMeta'. But
> that should still be okay as a general diagnostic tool.
>
> I would suggest to present the results with and without the outlier.
>
> Also, you cannot get a test for (residual) heterogeneity based on
> (cluster) robust methods. A test for (residual) heterogeneity is based on a
> model that assumes that the only source of variability (and covariation) is
> contained/captured by V. But (cluster) robust methods try to approximate
> all source of variability (and covariation), not just those due to V. Given
> that V in your case is just a rough approximation, any test for (residual)
> heterogeneity should also be treated that way.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-
> bounces at r-project.org] On Behalf Of schlegei
> Sent: Thursday, August 17, 2017 15:23
> To: r-sig-meta-analysis at r-project.org
> Subject: Re: [R-meta] Multivariate meta-analysis with unknown covariances?
>
> First of all: Thanks a lot to the two of you for your kind and very
> helpful answers!
> What a fortunate coincidence that James Pustejovsky published his blog
> entry the same day I was asking how to deal with unknown covariances.
> Does anyone know a published reference, in which the three steps are
> recommended? Until now, I couldn‘t find one.
>
>
> I want to share my R-Code with the list (some explanations included).
> Maybe someone more experienced might check if my specification is
> correct. And maybe it helps other clueless people with the same problem:
>
> 1. Calculation of the effect sizes:
>
> data <- escalc(measure = "SMD", m1i = m12, sd1i = sd12, n1i = n12, m2i =
> m22, sd2i = sd22, n2i = n22, data = data, append = TRUE, replace =
> FALSE)
> data <- escalc(measure = "PBIT", ai = a1, bi = b1, ci = c1, di = d1,
> data = data, append = TRUE, replace = FALSE)
>
> I calculated the effect sizes with Hedges g („SMD“) and transformed
> dichotomized data to standardized mean differences with the help of the
> probit transformed risk difference („PBIT“).
>
>
> 2.  Imputation of the variance-covariance matrix:
>
> Vlist <- impute_covariance_matrix(vi = data$vi, cluster = data$study, r
> = 0.7)
>
> Right now, I fixed the correlation between all outcomes in the same
> study to 0.7. This is quite rough and I want to precise this guestimates
> (I asked most of the original autors if they can provide me with the
> correlations between the outcomes and will also precise this guestimate
> by substituting it with correlations from other studies that used the
> same outcomes).
>
>
> 3. Conduct the multivariate meta-analysis:
>
> MultiMeta <- rma.mv(yi = yi, V = Vlist, mods =  ~ factor(controlgroup)
> -1, random = ~ factor(outcome)|study, struct = "CS", data = data)
>
> I formulated a multivariate meta-analysis with random effects and
> included the imputed covariance matrix into the model. I had to fix the
> structure to a compound symmetric structure („CS“), because with less
> restrictive structures I received the following warning message:
> Fehler in rma.mv(yi = yi, V = VPostdicho, mods = ~factor(Kontrollgruppe)
> -  : Optimizer (nlminb) did not achieve convergence (convergence = 1).
> Zusätzlich: Warnmeldung: In .process.G.afterrmna(mf.g, g.nlevels,
> g.levels, struct[1], tau2, :Some combinations of the levels of the inner
> factor never occurred. Corresponding rho value(s) fixed to 0.
> Probably the warning message is due to the fact that I have 24 different
> outcome measures for 28 effect sizes – so there is no combination of
> outcomes that appears several times. In addition to that, I included a
> categorical moderator (mods =  ~ factor(controlgroup) -1) in the
> meta-analysis. When I set e.g. struct = „UN“, the optimizer is not
> converging. With struct = „CS“, it is.
> So setting struct = „CS“ seems to be the only possiblity here?
>
>
> 4. Compute robust tests and confidence intervals:
> I tried both options (in metafor as well as in club sandwich) to
> estimate robust standard errors and p-values:
>
> metafor_robust <- robust.rma.mv(MultiMeta, cluster = data$study)
> ClubSandwich_robust <- coef_test(MultiMeta, vcov = "CR2")
>
> Both options result in similar (but not identical) values. Probably, for
> my research it is interchangeable which option I choose?
>
> For further analysis (like Cook‘s distance or Egger‘s test) one uses
> exclusively the robust estimates, or?
> In my case, the test for residual heterogenity in the model with the
> imputed covariance matrix is highly significant. When I exclude one
> effect size that is an outlier, the residual heterogenity is not
> significant anymore. May I present this result, although it refers to
> the model with the imputed covariance matrix? (There is no test for
> residual heterogenity for robust estimates)
>
> Best regards,
> Isabel Schlegel
>
> Am 10.08.2017 21:23, schrieb Viechtbauer Wolfgang (SP):
> > Indeed, unknown correlations seems to be a 'hot topic' right now.
> >
> > Let me first clarify that metafor cannot somehow magically solve the
> > problem with missing covariances. One has to do some extra work to
> > deal with this issue. The post that Isabel refers to
> > (https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html)
> > discusses some possibilities. A defensible strategy is to:
> >
> > 1) Guestimate the unknown correlations and then compute approximate
> > covariances between the effect size estimates.
> > 2) Fit a proper multivariate model.
> > 3) Follow things up with a cluster-robust approach.
> >
> > This is basically what James just described in his post and on his
> > blog (and what Emily refers to):
> >
> > http://jepusto.github.io/imputing-covariance-matrices-
> for-multi-variate-meta-analysis
> >
> > And so, yes, I think you can conduct a multivariate meta-analysis
> > based on the data set described.
> >
> > Best,
> > Wolfgang
> >
> > -----Original Message-----
> > From: R-sig-meta-analysis
> > [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Emily
> > Finne
> > Sent: Thursday, August 10, 2017 20:25
> > To: r-sig-meta-analysis at r-project.org
> > Subject: Re: [R-meta] Multivariate meta-analysis with unknown
> > covariances?
> >
> > Dear Isabel,
> >
> > I'm not an expert in meta-analysis, so I have to leave your question -
> > if multivariate MA makes sense at all in your case - open to the
> > experts
> > around. If your different outcomes do measure different symptoms
> > originating in different disorders I would have my doubts if it makes
> > sense to combine them.
> >
> > But it seems to me that the follwing recent post is addressing your
> > problem with the missing covariances quite well:
> > https://stat.ethz.ch/pipermail/r-sig-meta-analysis/
> 2017-August/000094.html
> >
> > How to construct the var-cov matrix for multiple endpoint studies in
> > metafor is illustrated here:
> > http://www.metafor-project.org/doku.php/analyses:gleser2009. (But you
> > would have to guess a within-study correlation between different
> > outcomes).
> >
> > Good luck with your thesis!
> >
> > Best,
> > Emily
> >
> > Am 10.08.2017 um 18:54 schrieb schlegei:
> >> for my master thesis, I want to conduct a multivariate meta-analysis
> >> with the R-package metafor.
> >> Unfortunately, I‘m not sure if it is possible to conduct this analysis
> >> based on my data set.
> >>
> >> To illustrate my problem, a few words about my research question: I
> >> investigate the efficacy of gestalt therapy (a psychotherapeutic
> >> approach) for people with a mental disorder according to DSM-IV/ICD-10
> >> (my focus is on symptom reduction). My literature research resulted in
> >> 12 randomized controlled trials (RCTs).
> >> From this 12 studies, I extracted 28 outcomes and calculated the
> >> effect sizes and variances (standardized mean differences). I assume
> >> outcomes from the same study are dependent. Unfortunately, in no study
> >> correlations/covariances between the sampling errors of the outcomes
> >> are reported. So the within study covariance structure is totally
> >> missing.
> >> Because I included studies about people with different mental
> >> disorders, my outcomes are pretty heterogeneous: Only one
> >> questionnaire (Beck‘s Depression Inventory) was used in 4 studies,
> >> apart from that the outcomes don‘t overlap between the studies.
> >>
> >> Summed up, my data set consists of the following variables:
> >> ES_ID = idenfification number for every effect size (I have 28 effect
> >> sizes)
> >> study = every study gets one number (I included 12 studies)
> >> outcome = every questionnaire/outcome gets one letter (I included 24
> >> different outcomes)
> >> yi = effect size
> >> vi = variance of the effect size
> >>
> >> Is it possible to conduct a multivariate meta-analysis based on this
> >> data set?
> >>
> >> My supervisor told me, the missing within study covariances can be
> >> easily estimated with the R-package metafor. Up until now, I do not
> >> understand how. Following the discussions on stackexchange and this
> >> mailing list (e.g.
> >> https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html),
> >> it seems to me that estimating the whole covariance structure is not
> >> that easy and is attended by some disadvantages/assumptions . This
> >> also coincides with other articles I have read about multivariate
> >> meta-analysis and in which missing covariances are described as a
> >> major problem. When I contacted my supervisor, he just told me that
> >> the literature I read is out-dated and repeated that the problem of
> >> missing covariances can be solved with metafor (unfortunately, he
> >> didn‘t recommend up-to-date articles to me).
> >>
> >> Right now, I feel a bit locked in a stalemate. Is there a simple,
> >> up-to-date solution for my problem with the missing covariances that I
> >> have overseen?
> >> If yes: I would be extremely happy about any tip!
> >> If no: Do you think, it makes sense to conduct a multivariate
> >> meta-analysis based on my data set or is it more appropriate to choose
> >> one effect size per study (univariate meta-analysis)? If it is
> >> possible to conduct a multivariate meta-analysis based on my data: Is
> >> there a strategy – like making a rough guess of the correlations or
> >> using robust methods – that you would recommend?
> >>
> >> I would be really happy to hear a response,
> >> Isabel Schlegel
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

	[[alternative HTML version deleted]]