[R-meta] Multivariate meta-analysis with unknown covariances?

Thu Aug 17 17:11:45 CEST 2017

Looks good.

A few points:

About the warning: "Some combinations of the levels of the inner factor never occurred. Corresponding rho value(s) fixed to 0." Indeed, this is a result of some pairs of outcomes never occurring within studies. Hence, it is not possible to estimate the correlation for such pairs.

With 24 different outcome measures for 28 effect sizes, struct="CS" is indeed the most you can do. Any more complex structure would either be a total overfit or will not converge. Note that struct="CS" implies that the amount of heterogeneity is the same for all outcomes and that the correlation between outcomes is the same, regardless of which pair of outcomes you are looking at. Obviously, the results from your model are then an approximation to a more complex reality.

In most cases, whether you use metafor or clubSandwich for the 'cluster robust' computations won't make much of a difference. The nice thing about clubSandwich is that it provides some refined robust methods (esp. vcov="CR2" and vcov="CR3") that should in principle lead to more accurate results in small samples. You might as well use them.

Note that cooks.distance() won't work with an object returned by robust(). So, you would have to directly apply cooks.distance() to 'MultiMeta'. But that should still be okay as a general diagnostic tool.

I would suggest to present the results with and without the outlier.

Also, you cannot get a test for (residual) heterogeneity based on (cluster) robust methods. A test for (residual) heterogeneity is based on a model that assumes that the only source of variability (and covariation) is contained/captured by V. But (cluster) robust methods try to approximate all source of variability (and covariation), not just those due to V. Given that V in your case is just a rough approximation, any test for (residual) heterogeneity should also be treated that way.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of schlegei
Sent: Thursday, August 17, 2017 15:23
To: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] Multivariate meta-analysis with unknown covariances?

First of all: Thanks a lot to the two of you for your kind and very 
helpful answers!
What a fortunate coincidence that James Pustejovsky published his blog 
entry the same day I was asking how to deal with unknown covariances.
Does anyone know a published reference, in which the three steps are 
recommended? Until now, I couldn‘t find one.

I want to share my R-Code with the list (some explanations included). 
Maybe someone more experienced might check if my specification is 
correct. And maybe it helps other clueless people with the same problem:

1. Calculation of the effect sizes:

data <- escalc(measure = "SMD", m1i = m12, sd1i = sd12, n1i = n12, m2i = 
m22, sd2i = sd22, n2i = n22, data = data, append = TRUE, replace = 
FALSE)
data <- escalc(measure = "PBIT", ai = a1, bi = b1, ci = c1, di = d1, 
data = data, append = TRUE, replace = FALSE)

I calculated the effect sizes with Hedges g („SMD“) and transformed 
dichotomized data to standardized mean differences with the help of the 
probit transformed risk difference („PBIT“).

2.  Imputation of the variance-covariance matrix:

Vlist <- impute_covariance_matrix(vi = data$vi, cluster = data$study, r 
= 0.7)

Right now, I fixed the correlation between all outcomes in the same 
study to 0.7. This is quite rough and I want to precise this guestimates 
(I asked most of the original autors if they can provide me with the 
correlations between the outcomes and will also precise this guestimate 
by substituting it with correlations from other studies that used the 
same outcomes).

3. Conduct the multivariate meta-analysis:

MultiMeta <- rma.mv(yi = yi, V = Vlist, mods =  ~ factor(controlgroup) 
-1, random = ~ factor(outcome)|study, struct = "CS", data = data)

I formulated a multivariate meta-analysis with random effects and 
included the imputed covariance matrix into the model. I had to fix the 
structure to a compound symmetric structure („CS“), because with less 
restrictive structures I received the following warning message:
Fehler in rma.mv(yi = yi, V = VPostdicho, mods = ~factor(Kontrollgruppe) 
-  : Optimizer (nlminb) did not achieve convergence (convergence = 1).
Zusätzlich: Warnmeldung: In .process.G.afterrmna(mf.g, g.nlevels, 
g.levels, struct[1], tau2, :Some combinations of the levels of the inner 
factor never occurred. Corresponding rho value(s) fixed to 0.
Probably the warning message is due to the fact that I have 24 different 
outcome measures for 28 effect sizes – so there is no combination of 
outcomes that appears several times. In addition to that, I included a 
categorical moderator (mods =  ~ factor(controlgroup) -1) in the 
meta-analysis. When I set e.g. struct = „UN“, the optimizer is not 
converging. With struct = „CS“, it is.
So setting struct = „CS“ seems to be the only possiblity here?

4. Compute robust tests and confidence intervals:
I tried both options (in metafor as well as in club sandwich) to 
estimate robust standard errors and p-values:

metafor_robust <- robust.rma.mv(MultiMeta, cluster = data$study)
ClubSandwich_robust <- coef_test(MultiMeta, vcov = "CR2")

Both options result in similar (but not identical) values. Probably, for 
my research it is interchangeable which option I choose?

For further analysis (like Cook‘s distance or Egger‘s test) one uses 
exclusively the robust estimates, or?
In my case, the test for residual heterogenity in the model with the 
imputed covariance matrix is highly significant. When I exclude one 
effect size that is an outlier, the residual heterogenity is not 
significant anymore. May I present this result, although it refers to 
the model with the imputed covariance matrix? (There is no test for 
residual heterogenity for robust estimates)

Best regards,
Isabel Schlegel

Am 10.08.2017 21:23, schrieb Viechtbauer Wolfgang (SP):
> Indeed, unknown correlations seems to be a 'hot topic' right now.
> 
> Let me first clarify that metafor cannot somehow magically solve the
> problem with missing covariances. One has to do some extra work to
> deal with this issue. The post that Isabel refers to
> (https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html)
> discusses some possibilities. A defensible strategy is to:
> 
> 1) Guestimate the unknown correlations and then compute approximate
> covariances between the effect size estimates.
> 2) Fit a proper multivariate model.
> 3) Follow things up with a cluster-robust approach.
> 
> This is basically what James just described in his post and on his
> blog (and what Emily refers to):
> 
> http://jepusto.github.io/imputing-covariance-matrices-for-multi-variate-meta-analysis
> 
> And so, yes, I think you can conduct a multivariate meta-analysis
> based on the data set described.
> 
> Best,
> Wolfgang
> 
> -----Original Message-----
> From: R-sig-meta-analysis
> [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Emily
> Finne
> Sent: Thursday, August 10, 2017 20:25
> To: r-sig-meta-analysis at r-project.org
> Subject: Re: [R-meta] Multivariate meta-analysis with unknown 
> covariances?
> 
> Dear Isabel,
> 
> I'm not an expert in meta-analysis, so I have to leave your question -
> if multivariate MA makes sense at all in your case - open to the 
> experts
> around. If your different outcomes do measure different symptoms
> originating in different disorders I would have my doubts if it makes
> sense to combine them.
> 
> But it seems to me that the follwing recent post is addressing your
> problem with the missing covariances quite well:
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-August/000094.html
> 
> How to construct the var-cov matrix for multiple endpoint studies in
> metafor is illustrated here:
> http://www.metafor-project.org/doku.php/analyses:gleser2009. (But you
> would have to guess a within-study correlation between different 
> outcomes).
> 
> Good luck with your thesis!
> 
> Best,
> Emily
> 
> Am 10.08.2017 um 18:54 schrieb schlegei:
>> for my master thesis, I want to conduct a multivariate meta-analysis
>> with the R-package metafor.
>> Unfortunately, I‘m not sure if it is possible to conduct this analysis
>> based on my data set.
>> 
>> To illustrate my problem, a few words about my research question: I
>> investigate the efficacy of gestalt therapy (a psychotherapeutic
>> approach) for people with a mental disorder according to DSM-IV/ICD-10
>> (my focus is on symptom reduction). My literature research resulted in
>> 12 randomized controlled trials (RCTs).
>> From this 12 studies, I extracted 28 outcomes and calculated the
>> effect sizes and variances (standardized mean differences). I assume
>> outcomes from the same study are dependent. Unfortunately, in no study
>> correlations/covariances between the sampling errors of the outcomes
>> are reported. So the within study covariance structure is totally
>> missing.
>> Because I included studies about people with different mental
>> disorders, my outcomes are pretty heterogeneous: Only one
>> questionnaire (Beck‘s Depression Inventory) was used in 4 studies,
>> apart from that the outcomes don‘t overlap between the studies.
>> 
>> Summed up, my data set consists of the following variables:
>> ES_ID = idenfification number for every effect size (I have 28 effect
>> sizes)
>> study = every study gets one number (I included 12 studies)
>> outcome = every questionnaire/outcome gets one letter (I included 24
>> different outcomes)
>> yi = effect size
>> vi = variance of the effect size
>> 
>> Is it possible to conduct a multivariate meta-analysis based on this
>> data set?
>> 
>> My supervisor told me, the missing within study covariances can be
>> easily estimated with the R-package metafor. Up until now, I do not
>> understand how. Following the discussions on stackexchange and this
>> mailing list (e.g.
>> https://stat.ethz.ch/pipermail/r-sig-mixed-models/2015q2/023727.html),
>> it seems to me that estimating the whole covariance structure is not
>> that easy and is attended by some disadvantages/assumptions . This
>> also coincides with other articles I have read about multivariate
>> meta-analysis and in which missing covariances are described as a
>> major problem. When I contacted my supervisor, he just told me that
>> the literature I read is out-dated and repeated that the problem of
>> missing covariances can be solved with metafor (unfortunately, he
>> didn‘t recommend up-to-date articles to me).
>> 
>> Right now, I feel a bit locked in a stalemate. Is there a simple,
>> up-to-date solution for my problem with the missing covariances that I
>> have overseen?
>> If yes: I would be extremely happy about any tip!
>> If no: Do you think, it makes sense to conduct a multivariate
>> meta-analysis based on my data set or is it more appropriate to choose
>> one effect size per study (univariate meta-analysis)? If it is
>> possible to conduct a multivariate meta-analysis based on my data: Is
>> there a strategy – like making a rough guess of the correlations or
>> using robust methods – that you would recommend?
>> 
>> I would be really happy to hear a response,
>> Isabel Schlegel
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

_______________________________________________
R-sig-meta-analysis mailing list
R-sig-meta-analysis at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis