[R-meta] imputing covariance matrices for meta-analysis of dependent effects

[Viechtbauer Wolfgang (SP)]

Hi James,

This is indeed useful.

However, I am not sure if you are computing the covariances correctly. You are essentially using:

covariance = correlation * sqrt(variance1 * variance2)

But apparently, if one goes through the derivation for the covariance for standardized mean differences, one ends up with a different equation (equation 10 in Kalaian and Raudenbush, 1996). In fact, the equation for the covariance depends on the measure used. See, for example:

Wei, Y., & Higgins, J. P. (2013). Estimating within-study covariances in multivariate meta-analysis with multiple outcomes. Statistics in Medicine, 32(7), 1191-1205.

Historical side note: Interestingly, the 'covariance = correlation * sqrt(variance1 * variance2)' equation for standardized mean differences was also used in:

Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111-120.

but this was later corrected in Kalaian and Raudenbush (1996) based on Gleser and Olkin (1994).

In practice, it probably makes relatively difference how exactly one computes those covariances, esp. if one is 'guestimating' the correlation between the measures anyway (and then follows things up with some kind of cluster-robust approach, as you describe on your blog). As far as I am concerned, it is important though that one actually computes some kind of covariances (to get a better 'working' var-cov matrix to begin with). I am seeing an increasing number of papers where multiple effect size estimates based on the same sample (so, multivariate data) are being analyzed using a multilevel model like the one described by Konstantopoulos (2011). But that model assumes that sampling errors are uncorrelated, so this is a misapplication of that model. That is also why I added this at one point:

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#uncorrelated_sampling_errors

As for the different results reported in Kalaian and Raudenbush (1996) and the ones obtained with metafor -- a couple years ago, I also entered those data (from Table 1) and tried to re-analyze the dataset (as I was planning to include that dataset also in metafor) and ran into similar discrepancies. In fact, if one re-creates the scatterplots that are shown in Figure 1 from the data reported in Table 1, it becomes clear that there must be several printing errors. So, it's not really possible to reproduce the results from that paper, which is unfortunate, since it would be a nice illustration of the multivariate approach.

Best,
Wolfgang

-- 
Wolfgang Viechtbauer, Ph.D., Statistician | Department of Psychiatry and    
Neuropsychology | Maastricht University | P.O. Box 616 (VIJV1) | 6200 MD    
Maastricht, The Netherlands | +31 (43) 388-4170 | http://www.wvbauer.com    

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of James Pustejovsky
Sent: Thursday, August 10, 2017 17:59
To: Michael Dewey
Cc: r-sig-meta-analysis at r-project.org
Subject: Re: [R-meta] imputing covariance matrices for meta-analysis of dependent effects

Michael,

I was not aware of the metavcov package, so thank you for pointing it out.

On first glance, it looks like the metavcov functions are configured based
on the assumption that you have detailed information about the correlations
between outcomes for each study (i.e., it requires a list of correlation
matrices as input). The function from my previous message is a simpler
utility function, for use when you need to make more or less ad hoc
assumptions about the correlations. So I would say that it does complement
the metavcov package, but I would welcome corrections if this is not an
accurate assessment.

Best,
James

On Thu, Aug 10, 2017 at 10:43 AM, Michael Dewey <lists at dewey.myzen.co.uk>
wrote:

> Dear James
>
> Not sure how relevant this is but does it complement in any way the
> package  https://CRAN.R-project.org/package=metavcov ? I have not used it
> by the way.
>
> Michael
>
> On 10/08/2017 15:04, James Pustejovsky wrote:
>
>> All,
>>
>> A common problem in multivariate meta-analysis is that the information
>> needed to calculate the correlation between effect size estimates is not
>> reported in available sources, even when the variances of the estimates
>> can
>> be calculated. One approach to handling this situation is to simply make
>> an
>> informed guess about the correlation between the effect sizes. I use this
>> approach fairly often and have written a function that makes some of the
>> calculations easier. The function calculates a block-diagonal
>> variance-covariance matrix based on the sampling variances and a guess
>> about the degree of correlation. More details available here:
>>
>> http://jepusto.github.io/imputing-covariance-matrices-for-
>> multi-variate-meta-analysis
>>
>> There's nothing innovative about the methods I describe, but I figured
>> that
>> others might find the function useful. I would welcome comments,
>> questions,
>> or debate about the utility of the approach I used.
>>
>> Cheers,
>> James