[R-meta] Computing var-covariance matrix with correlations of six non-independent outcomes

[Viechtbauer, Wolfgang (SP)]

Dear Mika,

Strictly speaking, the correct way to compute the covariances between multiple Cohen's d (or Hedges' g) values would require using methods that are described in this chapter:

Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357–376). New York: Russell Sage Foundation.

In particular, equation (19.27) gives the covariance between two d values for two different outcomes computed based on the same sample of subjects (and we assume that the correlation between the two outcomes is the same for subjects in the treatment and the control group). An example illustrating an application of this is provided here:

http://www.metafor-project.org/doku.php/analyses:gleser2009#multiple-endpoint_studies

although the code for the example is a bit simpler because each study has exactly 2 d values.

However, one can take a simpler approach that gives very similar results. Namely, if v_1 and v_2 are the sampling variances of d_1 and d_2 and r is the correlation between the two outcomes, then r * sqrt(v_1 * v_2) is usually a close approximation to what (19.27) would provide.

Based on your example, this can be computed here as follows:

# this is only required because you coded missings as "NA" (and not as NA)
corlist <- lapply(corlist, function(x) matrix(as.numeric(x), nrow=6, ncol=6))

# we only need a correlation matrix for the levels of motivation actually observed
mots <- split(meta$motivation, meta$study)
corlist <- mapply(function(rmat, mots) R[mots,mots], corlist, mots)
corlist

Vfun <- function(vi,rmat) {
   if (length(vi) == 1L) {
      matrix(vi)
   } else {
      S <- diag(sqrt(vi))
      S %*% rmat %*% S
   }
}

V <- mapply(Vfun, split(meta$v, meta$study), corlist)

Now V is the var-cov matrix of your SMD values, so you could proceed with a multivariate model with:

res <- rma.mv(g, V, mods = ~ factor(motivation) - 1, random = ~ factor(motivation) | study, struct="UN", data=meta)
res

This is a very general model allowing each level of motivation to have its own heterogeneity variance component and each pair to have its own correlation. I wouldn't recommend this with only 8 studies, but with 25 studies, this might be more sensible.

Best,
Wolfgang

>-----Original Message-----
>From: Mika Manninen [mailto:mixu89 using gmail.com]
>Sent: Tuesday, 07 July, 2020 21:53
>To: James Pustejovsky
>Cc: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
>six non-independent outcomes
>
>James,
>
>Thank you so much for replying.
>
>The dataset (8 studies out of 25 included) can be created with the following
>code:
>
>###Creating vectors for the outcome, study, motivation, effect size, and
>variance estimate
>
>#Total effects/outcomes
>outcome <- c(1:28)
>
>#Study
>study <- c(1,1,1,1,1,2,2,2,2,3,3,3,3,3,4,4,4,4,4,5,6,7,8,8,8,8,8,8)
>
>#Six different motivation types measured in the studies
>motivation <-
>c(1,3,4,5,6,1,3,4,5,1,3,4,5,6,1,3,4,5,6,6,6,6,1,2,3,4,5,6)
>
>#Hedges g:s
>g <- c(0.6068, 0.0603, 0.2684, -0.0886, 0.0415, 1.592, 1.4031, 0.7928,
>0.2013, 0.541, 0.1169,
>       0.3129, -0.0275, -0.3536, 1.5886, 2.7218, -1.6273, -0.4375, -1.0695,
>-0.1247, -0.1038,
>       -0.2706, 0.2045, -0.2701, 0.3763, -0.7371, -0.0666, 0.2789)
>
>#Variance estimates
>v <- c( 0.0162, 0.0155, 0.0157, 0.0155, 0.0155, 0.1889,
>0.17875984,0.15484225,
>        0.14432401, 0.0329, 0.0318, 0.0322, 0.0318, 0.0323, 0.1886, 0.2758,
>        0.1909, 0.147, 0.164, 0.0067, 0.0028, 0.004, 0.0726, 0.0729, 0.0735,
>        0.0771, 0.0723, 0.073)
>
>###Dataset matrix with different levels and motivation types
>meta <- cbind(outcome, study, motivation, g, v)
>View(meta)
>
>I created 6x6 correlation matrices (with 15 unique correlations) for the
>eight included studies. I am not sure if this is the most sensible approach
>as only a few studies have measured all six outcomes. Value NA reflects the
>absence of a correlation (absence of that one or both of the motivation
>types in the study). The row and column numbers (1-6) correspond to the
>variable motivation in the created dataset.
>
>### Correlation matrices for studies 1-8
>
>study1c <- rbind(c(1, "NA", .96, .34, -.33, -.66), c("NA", "NA", "NA", "NA",
>"NA", "NA"), c(.96, "NA", 1, .55, -.12, -.50),
>                 c(.34, "NA", .55, 1, .53, .05), c(-.33, "NA", -.12, .53, 1,
>.72), c(-.66, "NA", -.50, .05, .72, 1))
>
>study2c <- rbind(c(1, "NA", .96, .34, -.33, "NA"), c("NA", "NA", "NA", "NA",
>"NA", "NA"), c(.96, "NA", 1, .55, -.12, "NA"),
>                c(.34, "NA", .55, 1, .53, "NA"), c(-.33, "NA", -.12, .53, 1,
>"NA"), c("NA", "NA", "NA", "NA", "NA", "NA"))
>
>study3c <- rbind(c(1, "NA", .96, .34, -.33, -.66), c("NA", "NA", "NA", "NA",
>"NA", "NA"), c(.96, "NA", 1, .55, -.12, -.50),
>                 c(.34, "NA", .55, 1, .53, .05), c(-.33, "NA", -.12, .53, 1,
>.72), c(-.66, "NA", -.50, .05, .72, 1))
>
>study4c <- rbind(c(1, "NA", .85, .35, .06, -.44), c("NA", "NA", "NA", "NA",
>"NA", "NA"), c(.85, "NA", 1, .59, .46, -.14),
>                 c(.35, "NA", .59, 1, .71, .27), c(.06, "NA", .46, .71, 1,
>.52), c(-.44, "NA", -.14, .27, .52, 1))
>
>study5c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>
>study6c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>
>study7c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>
>study8c <- rbind(c(1, .79, .84, .24, -.20, -.50), c(.79, 1, .93, .47, .00, -
>.31), c(.84, .93, 1, .57, -.07, -.52),
>                    c(.24, .47, .57, 1, .43, .01), c(-.20, .00, -.07, .43,
>1, .55), c(-.50, -.31, -.52, .01, .55, 1))
>
>#list with all the correlations matrices
>
>corlist <- list(study1c, study2c, study3c, study4c, study5c, study6c,
>study7c, study8c)
>
>Mika
>
>ti 7. heinäk. 2020 klo 19.54 James Pustejovsky (jepusto using gmail.com)
>kirjoitti:
>Hi Mika,
>
>To add to Wolfgang's question, could you tell us a little bit more about how
>you have structured the data on correlations between types of motivation? Is
>it just one correlation matrix (6X6 matrix, with 1 + 2 + 3 + 4 + 5 = 15
>unique correlations)? Or is it study-specific?
>
>This sort of calculation is a bit tricky to carry out so I am not surprised
>that you haven't found a solution in the listserv archives. If you are
>willing to share your dataset (or a subset thereof, say 3-4 studies worth of
>data), it may make it easier for us to help problem solve.
>
>James
>
>On Tue, Jul 7, 2020 at 11:42 AM Viechtbauer, Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Mika,
>
>What effect size measure are you using for the meta-analysis?
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>project.org]
>>On Behalf Of Mika Manninen
>>Sent: Tuesday, 07 July, 2020 18:10
>>To: r-sig-meta-analysis using r-project.org
>>Subject: [R-meta] Computing var-covariance matrix with correlations of six
>>non-independent outcomes
>>
>>Hello,
>>
>>I am doing a meta-analysis looking at the effect of a teaching intervention
>>(versus control) on six types of motivation/behavioral regulation.
>>Theoretically and empirically these constructs form a continuum in which
>>the continuum neighbors are most strongly positively correlated and the
>>furthest from one another most negatively correlated.
>>
>>I have 95 effects. These effects come from 25 studies, each reporting
>>scores for between 1-6 motivation types. The number of effects per
>>motivation ranges from 22 to 13. In some studies, they have measured only
>>one or two types whereas in others they have measured 5 or all 6 types of
>>motivation.
>>
>>I originally ran a separate random-effects meta-analysis for all the six
>>outcomes. However, I got feedback that the dependency of the motivation
>>types should be taken into account and a 3-level meta-analysis was
>>recommended. After looking into it, the 3-level model seems to be a worse
>>approach than the multivariate approach.
>>
>>As is not usually the case, I have succeeded in gathering all correlations
>>between all the motivation types for all studies (some from original
>>reporting and some from previous meta-analysis findings).
>>
>>My question is, how do I compute the V-matrix for this data in order to run
>>the multivariate analysis? I read the whole archive but I could not find a
>>clear answer to the problem.
>>
>>Thank you very much in advance,
>>
>>Mika