[R-meta] Computing var-covariance matrix with correlations of six non-independent outcomes

[Viechtbauer, Wolfgang (SP)]

I swear that James and I did not coordinate our responses. But this is seriously scary how similar our responses were.

>-----Original Message-----
>From: James Pustejovsky [mailto:jepusto using gmail.com]
>Sent: Tuesday, 07 July, 2020 22:41
>To: Viechtbauer, Wolfgang (SP)
>Cc: Mika Manninen; r-sig-meta-analysis using r-project.org
>Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
>six non-independent outcomes
>
>Ha! Following best practice for systematic reviews, a response to your
>question has been provided by two coders, working independently. I think our
>reliability is pretty decent too.
>
>On Tue, Jul 7, 2020 at 3:33 PM Viechtbauer, Wolfgang (SP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Mika,
>
>Strictly speaking, the correct way to compute the covariances between
>multiple Cohen's d (or Hedges' g) values would require using methods that
>are described in this chapter:
>
>Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In
>H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research
>synthesis and meta-analysis (2nd ed., pp. 357–376). New York: Russell Sage
>Foundation.
>
>In particular, equation (19.27) gives the covariance between two d values
>for two different outcomes computed based on the same sample of subjects
>(and we assume that the correlation between the two outcomes is the same for
>subjects in the treatment and the control group). An example illustrating an
>application of this is provided here:
>
>http://www.metafor-project.org/doku.php/analyses:gleser2009#multiple-
>endpoint_studies
>
>although the code for the example is a bit simpler because each study has
>exactly 2 d values.
>
>However, one can take a simpler approach that gives very similar results.
>Namely, if v_1 and v_2 are the sampling variances of d_1 and d_2 and r is
>the correlation between the two outcomes, then r * sqrt(v_1 * v_2) is
>usually a close approximation to what (19.27) would provide.
>
>Based on your example, this can be computed here as follows:
>
># this is only required because you coded missings as "NA" (and not as NA)
>corlist <- lapply(corlist, function(x) matrix(as.numeric(x), nrow=6,
>ncol=6))
>
># we only need a correlation matrix for the levels of motivation actually
>observed
>mots <- split(meta$motivation, meta$study)
>corlist <- mapply(function(rmat, mots) R[mots,mots], corlist, mots)
>corlist
>
>Vfun <- function(vi,rmat) {
>   if (length(vi) == 1L) {
>      matrix(vi)
>   } else {
>      S <- diag(sqrt(vi))
>      S %*% rmat %*% S
>   }
>}
>
>V <- mapply(Vfun, split(meta$v, meta$study), corlist)
>
>Now V is the var-cov matrix of your SMD values, so you could proceed with a
>multivariate model with:
>
>res <- rma.mv(g, V, mods = ~ factor(motivation) - 1, random = ~
>factor(motivation) | study, struct="UN", data=meta)
>res
>
>This is a very general model allowing each level of motivation to have its
>own heterogeneity variance component and each pair to have its own
>correlation. I wouldn't recommend this with only 8 studies, but with 25
>studies, this might be more sensible.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Mika Manninen [mailto:mixu89 using gmail.com]
>>Sent: Tuesday, 07 July, 2020 21:53
>>To: James Pustejovsky
>>Cc: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
>>Subject: Re: [R-meta] Computing var-covariance matrix with correlations of
>>six non-independent outcomes
>>
>>James,
>>
>>Thank you so much for replying.
>>
>>The dataset (8 studies out of 25 included) can be created with the
>following
>>code:
>>
>>###Creating vectors for the outcome, study, motivation, effect size, and
>>variance estimate
>>
>>#Total effects/outcomes
>>outcome <- c(1:28)
>>
>>#Study
>>study <- c(1,1,1,1,1,2,2,2,2,3,3,3,3,3,4,4,4,4,4,5,6,7,8,8,8,8,8,8)
>>
>>#Six different motivation types measured in the studies
>>motivation <-
>>c(1,3,4,5,6,1,3,4,5,1,3,4,5,6,1,3,4,5,6,6,6,6,1,2,3,4,5,6)
>>
>>#Hedges g:s
>>g <- c(0.6068, 0.0603, 0.2684, -0.0886, 0.0415, 1.592, 1.4031, 0.7928,
>>0.2013, 0.541, 0.1169,
>>       0.3129, -0.0275, -0.3536, 1.5886, 2.7218, -1.6273, -0.4375, -1.0695,
>>-0.1247, -0.1038,
>>       -0.2706, 0.2045, -0.2701, 0.3763, -0.7371, -0.0666, 0.2789)
>>
>>#Variance estimates
>>v <- c( 0.0162, 0.0155, 0.0157, 0.0155, 0.0155, 0.1889,
>>0.17875984,0.15484225,
>>        0.14432401, 0.0329, 0.0318, 0.0322, 0.0318, 0.0323, 0.1886, 0.2758,
>>        0.1909, 0.147, 0.164, 0.0067, 0.0028, 0.004, 0.0726, 0.0729,
>0.0735,
>>        0.0771, 0.0723, 0.073)
>>
>>###Dataset matrix with different levels and motivation types
>>meta <- cbind(outcome, study, motivation, g, v)
>>View(meta)
>>
>>I created 6x6 correlation matrices (with 15 unique correlations) for the
>>eight included studies. I am not sure if this is the most sensible approach
>>as only a few studies have measured all six outcomes. Value NA reflects the
>>absence of a correlation (absence of that one or both of the motivation
>>types in the study). The row and column numbers (1-6) correspond to the
>>variable motivation in the created dataset.
>>
>>### Correlation matrices for studies 1-8
>>
>>study1c <- rbind(c(1, "NA", .96, .34, -.33, -.66), c("NA", "NA", "NA",
>"NA",
>>"NA", "NA"), c(.96, "NA", 1, .55, -.12, -.50),
>>                 c(.34, "NA", .55, 1, .53, .05), c(-.33, "NA", -.12, .53,
>1,
>>.72), c(-.66, "NA", -.50, .05, .72, 1))
>>
>>study2c <- rbind(c(1, "NA", .96, .34, -.33, "NA"), c("NA", "NA", "NA",
>"NA",
>>"NA", "NA"), c(.96, "NA", 1, .55, -.12, "NA"),
>>                c(.34, "NA", .55, 1, .53, "NA"), c(-.33, "NA", -.12, .53,
>1,
>>"NA"), c("NA", "NA", "NA", "NA", "NA", "NA"))
>>
>>study3c <- rbind(c(1, "NA", .96, .34, -.33, -.66), c("NA", "NA", "NA",
>"NA",
>>"NA", "NA"), c(.96, "NA", 1, .55, -.12, -.50),
>>                 c(.34, "NA", .55, 1, .53, .05), c(-.33, "NA", -.12, .53,
>1,
>>.72), c(-.66, "NA", -.50, .05, .72, 1))
>>
>>study4c <- rbind(c(1, "NA", .85, .35, .06, -.44), c("NA", "NA", "NA", "NA",
>>"NA", "NA"), c(.85, "NA", 1, .59, .46, -.14),
>>                 c(.35, "NA", .59, 1, .71, .27), c(.06, "NA", .46, .71, 1,
>>.52), c(-.44, "NA", -.14, .27, .52, 1))
>>
>>study5c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>>
>>study6c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>>
>>study7c <- rbind(c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", "NA"),
>>                c("NA", "NA", "NA", "NA", "NA", "NA"), c("NA", "NA", "NA",
>>"NA", "NA", "NA"), c("NA", "NA", "NA", "NA", "NA", 1))
>>
>>study8c <- rbind(c(1, .79, .84, .24, -.20, -.50), c(.79, 1, .93, .47, .00,
>-
>>.31), c(.84, .93, 1, .57, -.07, -.52),
>>                    c(.24, .47, .57, 1, .43, .01), c(-.20, .00, -.07, .43,
>>1, .55), c(-.50, -.31, -.52, .01, .55, 1))
>>
>>#list with all the correlations matrices
>>
>>corlist <- list(study1c, study2c, study3c, study4c, study5c, study6c,
>>study7c, study8c)
>>
>>Mika
>>
>>ti 7. heinäk. 2020 klo 19.54 James Pustejovsky (jepusto using gmail.com)
>>kirjoitti:
>>Hi Mika,
>>
>>To add to Wolfgang's question, could you tell us a little bit more about
>how
>>you have structured the data on correlations between types of motivation?
>Is
>>it just one correlation matrix (6X6 matrix, with 1 + 2 + 3 + 4 + 5 = 15
>>unique correlations)? Or is it study-specific?
>>
>>This sort of calculation is a bit tricky to carry out so I am not surprised
>>that you haven't found a solution in the listserv archives. If you are
>>willing to share your dataset (or a subset thereof, say 3-4 studies worth
>of
>>data), it may make it easier for us to help problem solve.
>>
>>James
>>
>>On Tue, Jul 7, 2020 at 11:42 AM Viechtbauer, Wolfgang (SP)
>><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>>Dear Mika,
>>
>>What effect size measure are you using for the meta-analysis?
>>
>>Best,
>>Wolfgang
>>
>>>-----Original Message-----
>>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
>>project.org]
>>>On Behalf Of Mika Manninen
>>>Sent: Tuesday, 07 July, 2020 18:10
>>>To: r-sig-meta-analysis using r-project.org
>>>Subject: [R-meta] Computing var-covariance matrix with correlations of six
>>>non-independent outcomes
>>>
>>>Hello,
>>>
>>>I am doing a meta-analysis looking at the effect of a teaching
>intervention
>>>(versus control) on six types of motivation/behavioral regulation.
>>>Theoretically and empirically these constructs form a continuum in which
>>>the continuum neighbors are most strongly positively correlated and the
>>>furthest from one another most negatively correlated.
>>>
>>>I have 95 effects. These effects come from 25 studies, each reporting
>>>scores for between 1-6 motivation types. The number of effects per
>>>motivation ranges from 22 to 13. In some studies, they have measured only
>>>one or two types whereas in others they have measured 5 or all 6 types of
>>>motivation.
>>>
>>>I originally ran a separate random-effects meta-analysis for all the six
>>>outcomes. However, I got feedback that the dependency of the motivation
>>>types should be taken into account and a 3-level meta-analysis was
>>>recommended. After looking into it, the 3-level model seems to be a worse
>>>approach than the multivariate approach.
>>>
>>>As is not usually the case, I have succeeded in gathering all correlations
>>>between all the motivation types for all studies (some from original
>>>reporting and some from previous meta-analysis findings).
>>>
>>>My question is, how do I compute the V-matrix for this data in order to
>run
>>>the multivariate analysis? I read the whole archive but I could not find a
>>>clear answer to the problem.
>>>
>>>Thank you very much in advance,
>>>
>>>Mika