[R-meta] Negative Weights in a Multivariate Meta-Analysis

[Viechtbauer, Wolfgang (NP)]

Dear Martin,

In principle, res.1 is correct from a purely 'mathematical/statistical' perspective, but it's just very non-intuitive and I would also consider some other approach to avoid this issue.

As a sort-of compromise, you could use:

res.3 <- rma.mv(yi, V=V_list.1, W=1/(vi*rep(c(3,4), c(3,4))), method="FE", data=dat)
res.3

This uses inverse-variance weights that are adjusted based on the number of estimates within each study. That way, a study with many estimates would not overwhelm the other studies. In computing the standard error of the pooled estimate, this method still accounts for the dependency in the estimates via the V matrix.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Prof. Dr. Martin Brunner
>Sent: Friday, 26 August, 2022 11:56
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Negative Weights in a Multivariate Meta-Analysis
>
>Dear listmembers,
>I would like to use a multivariate fixed-effects model to estimate an
>average (true) effect size across several dependent effect sizes. The
>dependency among effect sizes arises because these effect sizes were
>obtained for the same sample of individuals within studies.
>To set up my meta-analytic model I use the impute_covariance_matrix function
>from the clubsandwich package assuming that the effect sizes correlate with
>r = .70 within studies (see below).
>The potential problem is now that the average effect size that is estimated
>by the fixed-effects model is smaller than any of the observed effect sizes.
>This does not seem plausible to me. The reason is that some effect sizes
>receive negative weights.
>In his great blog post James (Pustejovsky) anticipated such issues (see
>https://www.jepusto.com/weighting-in-multivariate-meta-analysis/):
>Specifically, he wrote: "One wrinkle with the more general form of the
>weights is that the effect-size level weights can sometimes be negative
>(i.e., negative ). This will tend to happen when the sampling variances
>within a study are discrepant, such as when one is much smaller than the
>others in study , when the (assumed or estimated) sampling correlation is
>high, and when is zero or small. This is something that warrants further
>investigation."
>Does anyone have some thougths how one can deal with such situations? For
>example, I could imagine using a different weighting scheme where the
>within-study sampling variances and covariances are assumed to be the same
>(i.e., setting smooth_vi to TRUE). The good thing is that the average effect
>size seems now much more plausible to me. However, using this approach
>implies that each effect size estimate that was observed for a certain study
>receives the same weight which is not well aligned with the idea of using
>inverse variance weights.
>Any ideas are highly appreciated.
>Best,
>Martin
>
>Here is a reprocible R syntax code.
>
>library(clubSandwich)
>library(metafor)
>
>yi <- c(0.27689888, 0.62431603 ,0.09817025, 0.28281538, 0.26482706
>,0.17486036, 0.15944605)
>vi <- c(0.003103681 ,0.012500222, 0.001569756, 0.003748064, 0.005302781
>,0.002147452, 0.004888748)
>study <- c(rep(1,3), rep(2,4))
>
>dat <- data.frame(cbind(yi,vi,study))
>
>V_list.1 <- impute_covariance_matrix(vi = dat$vi, cluster = dat$study,r =
>.7,smooth_vi = FALSE)
>V_list.1
>
>V_list.2 <- impute_covariance_matrix(vi = dat$vi, cluster = dat$study,r =
>.7,smooth_vi = TRUE)
>V_list.2
>
>res.1 <- rma.mv(yi, V=V_list.1, method="FE", sparse=TRUE, data=dat)
>res.1
>
>W.1 <- weights(res.1, type="matrix")
>rowSums(W.1)
>sum(rowSums(W.1) * yi) / sum(W.1)
>
>res.2 <- rma.mv(yi, V=V_list.2, method="FE", sparse=TRUE, data=dat)
>res.2
>
>W.2 <- weights(res.2, type="matrix")
>rowSums(W.2)
>sum(rowSums(W.2) * yi) / sum(W.2)
>
>par(mfrow=c(1,2))
>
>forest(res.1, slab=c(paste("ES",seq(1:length(yi)))), showweights=TRUE)
>title("rho=0.7, smooth_vi=FALSE")
>
>forest(res.2, slab=c(paste("ES",seq(1:length(yi)))), showweights=TRUE)
>title("rho=0.7, smooth_vi=TRUE")