[R-meta] weight in rmv metafor

[James Pustejovsky]

Hi Huang,

I've written up some notes that add a bit of further intuition to the
discussion that Wolfgang provided. The main case that I focus on is a model
that is just a meta-analysis (i.e., no predictors) and that includes random
effects to capture both between-study and within-study heterogeneity. I
also say a little bit about meta-regression models with only study-level
predictors.

https://www.jepusto.com/weighting-in-multivariate-meta-analysis/

Best,
James

On Sun, Jun 7, 2020 at 4:11 PM Viechtbauer, Wolfgang (SP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Of course the weights "impact the estimated fixed effects". But whether
> studies with multiple effect sizes tend to receive more weight depends on
> various factors, including the variances of the random effects and the
> sampling error (co)variances.
>
> A more detailed discussion around the way weighting works in rma.mv
> models can be found here:
>
> http://www.metafor-project.org/doku.php/tips:weights_in_rma.mv_models
>
> Note that weights(res, type="rowsum") currently only works in the 'devel'
> version of metafor, so follow
> https://wviechtb.github.io/metafor/#installation if you want to reproduce
> this part as well.
>
> I hope this clarifies things.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: Huang Wu [mailto:huang.wu using wmich.edu]
> >Sent: Sunday, 07 June, 2020 19:52
> >To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> >Subject: RE: [R-meta] weight in rmv metafor
> >
> >Dear Dr. Viechtbauer,
> >
> >Thank you very much for your helpful reply.
> >
> >To be clear, I wonder if the multivariate approach will downweight
> estimates
> >from a study that contains multiple effect sizes?
> >I saw in a previous posts (https://stat.ethz.ch/pipermail/r-help/2017-
> >February/444703.html), your said, "if you fit an appropriate model to the
> >data at hand, the 'default weights' used by rma.mv() will be just fine."
> >Does that mean that weights in rma.mv model would not impact the
> estimated
> >fixed effects?
> >I found that in the forest plot I generate through forest(), studies with
> >multiple effect sizes tend to have bigger weights. I also used weights()
> to
> >check the weights given to each effect sizes and found the same thing (see
> >below for my code). I wonder if the weights for each effect sizes
> presented
> >in forest plot is correct?
> >
> >Thank you very much again for your help.
> >
> >Best wishes
> >Huang
> >
> >Vt <- impute_covariance_matrix(vi = try$v,  #known correlation vector
> >                                   cluster = try$ID, #study ID
> >                                  r = 0.80) #assumed correlation
> >
> >Mt <- rma.mv(yi=d, #effect size
> >                   V = Vt, #variance (tHIS IS WHAt CHANGES FROM HEmodel)
> >                   random = ~1 | ID/IID, #nesting structure
> >                   test= "t", #use t-tests
> >                   data=try,
> >                   method="REML")
> >weights(Mt)
> >
> >From: Viechtbauer, Wolfgang (SP)
> >Sent: Sunday, June 7, 2020 6:56 AM
> >To: Huang Wu; r-sig-meta-analysis using r-project.org
> >Subject: RE: [R-meta] weight in rmv metafor
> >
> >Dear Huang,
> >
> >The weighting in rma.mv() models is more complex than in 'simple' models
> >fitted with rma() (same as rma.uni()). Depending on the particular model
> you
> >are fitting with rma.mv(), the model-implied marginal var-cov matrix of
> the
> >estimates (which you can see with vcov(<model>, type="obs")) is not just a
> >diagonal matrix (as is the case for rma() models), but also involves
> >covariances. The inverse of this matrix is the weight matrix, which is
> then
> >also not just a diagonal matrix.
> >
> >For example, when some studies contribute multiple estimates, we might
> >consider fitting a multilevel/multivariate model with random effects for
> >studies and random effects for estimates within studies. When the
> estimated
> >between-study variance component is greater than zero, then this implies a
> >certain amount of covariance for effects from the same study. This leads
> to
> >negative off-diagonal elements in the weight matrix for estimates from the
> >same study. As a result, if the ith study contributes k_i estimates, it is
> >not treated as if there were k_i independent studies.
> >
> >This has been discussed in the past on this mailing list, so you might
> want
> >to search the archives for some relevant posts. Googling for:
> >
> >site:https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ rma.mv weights
> >
> >brings up some relevant posts.
> >
> >Roughly speaking, the robust variance estimation method works as follows.
> We
> >start with a 'working model' that is hopefully some decent approximation
> to
> >the true model and that also captures the dependencies in the estimates.
> >This model provides us with the estimates of the fixed effects. However,
> >because we might not be able to capture all dependencies correctly with
> this
> >working model, the var-cov matrix of the estimated fixed effects might not
> >be correct. Hence, based on the working model, we can use the robust
> >variance estimation method to obtain a var-cov matrix that is
> >(asymptotically) correct and use this for testing the fixed effects.
> >
> >Therefore, the robust variance estimation method does not actually lead to
> >changes in the estimated fixed effects. Those are determined based on the
> >working model. That is why coef_test() will give you the exact same
> >estimates of the fixed effects as those from the working model you use as
> >input to this function.
> >
> >That is why it is important to use a working model that is at least some
> >decent approximation. While the fixed effects estimates might even be
> >unbiased when using a really poor working model, the estimates will not be
> >very efficient.
> >
> >Best,
> >Wolfgang
> >
> >>-----Original Message-----
> >>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-
> >project.org]
> >>On Behalf Of Huang Wu
> >>Sent: Sunday, 07 June, 2020 0:37
> >>To: r-sig-meta-analysis using r-project.org
> >>Subject: [R-meta] weight in rmv metafor
> >>
> >>Hi, all,
> >>
> >>I am conducting a multivariate meta-analysis using rmv in metaphor
> package.
> >>I wonder how rmv calculate weights for each effect sizes? I wonder if
> >>studies with more effect sizes get more total weights?
> >>
> >>I read an article saying "The robust variance estimation methods upweight
> >>effect sizes that are estimated with greater precision (due to
> differences
> >>in sample sizes, level of randomization, predictive power of covariates,
> >>etc.) and downweight estimates from studies that contribute multiple
> effect
> >>size estimates". (Kraft,Blazar, Hogan, 2018). Is that right?
> >>
> >>I am using rmv in metafor package to estimate the model and use coef_test
> >in
> >>sandwich package to do significance test. Both give the same pooled
> effect
> >>sizes though. I understand that weights also impact pooled effect size
> >>estimate. In this case, how will robust variance estimation impact my
> >weight
> >>mean effect size? Thanks
> >>
> >>Best wishes
> >>Huang
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]