[R-meta] weight in rmv metafor

Thu Jun 11 14:12:36 CEST 2020

Hi all, 

I read this discussion and one question came to my mind : I also had some studies that contributed multiple effect sizes in the meta-analysis that I recently ran thanks to Dr Viechtbauer's advice. 
For now I only used the rma function, but should I have used rma.mv because of these stuides that had multiple effect sizes ? 

Thank you ! 

Norman 

De: "James Pustejovsky" <jepusto using gmail.com> 
�: "Wolfgang Viechtbauer" <wolfgang.viechtbauer using maastrichtuniversity.nl> 
Cc: "r-sig-meta-analysis" <r-sig-meta-analysis using r-project.org>, "Huang Wu" <huang.wu using wmich.edu> 
Envoy�: Mercredi 10 Juin 2020 05:08:09 
Objet: Re: [R-meta] weight in rmv metafor 

Hi Huang, 

I've written up some notes that add a bit of further intuition to the 
discussion that Wolfgang provided. The main case that I focus on is a model 
that is just a meta-analysis (i.e., no predictors) and that includes random 
effects to capture both between-study and within-study heterogeneity. I 
also say a little bit about meta-regression models with only study-level 
predictors. 

https://www.jepusto.com/weighting-in-multivariate-meta-analysis/ 

Best, 
James 

On Sun, Jun 7, 2020 at 4:11 PM Viechtbauer, Wolfgang (SP) < 
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote: 

> Of course the weights "impact the estimated fixed effects". But whether 
> studies with multiple effect sizes tend to receive more weight depends on 
> various factors, including the variances of the random effects and the 
> sampling error (co)variances. 
> 
> A more detailed discussion around the way weighting works in rma.mv 
> models can be found here: 
> 
> http://www.metafor-project.org/doku.php/tips:weights_in_rma.mv_models 
> 
> Note that weights(res, type="rowsum") currently only works in the 'devel' 
> version of metafor, so follow 
> https://wviechtb.github.io/metafor/#installation if you want to reproduce 
> this part as well. 
> 
> I hope this clarifies things. 
> 
> Best, 
> Wolfgang 
> 
> >-----Original Message----- 
> >From: Huang Wu [mailto:huang.wu using wmich.edu] 
> >Sent: Sunday, 07 June, 2020 19:52 
> >To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org 
> >Subject: RE: [R-meta] weight in rmv metafor 
> > 
> >Dear Dr. Viechtbauer, 
> > 
> >Thank you very much for your helpful reply. 
> > 
> >To be clear, I wonder if the multivariate approach will downweight 
> estimates 
> >from a study that contains multiple effect sizes? 
> >I saw in a previous posts (https://stat.ethz.ch/pipermail/r-help/2017- 
> >February/444703.html), your said, "if you fit an appropriate model to the 
> >data at hand, the 'default weights' used by rma.mv() will be just fine." 
> >Does that mean that weights in rma.mv model would not impact the 
> estimated 
> >fixed effects? 
> >I found that in the forest plot I generate through forest(), studies with 
> >multiple effect sizes tend to have bigger weights. I also used weights() 
> to 
> >check the weights given to each effect sizes and found the same thing (see 
> >below for my code). I wonder if the weights for each effect sizes 
> presented 
> >in forest plot is correct? 
> > 
> >Thank you very much again for your help. 
> > 
> >Best wishes 
> >Huang 
> > 
> >Vt <- impute_covariance_matrix(vi = try$v, #known correlation vector 
> > cluster = try$ID, #study ID 
> > r = 0.80) #assumed correlation 
> > 
> >Mt <- rma.mv(yi=d, #effect size 
> > V = Vt, #variance (tHIS IS WHAt CHANGES FROM HEmodel) 
> > random = ~1 | ID/IID, #nesting structure 
> > test= "t", #use t-tests 
> > data=try, 
> > method="REML") 
> >weights(Mt) 
> > 
> >From: Viechtbauer, Wolfgang (SP) 
> >Sent: Sunday, June 7, 2020 6:56 AM 
> >To: Huang Wu; r-sig-meta-analysis using r-project.org 
> >Subject: RE: [R-meta] weight in rmv metafor 
> > 
> >Dear Huang, 
> > 
> >The weighting in rma.mv() models is more complex than in 'simple' models 
> >fitted with rma() (same as rma.uni()). Depending on the particular model 
> you 
> >are fitting with rma.mv(), the model-implied marginal var-cov matrix of 
> the 
> >estimates (which you can see with vcov(<model>, type="obs")) is not just a 
> >diagonal matrix (as is the case for rma() models), but also involves 
> >covariances. The inverse of this matrix is the weight matrix, which is 
> then 
> >also not just a diagonal matrix. 
> > 
> >For example, when some studies contribute multiple estimates, we might 
> >consider fitting a multilevel/multivariate model with random effects for 
> >studies and random effects for estimates within studies. When the 
> estimated 
> >between-study variance component is greater than zero, then this implies a 
> >certain amount of covariance for effects from the same study. This leads 
> to 
> >negative off-diagonal elements in the weight matrix for estimates from the 
> >same study. As a result, if the ith study contributes k_i estimates, it is 
> >not treated as if there were k_i independent studies. 
> > 
> >This has been discussed in the past on this mailing list, so you might 
> want 
> >to search the archives for some relevant posts. Googling for: 
> > 
> >site:https://stat.ethz.ch/pipermail/r-sig-meta-analysis/ rma.mv weights 
> > 
> >brings up some relevant posts. 
> > 
> >Roughly speaking, the robust variance estimation method works as follows. 
> We 
> >start with a 'working model' that is hopefully some decent approximation 
> to 
> >the true model and that also captures the dependencies in the estimates. 
> >This model provides us with the estimates of the fixed effects. However, 
> >because we might not be able to capture all dependencies correctly with 
> this 
> >working model, the var-cov matrix of the estimated fixed effects might not 
> >be correct. Hence, based on the working model, we can use the robust 
> >variance estimation method to obtain a var-cov matrix that is 
> >(asymptotically) correct and use this for testing the fixed effects. 
> > 
> >Therefore, the robust variance estimation method does not actually lead to 
> >changes in the estimated fixed effects. Those are determined based on the 
> >working model. That is why coef_test() will give you the exact same 
> >estimates of the fixed effects as those from the working model you use as 
> >input to this function. 
> > 
> >That is why it is important to use a working model that is at least some 
> >decent approximation. While the fixed effects estimates might even be 
> >unbiased when using a really poor working model, the estimates will not be 
> >very efficient. 
> > 
> >Best, 
> >Wolfgang 
> > 
> >>-----Original Message----- 
> >>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r- 
> >project.org] 
> >>On Behalf Of Huang Wu 
> >>Sent: Sunday, 07 June, 2020 0:37 
> >>To: r-sig-meta-analysis using r-project.org 
> >>Subject: [R-meta] weight in rmv metafor 
> >> 
> >>Hi, all, 
> >> 
> >>I am conducting a multivariate meta-analysis using rmv in metaphor 
> package. 
> >>I wonder how rmv calculate weights for each effect sizes? I wonder if 
> >>studies with more effect sizes get more total weights? 
> >> 
> >>I read an article saying "The robust variance estimation methods upweight 
> >>effect sizes that are estimated with greater precision (due to 
> differences 
> >>in sample sizes, level of randomization, predictive power of covariates, 
> >>etc.) and downweight estimates from studies that contribute multiple 
> effect 
> >>size estimates". (Kraft,Blazar, Hogan, 2018). Is that right? 
> >> 
> >>I am using rmv in metafor package to estimate the model and use coef_test 
> >in 
> >>sandwich package to do significance test. Both give the same pooled 
> effect 
> >>sizes though. I understand that weights also impact pooled effect size 
> >>estimate. In this case, how will robust variance estimation impact my 
> >weight 
> >>mean effect size? Thanks 
> >> 
> >>Best wishes 
> >>Huang 
> 
> _______________________________________________ 
> R-sig-meta-analysis mailing list 
> R-sig-meta-analysis using r-project.org 
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis 
> 

[[alternative HTML version deleted]] 

_______________________________________________ 
R-sig-meta-analysis mailing list 
R-sig-meta-analysis using r-project.org 
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis 

	[[alternative HTML version deleted]]