[R-meta] questions on some functions in metafor and clubsandwich

Thu Feb 17 05:32:11 CET 2022

Hi Fred,

Your intuition here is spot on, although the weight calculations are a bit
more complicated with this model. Using weights(z) returns only the
diagonal entries of the weight matrix, but rma.mv models can have
off-diagonal entries too. For a model with separate intercepts, I would
approach the weight calculations as follows:

dat <- read.csv("https://raw.githubusercontent.com/ilzl/i/master/j.csv")
V <- with(dat, impute_covariance_matrix(vi, study, .6))
z <- rma.mv(yi ~ 0 + factor(outcome), V, random = ~ 1 |study/es, data = dat)
W <- weights(z, type="matrix")
X <- model.matrix(z)
WX <- W %*% X
B <- solve(t(X) %*% WX)
weighting_mat <- WX %*% B

The result is a matrix with rows corresponding to each observation and
columns for each coefficient in the model (so here, each level of
factor(outcome). A negative entry in weighting_mat[i,j] means that
observation i gets negative weight in calculating the estimate of
coefficient j.

For more on the details behind these calculations, see
https://www.metafor-project.org/doku.php/tips:weights_in_rma.mv_models

James

On Wed, Feb 16, 2022 at 7:34 PM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:

> Hi James,
>
> Thank you for the clarification. Per your insightful response, I would
> like to run some sanity checks on my model to make sure that my use of
> a single within-study r has not led to weird stuff happening in my
> data.
>
> Are the following sufficient to capture the occurrence of weird stuff?
> (I'm sure this will be useful to a lot of the list members.)
>
> Many thanks,
> Fred
>
> dat <- read.csv("https://raw.githubusercontent.com/ilzl/i/master/j.csv")
>
> V <- with(dat, impute_covariance_matrix(vi, study, .6))
>
> z <- rma.mv(yi ~ factor(outcome), V, random = ~ 1 |study/es, data = dat)
>
> any(weights(z) < 0)
>
> any(z$b[,1] < min(dat$yi, na.rm = TRUE))
>
> any(z$b[,1] > max(dat$yi, na.rm = TRUE))
>
>
> On Tue, Feb 15, 2022 at 4:27 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
> >
> > Hi Fred,
> >
> > Unfortunately, we did not really get into the details behind smoothing
> the sampling variances in that paper (mainly due to page restrictions--even
> as written, we were well over the recommended page count for that journal).
> >
> > If you are using an imputed covariance matrix with model-based standard
> errors/inferential results, then the key thing is to make the assumptions
> as realistic and defensible as is feasible. If your clusters of correlated
> effect size estimates arise from having multiple measures of a common
> outcome construct or set of constructs, then I would guess that using
> smooth_vi = TRUE will usually be pretty reasonable (because the sampling
> variances from a given study are probably all quite similar anyways, so
> averaging them together won't really change much).
> >
> > For more complex cases, such as where you have multiple measures of a
> common outcome, each assessed at several points in time, with multiple
> treatment groups compared to a common control group, then I would want to
> be more cautious about smoothing the variances and, generally, more
> cautious in constructing the imputed covariance matrix, such as by using
> the new vcalc() function in metafor (
> https://wviechtb.github.io/metafor/reference/vcalc.html).
> >
> > My comments about "weird stuff happening when using inverse-variance
> weights and a correlated effect structure" pertain to what happens with the
> _weights_ assigned to each effect size estimate. Thus, they're relevant
> both to model-based and robust inference approaches.
> >
> > James
> >
> >
> > On Wed, Feb 9, 2022 at 9:45 PM Farzad Keyhan <f.keyhaniha using gmail.com>
> wrote:
> >>
> >> Dear James,
> >>
> >> Thanks for this information. Did you possibly reflect on/emphasize
> >> this in your paper [https://doi.org/10.1007/s11121-021-01246-3]?
> >>
> >> I ask this for two reasons.
> >>
> >> First, some folks may not want to apply an RVE after fitting an
> >> rma.mv() call and instead use the model-based results (i.e., they
> >> solely want to account for their correlated errors).
> >>
> >> Second, some folks cannot apply an RVE after fitting an rma.mv() call
> >> because their model contains a pair of random-effects that are crossed
> >> with each other, but still want to account for their correlated
> >> errors.
> >>
> >> Should we possibly be concerned about our final results when using
> >> somooth_vi = TRUE, if we fall into these two categories?
> >>
> >> Many thanks for your attention,
> >> Fred
> >>
> >> On Wed, Feb 9, 2022 at 8:58 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
> >> >
> >> > Hi Brendan,
> >> >
> >> > The option to "smooth" the sampling variances (i.e., averaging them
> >> > together across effect size estimates from the same sample) can be
> helpful
> >> > for two reasons. The main one (as discussed in the original RVE paper
> by
> >> > Hedges, Tipton, and Johnson, 2010) is that effect size estimates from
> the
> >> > same sample often tend to have very similar sampling variances, and
> the
> >> > main reason for differences in sampling variances could be effectively
> >> > random error in their estimation. Smoothing them out within a given
> sample
> >> > might therefore cut down on the random error in the sampling variance
> >> > estimates. Further, if inference is based on RVE, then we don't need
> >> > sampling variances that are exactly correct anyways, so we have a fair
> >> > amount of "wiggle room" here.
> >> >
> >> > A secondary reason that smoothing can be helpful is that it avoids
> some
> >> > weird behavior that can happen when you use inverse-variance weights
> (which
> >> > is what we usually do) and a correlated effect structure with
> *dis-similar*
> >> > sampling variances. If the sampling variances of the effect size
> estimates
> >> > from a given sample are far from equal, then you can end up in a
> situation
> >> > where the effect sizes with the largest sampling variances end up
> getting
> >> > *negative* weight in the overall meta-analysis. I gave an example of
> this
> >> > recently in the context of aggregating effect sizes prior to analysis:
> >> >
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-January/003728.html
> >> > But effectively the same thing can happen also implicitly in a
> >> > meta-analytic model.
> >> >
> >> > James
> >> >
> >> > On Wed, Feb 9, 2022 at 6:49 AM Brendan Hutchinson <
> >> > Brendan.Hutchinson using anu.edu.au> wrote:
> >> >
> >> > > Dear Wolfgang,
> >> > >
> >> > > Thank you very much for your quick response! Your responses are very
> >> > > helpful and appreciated.
> >> > >
> >> > > In relation to the second question, this is precisely what I
> thought it
> >> > > might be doing. However, I'm still a bit confused. To be more
> precise, if
> >> > > you examine this code sample from Puchevosky et al 2021 (
> >> > > https://osf.io/z27wt/), in particular the CHE model, they have set
> >> > > smooth_VI to true and specified a random effects model with effect
> sizes
> >> > > nested within studies. This is what is confusing me - would you not
> wish to
> >> > > retain the differences in sampling variance in such a model, rather
> than
> >> > > setting them all to the average?
> >> > >
> >> > > Best,
> >> > > Brendan
> >> > >
> >> > >
> >> > > Brendan Hutchinson
> >> > > Research School of Psychology
> >> > > ANU College of Medicine, Biology and Environment
> >> > > Building 39 University Ave | The Australian National University |
> ACTON
> >> > > ACT 2601 Australia
> >> > > T: +61 2 6125 2716 | E: brendan.hutchinson using anu.edu.au | W:  Brendan
> >> > > Hutchinson | ANU Research School of Psychology<
> >> > > https://psychology.anu.edu.au/people/students/brendan-hutchinson>
> >> > >
> >> > > ________________________________
> >> > > From: Viechtbauer, Wolfgang (SP) <
> >> > > wolfgang.viechtbauer using maastrichtuniversity.nl>
> >> > > Sent: Wednesday, 9 February 2022 7:06 PM
> >> > > To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>;
> >> > > r-sig-meta-analysis using r-project.org <
> r-sig-meta-analysis using r-project.org>
> >> > > Subject: RE: [R-meta] questions on some functions in metafor and
> >> > > clubsandwich
> >> > >
> >> > > Dear Brendan,
> >> > >
> >> > > Please see below.
> >> > >
> >> > > Best,
> >> > > Wolfgang
> >> > >
> >> > > >-----Original Message-----
> >> > > >From: R-sig-meta-analysis [mailto:
> >> > > r-sig-meta-analysis-bounces using r-project.org] On
> >> > > >Behalf Of Brendan Hutchinson
> >> > > >Sent: Wednesday, 09 February, 2022 7:54
> >> > > >To: r-sig-meta-analysis using r-project.org
> >> > > >Subject: [R-meta] questions on some functions in metafor and
> clubsandwich
> >> > > >
> >> > > > Hi mailing list,
> >> > > >
> >> > > >Thanks in advance for any help regarding my questions - I have two
> and
> >> > > they
> >> > > >concern the metafor and clubsandwich packages, and multilevel
> modelling.
> >> > > >
> >> > > >1. My first question concerns the difference between the robust()
> >> > > function in
> >> > > >metafor and the coef_test() function in clubsandwich - I'm a little
> >> > > confused as
> >> > > >to the precise difference between these. Do they not perform the
> same
> >> > > operation?
> >> > > >Is there any situations in which one would be preferred over
> another?
> >> > >
> >> > > coef_test() in itself is just a function for testing coefficients.
> The
> >> > > real difference between robust() and clubSandwich is the kind of
> >> > > adjustments they provide for the var-cov matrix and how they
> estimate the
> >> > > dfs. Note that metafor can now directly interface with
> clubSandwich. See:
> >> > >
> >> > > See:
> >> > >
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwviechtb.github.io%2Fmetafor%2Freference%2Frobust.html&data=04%7C01%7CBrendan.Hutchinson%40anu.edu.au%7Cd1e97df0aafc4e31775b08d9eba312dc%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637799907910897611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bySiP2DTn0vKfszQhiKLSKzHIOMofOOC8N5X3rvs0k0%3D&reserved=0
> >> > >
> >> > > >2. Second, in order to control for correlated effect sizes and
> correlated
> >> > > >sampling variance in my own dataset, I will need to produce a
> >> > > variance-covariance
> >> > > >matrix for my data using the impute_covariance_matrix() function in
> >> > > clubsandwich,
> >> > > >which will then be fed into a multilevel model (effect sizes
> nested within
> >> > > >studies) specified in the metafor function rma.mv().
> >> > > >
> >> > > >My question here concerns the "smooth_vi" input of the
> >> > > impute_covariance_matrix()
> >> > > >function. I am a little unclear as to its use. The help page
> specifies "If
> >> > > >smooth_vi = TRUE, then all of the variances within cluster j will
> be set
> >> > > equal to
> >> > > >the average variance of cluster j".
> >> > > >
> >> > > >I interpreted this as though it is simply removing variance within
> >> > > clusters (i.e.
> >> > > >studies) via averaging, which I suspect would be inappropriate for
> a
> >> > > multi-level
> >> > > >meta-analysis in which we would want to capture that variance -
> indeed,
> >> > > is this
> >> > > >not the reason we specify a multilevel structure in the first
> place? What
> >> > > is
> >> > > >confusing to me is the only example code I have seen online
> appears to set
> >> > > >smooth_VI to true when specifying a multi-level model (in which
> effects
> >> > > are
> >> > > >nested within studies), so I am a little lost.
> >> > >
> >> > > I think you are misunderstanding this option. Say you have two
> effect
> >> > > sizes with sampling variances equal to .01 and .03 within a
> cluster. Then
> >> > > with smooth_vi=TRUE, the sampling variances would be set to .02 and
> .02 for
> >> > > the two estimates.
> >> > >
> >> > > >Once again, any help on the above is greatly appreciated!
> >> > > >
> >> > > >Brendan
> >> > >
> >> > >         [[alternative HTML version deleted]]
> >> > >
> >> > > _______________________________________________
> >> > > R-sig-meta-analysis mailing list
> >> > > R-sig-meta-analysis using r-project.org
> >> > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >> > >
> >> >
> >> >         [[alternative HTML version deleted]]
> >> >
> >> > _______________________________________________
> >> > R-sig-meta-analysis mailing list
> >> > R-sig-meta-analysis using r-project.org
> >> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]