[R-meta] questions on some functions in metafor and clubsandwich

Tue Feb 15 23:27:44 CET 2022

Hi Fred,

Unfortunately, we did not really get into the details behind smoothing the
sampling variances in that paper (mainly due to page restrictions--even as
written, we were well over the recommended page count for that journal).

If you are using an imputed covariance matrix with model-based standard
errors/inferential results, then the key thing is to make the assumptions
as realistic and defensible as is feasible. If your clusters of correlated
effect size estimates arise from having multiple measures of a common
outcome construct or set of constructs, then I would guess that using
smooth_vi = TRUE will usually be pretty reasonable (because the sampling
variances from a given study are probably all quite similar anyways, so
averaging them together won't really change much).

For more complex cases, such as where you have multiple measures of a
common outcome, each assessed at several points in time, with multiple
treatment groups compared to a common control group, then I would want to
be more cautious about smoothing the variances and, generally, more
cautious in constructing the imputed covariance matrix, such as by using
the new vcalc() function in metafor (
https://wviechtb.github.io/metafor/reference/vcalc.html).

My comments about "weird stuff happening when using inverse-variance
weights and a correlated effect structure" pertain to what happens with the
_weights_ assigned to each effect size estimate. Thus, they're relevant
both to model-based and robust inference approaches.

James

On Wed, Feb 9, 2022 at 9:45 PM Farzad Keyhan <f.keyhaniha using gmail.com> wrote:

> Dear James,
>
> Thanks for this information. Did you possibly reflect on/emphasize
> this in your paper [https://doi.org/10.1007/s11121-021-01246-3]?
>
> I ask this for two reasons.
>
> First, some folks may not want to apply an RVE after fitting an
> rma.mv() call and instead use the model-based results (i.e., they
> solely want to account for their correlated errors).
>
> Second, some folks cannot apply an RVE after fitting an rma.mv() call
> because their model contains a pair of random-effects that are crossed
> with each other, but still want to account for their correlated
> errors.
>
> Should we possibly be concerned about our final results when using
> somooth_vi = TRUE, if we fall into these two categories?
>
> Many thanks for your attention,
> Fred
>
> On Wed, Feb 9, 2022 at 8:58 PM James Pustejovsky <jepusto using gmail.com>
> wrote:
> >
> > Hi Brendan,
> >
> > The option to "smooth" the sampling variances (i.e., averaging them
> > together across effect size estimates from the same sample) can be
> helpful
> > for two reasons. The main one (as discussed in the original RVE paper by
> > Hedges, Tipton, and Johnson, 2010) is that effect size estimates from the
> > same sample often tend to have very similar sampling variances, and the
> > main reason for differences in sampling variances could be effectively
> > random error in their estimation. Smoothing them out within a given
> sample
> > might therefore cut down on the random error in the sampling variance
> > estimates. Further, if inference is based on RVE, then we don't need
> > sampling variances that are exactly correct anyways, so we have a fair
> > amount of "wiggle room" here.
> >
> > A secondary reason that smoothing can be helpful is that it avoids some
> > weird behavior that can happen when you use inverse-variance weights
> (which
> > is what we usually do) and a correlated effect structure with
> *dis-similar*
> > sampling variances. If the sampling variances of the effect size
> estimates
> > from a given sample are far from equal, then you can end up in a
> situation
> > where the effect sizes with the largest sampling variances end up getting
> > *negative* weight in the overall meta-analysis. I gave an example of this
> > recently in the context of aggregating effect sizes prior to analysis:
> >
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-January/003728.html
> > But effectively the same thing can happen also implicitly in a
> > meta-analytic model.
> >
> > James
> >
> > On Wed, Feb 9, 2022 at 6:49 AM Brendan Hutchinson <
> > Brendan.Hutchinson using anu.edu.au> wrote:
> >
> > > Dear Wolfgang,
> > >
> > > Thank you very much for your quick response! Your responses are very
> > > helpful and appreciated.
> > >
> > > In relation to the second question, this is precisely what I thought it
> > > might be doing. However, I'm still a bit confused. To be more precise,
> if
> > > you examine this code sample from Puchevosky et al 2021 (
> > > https://osf.io/z27wt/), in particular the CHE model, they have set
> > > smooth_VI to true and specified a random effects model with effect
> sizes
> > > nested within studies. This is what is confusing me - would you not
> wish to
> > > retain the differences in sampling variance in such a model, rather
> than
> > > setting them all to the average?
> > >
> > > Best,
> > > Brendan
> > >
> > >
> > > Brendan Hutchinson
> > > Research School of Psychology
> > > ANU College of Medicine, Biology and Environment
> > > Building 39 University Ave | The Australian National University | ACTON
> > > ACT 2601 Australia
> > > T: +61 2 6125 2716 | E: brendan.hutchinson using anu.edu.au | W:  Brendan
> > > Hutchinson | ANU Research School of Psychology<
> > > https://psychology.anu.edu.au/people/students/brendan-hutchinson>
> > >
> > > ________________________________
> > > From: Viechtbauer, Wolfgang (SP) <
> > > wolfgang.viechtbauer using maastrichtuniversity.nl>
> > > Sent: Wednesday, 9 February 2022 7:06 PM
> > > To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>;
> > > r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> > > Subject: RE: [R-meta] questions on some functions in metafor and
> > > clubsandwich
> > >
> > > Dear Brendan,
> > >
> > > Please see below.
> > >
> > > Best,
> > > Wolfgang
> > >
> > > >-----Original Message-----
> > > >From: R-sig-meta-analysis [mailto:
> > > r-sig-meta-analysis-bounces using r-project.org] On
> > > >Behalf Of Brendan Hutchinson
> > > >Sent: Wednesday, 09 February, 2022 7:54
> > > >To: r-sig-meta-analysis using r-project.org
> > > >Subject: [R-meta] questions on some functions in metafor and
> clubsandwich
> > > >
> > > > Hi mailing list,
> > > >
> > > >Thanks in advance for any help regarding my questions - I have two and
> > > they
> > > >concern the metafor and clubsandwich packages, and multilevel
> modelling.
> > > >
> > > >1. My first question concerns the difference between the robust()
> > > function in
> > > >metafor and the coef_test() function in clubsandwich - I'm a little
> > > confused as
> > > >to the precise difference between these. Do they not perform the same
> > > operation?
> > > >Is there any situations in which one would be preferred over another?
> > >
> > > coef_test() in itself is just a function for testing coefficients. The
> > > real difference between robust() and clubSandwich is the kind of
> > > adjustments they provide for the var-cov matrix and how they estimate
> the
> > > dfs. Note that metafor can now directly interface with clubSandwich.
> See:
> > >
> > > See:
> > >
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwviechtb.github.io%2Fmetafor%2Freference%2Frobust.html&data=04%7C01%7CBrendan.Hutchinson%40anu.edu.au%7Cd1e97df0aafc4e31775b08d9eba312dc%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637799907910897611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bySiP2DTn0vKfszQhiKLSKzHIOMofOOC8N5X3rvs0k0%3D&reserved=0
> > >
> > > >2. Second, in order to control for correlated effect sizes and
> correlated
> > > >sampling variance in my own dataset, I will need to produce a
> > > variance-covariance
> > > >matrix for my data using the impute_covariance_matrix() function in
> > > clubsandwich,
> > > >which will then be fed into a multilevel model (effect sizes nested
> within
> > > >studies) specified in the metafor function rma.mv().
> > > >
> > > >My question here concerns the "smooth_vi" input of the
> > > impute_covariance_matrix()
> > > >function. I am a little unclear as to its use. The help page
> specifies "If
> > > >smooth_vi = TRUE, then all of the variances within cluster j will be
> set
> > > equal to
> > > >the average variance of cluster j".
> > > >
> > > >I interpreted this as though it is simply removing variance within
> > > clusters (i.e.
> > > >studies) via averaging, which I suspect would be inappropriate for a
> > > multi-level
> > > >meta-analysis in which we would want to capture that variance -
> indeed,
> > > is this
> > > >not the reason we specify a multilevel structure in the first place?
> What
> > > is
> > > >confusing to me is the only example code I have seen online appears
> to set
> > > >smooth_VI to true when specifying a multi-level model (in which
> effects
> > > are
> > > >nested within studies), so I am a little lost.
> > >
> > > I think you are misunderstanding this option. Say you have two effect
> > > sizes with sampling variances equal to .01 and .03 within a cluster.
> Then
> > > with smooth_vi=TRUE, the sampling variances would be set to .02 and
> .02 for
> > > the two estimates.
> > >
> > > >Once again, any help on the above is greatly appreciated!
> > > >
> > > >Brendan
> > >
> > >         [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-meta-analysis mailing list
> > > R-sig-meta-analysis using r-project.org
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> > >
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list
> > R-sig-meta-analysis using r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]