[R-meta] questions on some functions in metafor and clubsandwich

[James Pustejovsky]

Hi Brendan,

The option to "smooth" the sampling variances (i.e., averaging them
together across effect size estimates from the same sample) can be helpful
for two reasons. The main one (as discussed in the original RVE paper by
Hedges, Tipton, and Johnson, 2010) is that effect size estimates from the
same sample often tend to have very similar sampling variances, and the
main reason for differences in sampling variances could be effectively
random error in their estimation. Smoothing them out within a given sample
might therefore cut down on the random error in the sampling variance
estimates. Further, if inference is based on RVE, then we don't need
sampling variances that are exactly correct anyways, so we have a fair
amount of "wiggle room" here.

A secondary reason that smoothing can be helpful is that it avoids some
weird behavior that can happen when you use inverse-variance weights (which
is what we usually do) and a correlated effect structure with *dis-similar*
sampling variances. If the sampling variances of the effect size estimates
from a given sample are far from equal, then you can end up in a situation
where the effect sizes with the largest sampling variances end up getting
*negative* weight in the overall meta-analysis. I gave an example of this
recently in the context of aggregating effect sizes prior to analysis:
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2022-January/003728.html
But effectively the same thing can happen also implicitly in a
meta-analytic model.

James

On Wed, Feb 9, 2022 at 6:49 AM Brendan Hutchinson <
Brendan.Hutchinson using anu.edu.au> wrote:

> Dear Wolfgang,
>
> Thank you very much for your quick response! Your responses are very
> helpful and appreciated.
>
> In relation to the second question, this is precisely what I thought it
> might be doing. However, I'm still a bit confused. To be more precise, if
> you examine this code sample from Puchevosky et al 2021 (
> https://osf.io/z27wt/), in particular the CHE model, they have set
> smooth_VI to true and specified a random effects model with effect sizes
> nested within studies. This is what is confusing me - would you not wish to
> retain the differences in sampling variance in such a model, rather than
> setting them all to the average?
>
> Best,
> Brendan
>
>
> Brendan Hutchinson
> Research School of Psychology
> ANU College of Medicine, Biology and Environment
> Building 39 University Ave | The Australian National University | ACTON
> ACT 2601 Australia
> T: +61 2 6125 2716 | E: brendan.hutchinson using anu.edu.au | W:  Brendan
> Hutchinson | ANU Research School of Psychology<
> https://psychology.anu.edu.au/people/students/brendan-hutchinson>
>
> ________________________________
> From: Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl>
> Sent: Wednesday, 9 February 2022 7:06 PM
> To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>;
> r-sig-meta-analysis using r-project.org <r-sig-meta-analysis using r-project.org>
> Subject: RE: [R-meta] questions on some functions in metafor and
> clubsandwich
>
> Dear Brendan,
>
> Please see below.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Brendan Hutchinson
> >Sent: Wednesday, 09 February, 2022 7:54
> >To: r-sig-meta-analysis using r-project.org
> >Subject: [R-meta] questions on some functions in metafor and clubsandwich
> >
> > Hi mailing list,
> >
> >Thanks in advance for any help regarding my questions - I have two and
> they
> >concern the metafor and clubsandwich packages, and multilevel modelling.
> >
> >1. My first question concerns the difference between the robust()
> function in
> >metafor and the coef_test() function in clubsandwich - I'm a little
> confused as
> >to the precise difference between these. Do they not perform the same
> operation?
> >Is there any situations in which one would be preferred over another?
>
> coef_test() in itself is just a function for testing coefficients. The
> real difference between robust() and clubSandwich is the kind of
> adjustments they provide for the var-cov matrix and how they estimate the
> dfs. Note that metafor can now directly interface with clubSandwich. See:
>
> See:
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwviechtb.github.io%2Fmetafor%2Freference%2Frobust.html&data=04%7C01%7CBrendan.Hutchinson%40anu.edu.au%7Cd1e97df0aafc4e31775b08d9eba312dc%7Ce37d725cab5c46249ae5f0533e486437%7C0%7C0%7C637799907910897611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bySiP2DTn0vKfszQhiKLSKzHIOMofOOC8N5X3rvs0k0%3D&reserved=0
>
> >2. Second, in order to control for correlated effect sizes and correlated
> >sampling variance in my own dataset, I will need to produce a
> variance-covariance
> >matrix for my data using the impute_covariance_matrix() function in
> clubsandwich,
> >which will then be fed into a multilevel model (effect sizes nested within
> >studies) specified in the metafor function rma.mv().
> >
> >My question here concerns the "smooth_vi" input of the
> impute_covariance_matrix()
> >function. I am a little unclear as to its use. The help page specifies "If
> >smooth_vi = TRUE, then all of the variances within cluster j will be set
> equal to
> >the average variance of cluster j".
> >
> >I interpreted this as though it is simply removing variance within
> clusters (i.e.
> >studies) via averaging, which I suspect would be inappropriate for a
> multi-level
> >meta-analysis in which we would want to capture that variance - indeed,
> is this
> >not the reason we specify a multilevel structure in the first place? What
> is
> >confusing to me is the only example code I have seen online appears to set
> >smooth_VI to true when specifying a multi-level model (in which effects
> are
> >nested within studies), so I am a little lost.
>
> I think you are misunderstanding this option. Say you have two effect
> sizes with sampling variances equal to .01 and .03 within a cluster. Then
> with smooth_vi=TRUE, the sampling variances would be set to .02 and .02 for
> the two estimates.
>
> >Once again, any help on the above is greatly appreciated!
> >
> >Brendan
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]