[R-meta] Egger-type test for multi-level meta-analysis

[James Pustejovsky]

Hi Wolfgang,

Very much agree. My understanding of Egger/PET/PEESE in the multilevel
setting is that they are still essentially tests/adjustments for "small
study effects." Typically, the SEs of effect sizes from a given study will
be quite similar to each other, so sqrt(vi) is mostly going to vary at the
study level rather than within study. As a result, Egger's test here is
really looking at the correlation between the study's SE (mostly a function
of sample size and study design) and the *study-level average effect size
estimate*. Using different working models will change the weighting of that
meta-regression, but it's still almost entirely about study-level stuff.
Thus, these tests/adjustments are only really going to be sensitive to
forms of selective reporting that affect the study-level average effect.

I like to think of these tests/adjustments as kind of "agnostic" to the
specific form of selective reporting. The best they can do is flag a fishy
pattern (i.e., correlation between study size and average ES) and then it's
up to the analyst to decide whether it's something to worry about or not.
But since these methods aren't based on a specific generative model, they
can't really tell us much about *how* selective reporting might be
happening or the specific *degree* of selective reporting in the domain
under study. In contrast, selection models (like Vevea-Hedges selection
models or the Copas-type models) are more descriptive and specific about
selective reporting mechanisms.

James

On Tue, Jun 20, 2023 at 7:51 AM Viechtbauer, Wolfgang (NP) via
R-sig-meta-analysis <r-sig-meta-analysis using r-project.org> wrote:

> Just me thinking out loud here for the moment:
>
> While the extension of the 'Egger regression test' (and the PET/PEESE
> methods) to multilevel models is straightforward, a bit of thinking is
> required as to what we are really trying to capture by adding something
> like sqrt(vi) (or just vi or any other transformation thereof) to the model
> as a predictor. Selective reporting within studies of the larger /
> significant effects? Or selective reporting of studies in general? In the
> latter case, selection may depends on whether at least one effect is
> significant / large, the focal one (in case there is a defined primary
> endpoint), or something else. One could argue that cor(sqrt(vi), yi) (in
> essence what we are examining with the regression test) might capture a bit
> of all of this and maybe that's the best we can generally do.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of James Pustejovsky via R-sig-meta-analysis
> >Sent: Monday, 19 June, 2023 18:23
> >To: R Special Interest Group for Meta-Analysis
> >Cc: James Pustejovsky
> >Subject: Re: [R-meta] Egger-type test for multi-level meta-analysis
> >
> >Correct. The slope on sqrt(vi) indicates an association between SE and
> >effect size. The intercept is a "PET"-style estimate of the average effect
> >size in a population of infinitely large studies (i.e. SE = 0).
> >
> >James
> >
> >On Mon, Jun 19, 2023 at 9:21 AM Dr. Guido Schwarzer via
> R-sig-meta-analysis
> ><r-sig-meta-analysis using r-project.org> wrote:
> >
> >> Hi all,
> >>
> >> Another question on multi-level models (while I am still waiting for an
> >> answer on my previous one ;-) ).
> >>
> >> I would like to conduct a test for small-study effects for data from a
> >> three-level model, e.g., for the dataset dat.konstantopoulos2011.
> >>
> >> library("metafor")
> >> library("metadat")
> >> m.ml <- rma.mv(yi, vi, random = ~ 1 | district / school, data =
> >> dat.konstantopoulos2011)
> >>
> >> If I understand James' comment from February 2018 correctly (
> >>
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-February/000610.html
> ),
> >> I could conduct an Egger-type test for small study effects by using
> >> cluster-robust variance estimation following a multi-level
> meta-regression
> >> with the standard error as moderator:
> >>
> >> sse.ml <- update(m.ml, mods = sqrt(vi))
> >> library("clubSandwich")
> >> conf_int(sse.ml, vcov = "CR2")
> >>
> >> Did I get this right?
> >>
> >> Best wishes,
> >> Guido
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]