[R-meta] Assessing publication bias from multilevel modelling

[Célia Sofia Moreira]

I appreciate the previous explanations. They are very clear and useful.

Now I have another doubt: since funnel plots do not take into account
dependencies among effect sizes from the same paper, I would like to know
if they are a trusty / recommended method for assessing publication bias
when different papers contribute with a different number of effects in the
meta-analysis. Ultimately, I would like to know if it makes sense to
include funnel plots analysis (and asymmetric tests) in my paper. What is
your opinion?

2018-02-14 20:55 GMT+00:00 Viechtbauer Wolfgang (SP) <
wolfgang.viechtbauer at maastrichtuniversity.nl>:

> This aside, I just want to mention something with respect to:
>
> random = ~ 1 |Study, struct = "UN"
>
> 1) The 'struct' argument has no effect when specifying random effects
> terms of the form '~ 1 | var'. Only when you specify terms of the form '~
> var1 | var2' does 'struct' matter.
>
> 2) For multilevel data, 'random = ~ 1 | Study' is not sufficient. You
> should use:
>
> base$Id <- 1:nrow(base)
> mm <- rma.mv(y ~ 1, V = Vlist, random = ~ 1 | Study/Id, data = base)
>
> See also:
>
> http://www.metafor-project.org/doku.php/analyses:
> konstantopoulos2011#a_common_mistake_in_the_three-level_model
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
> >project.org] On Behalf Of James Pustejovsky
> >Sent: Wednesday, 14 February, 2018 19:37
> >To: Célia Sofia Moreira
> >Cc: r-sig-meta-analysis at r-project.org
> >Subject: Re: [R-meta] Assessing publication bias from multilevel
> >modelling
> >
> >A funnel plot is simply a scatter-plot of effect size estimates versus
> >their standard errors (or some other measure of precision). Modeling
> >assumptions about the dependencies among the effect size estimates don't
> >affect the plot itself, and so it makes sense that you would get the same
> >result from both of those functions.
> >
> >On Wed, Feb 14, 2018 at 11:56 AM, Célia Sofia Moreira <
> >celiasofiamoreira at gmail.com> wrote:
> >
> >> Dear Michael and James,
> >>
> >> Thank you very much for your nice recommendations. Meanwhile, I don't
> >know
> >> if it is just by chance, but the funnel plots of
> >>
> >> mm<-rma.mv(y ~ 1, V=Vlist, random = ~ 1 |Study, struct = "UN", data =
> >> base); #Vlist <- impute_covariance_matrix(vi = base$v, cluster =
> >> base$Study, r = .5)
> >> m<-rma(y ~ 1, v,  data = base);
> >>
> >> are precisely the same (although the models' summaries are different)!
> >I
> >> was not expecting this result.
> >> Does this make sense to you? Maybe "funnel(mm)" is not taking into
> >account
> >> the random part of the model, I don't know... Can you please comment
> >about
> >> this fact?
> >>
> >> 2018-02-14 17:36 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
> >>
> >>> I think regtest() is appropriate only for effect sizes that are
> >>> statistically independent. It would not be appropriate for Celiia's
> >case,
> >>> where there is dependence among multiple effects from a given study.
> >>>
> >>> Egger's regression test can be conducted simply by using the standard
> >>> errors of the effect size estimates (i.e., the "sei") as a predictor
> >in a
> >>> meta-regression. Using cluster-robust variance estimation with this
> >>> meta-regression will take care of the dependence issue. A significant
> >>> coefficient on the standard error predictor would indicate funnel plot
> >>> asymmetry. (Although as always, a lack of statistically significance
> >does
> >>> not *prove* that the funnel plot is symmetric.) When I've used this
> >>> technique, I fit the meta-regression without any random effects in the
> >>> model so that larger studies are given relatively more weight for
> >>> estimating the meta-regression coefficients.
> >>>
> >>> The Henmi-Copas model is limited to univariate models. I don't know of
> >a
> >>> way to generalize it for multi-variate meta-analysis.
> >>>
> >>> James
> >>>
> >>> On Wed, Feb 14, 2018 at 11:21 AM, Michael Dewey
> ><lists at dewey.myzen.co.uk>
> >>> wrote:
> >>>
> >>>> Dear Célia
> >>>>
> >>>> As far as I can see regtest can accept the yi, vi or sei parameters
> >so
> >>>> you should be OK there. I do not think the HC method is going to be
> >easy to
> >>>> do if it is even possible.
> >>>>
> >>>> Michael
> >>>>
> >>>>
> >>>> On 14/02/2018 16:14, Célia Sofia Moreira wrote:
> >>>>
> >>>>> Hi!
> >>>>>
> >>>>> I am studying a pretest-posttest controlled design and I'm using
> >metafor
> >>>>> and clubSandwich packages. In some papers, I collected more than one
> >>>>> effect
> >>>>> size (from the same study sample). For this reason, I did multilevel
> >>>>> modelling using rma.mv function, with
> >>>>> random = ~ 1 |Study, struct = "UN".
> >>>>>
> >>>>> I would like to perform a publication bias analysis, using funnel
> >plots,
> >>>>> tests for funnel plots asymmetry, and Hemni and copas (HC) model.
> >>>>> However,
> >>>>> since I am using rma.mv function, I can only depict (multilevel)
> >funnel
> >>>>> plots; unfortunately, tests for funnel plots asymmetry and HC model
> >are
> >>>>> not
> >>>>> available for rma.mv....
> >>>>>
> >>>>> Do you know any alternative way to test the asymmetry of my
> >"multilevel"
> >>>>> funnel plots, as well as to compare with the HC modelling?
> >>>>>
> >>>>> If not, can anybody please suggest other methods to assess
> >publication
> >>>>> bias
> >>>>> in this multilevel case?
> >>>>>
> >>>>> Thank you very much,
> >>>>> celia
>

	[[alternative HTML version deleted]]