[R-meta] Assessing publication bias from multilevel modelling

[Gerta Ruecker]

Dear Célia,

you should not show more than one effect size from the same study in the 
same funnel plot. Better report separate funnel plots for each outcome 
you have measured, each study at most once per plot.

Best,

Gerta Rücker


Am 15.02.2018 um 12:28 schrieb Célia Sofia Moreira:
> I appreciate the previous explanations. They are very clear and useful.
>
> Now I have another doubt: since funnel plots do not take into account
> dependencies among effect sizes from the same paper, I would like to know
> if they are a trusty / recommended method for assessing publication bias
> when different papers contribute with a different number of effects in the
> meta-analysis. Ultimately, I would like to know if it makes sense to
> include funnel plots analysis (and asymmetric tests) in my paper. What is
> your opinion?
>
> 2018-02-14 20:55 GMT+00:00 Viechtbauer Wolfgang (SP) <
> wolfgang.viechtbauer at maastrichtuniversity.nl>:
>
>> This aside, I just want to mention something with respect to:
>>
>> random = ~ 1 |Study, struct = "UN"
>>
>> 1) The 'struct' argument has no effect when specifying random effects
>> terms of the form '~ 1 | var'. Only when you specify terms of the form '~
>> var1 | var2' does 'struct' matter.
>>
>> 2) For multilevel data, 'random = ~ 1 | Study' is not sufficient. You
>> should use:
>>
>> base$Id <- 1:nrow(base)
>> mm <- rma.mv(y ~ 1, V = Vlist, random = ~ 1 | Study/Id, data = base)
>>
>> See also:
>>
>> http://www.metafor-project.org/doku.php/analyses:
>> konstantopoulos2011#a_common_mistake_in_the_three-level_model
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>>> project.org] On Behalf Of James Pustejovsky
>>> Sent: Wednesday, 14 February, 2018 19:37
>>> To: Célia Sofia Moreira
>>> Cc: r-sig-meta-analysis at r-project.org
>>> Subject: Re: [R-meta] Assessing publication bias from multilevel
>>> modelling
>>>
>>> A funnel plot is simply a scatter-plot of effect size estimates versus
>>> their standard errors (or some other measure of precision). Modeling
>>> assumptions about the dependencies among the effect size estimates don't
>>> affect the plot itself, and so it makes sense that you would get the same
>>> result from both of those functions.
>>>
>>> On Wed, Feb 14, 2018 at 11:56 AM, Célia Sofia Moreira <
>>> celiasofiamoreira at gmail.com> wrote:
>>>
>>>> Dear Michael and James,
>>>>
>>>> Thank you very much for your nice recommendations. Meanwhile, I don't
>>> know
>>>> if it is just by chance, but the funnel plots of
>>>>
>>>> mm<-rma.mv(y ~ 1, V=Vlist, random = ~ 1 |Study, struct = "UN", data =
>>>> base); #Vlist <- impute_covariance_matrix(vi = base$v, cluster =
>>>> base$Study, r = .5)
>>>> m<-rma(y ~ 1, v,  data = base);
>>>>
>>>> are precisely the same (although the models' summaries are different)!
>>> I
>>>> was not expecting this result.
>>>> Does this make sense to you? Maybe "funnel(mm)" is not taking into
>>> account
>>>> the random part of the model, I don't know... Can you please comment
>>> about
>>>> this fact?
>>>>
>>>> 2018-02-14 17:36 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
>>>>
>>>>> I think regtest() is appropriate only for effect sizes that are
>>>>> statistically independent. It would not be appropriate for Celiia's
>>> case,
>>>>> where there is dependence among multiple effects from a given study.
>>>>>
>>>>> Egger's regression test can be conducted simply by using the standard
>>>>> errors of the effect size estimates (i.e., the "sei") as a predictor
>>> in a
>>>>> meta-regression. Using cluster-robust variance estimation with this
>>>>> meta-regression will take care of the dependence issue. A significant
>>>>> coefficient on the standard error predictor would indicate funnel plot
>>>>> asymmetry. (Although as always, a lack of statistically significance
>>> does
>>>>> not *prove* that the funnel plot is symmetric.) When I've used this
>>>>> technique, I fit the meta-regression without any random effects in the
>>>>> model so that larger studies are given relatively more weight for
>>>>> estimating the meta-regression coefficients.
>>>>>
>>>>> The Henmi-Copas model is limited to univariate models. I don't know of
>>> a
>>>>> way to generalize it for multi-variate meta-analysis.
>>>>>
>>>>> James
>>>>>
>>>>> On Wed, Feb 14, 2018 at 11:21 AM, Michael Dewey
>>> <lists at dewey.myzen.co.uk>
>>>>> wrote:
>>>>>
>>>>>> Dear Célia
>>>>>>
>>>>>> As far as I can see regtest can accept the yi, vi or sei parameters
>>> so
>>>>>> you should be OK there. I do not think the HC method is going to be
>>> easy to
>>>>>> do if it is even possible.
>>>>>>
>>>>>> Michael
>>>>>>
>>>>>>
>>>>>> On 14/02/2018 16:14, Célia Sofia Moreira wrote:
>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I am studying a pretest-posttest controlled design and I'm using
>>> metafor
>>>>>>> and clubSandwich packages. In some papers, I collected more than one
>>>>>>> effect
>>>>>>> size (from the same study sample). For this reason, I did multilevel
>>>>>>> modelling using rma.mv function, with
>>>>>>> random = ~ 1 |Study, struct = "UN".
>>>>>>>
>>>>>>> I would like to perform a publication bias analysis, using funnel
>>> plots,
>>>>>>> tests for funnel plots asymmetry, and Hemni and copas (HC) model.
>>>>>>> However,
>>>>>>> since I am using rma.mv function, I can only depict (multilevel)
>>> funnel
>>>>>>> plots; unfortunately, tests for funnel plots asymmetry and HC model
>>> are
>>>>>>> not
>>>>>>> available for rma.mv....
>>>>>>>
>>>>>>> Do you know any alternative way to test the asymmetry of my
>>> "multilevel"
>>>>>>> funnel plots, as well as to compare with the HC modelling?
>>>>>>>
>>>>>>> If not, can anybody please suggest other methods to assess
>>> publication
>>>>>>> bias
>>>>>>> in this multilevel case?
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> celia
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-- 

Dr. rer. nat. Gerta Rücker, Dipl.-Math.

Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Stefan-Meier-Str. 26, D-79104 Freiburg, Germany

Phone:    +49/761/203-6673
Fax:      +49/761/203-6680
Mail:     ruecker at imbi.uni-freiburg.de
Homepage: https://portal.uni-freiburg.de/imbi/persons/ruecker?set_language=en