[R-meta] Assessing publication bias from multilevel modelling

[Viechtbauer Wolfgang (SP)]

This aside, I just want to mention something with respect to:

random = ~ 1 |Study, struct = "UN"

1) The 'struct' argument has no effect when specifying random effects terms of the form '~ 1 | var'. Only when you specify terms of the form '~ var1 | var2' does 'struct' matter.

2) For multilevel data, 'random = ~ 1 | Study' is not sufficient. You should use:

base$Id <- 1:nrow(base)
mm <- rma.mv(y ~ 1, V = Vlist, random = ~ 1 | Study/Id, data = base)

See also: 

http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011#a_common_mistake_in_the_three-level_model

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-
>project.org] On Behalf Of James Pustejovsky
>Sent: Wednesday, 14 February, 2018 19:37
>To: Célia Sofia Moreira
>Cc: r-sig-meta-analysis at r-project.org
>Subject: Re: [R-meta] Assessing publication bias from multilevel
>modelling
>
>A funnel plot is simply a scatter-plot of effect size estimates versus
>their standard errors (or some other measure of precision). Modeling
>assumptions about the dependencies among the effect size estimates don't
>affect the plot itself, and so it makes sense that you would get the same
>result from both of those functions.
>
>On Wed, Feb 14, 2018 at 11:56 AM, Célia Sofia Moreira <
>celiasofiamoreira at gmail.com> wrote:
>
>> Dear Michael and James,
>>
>> Thank you very much for your nice recommendations. Meanwhile, I don't
>know
>> if it is just by chance, but the funnel plots of
>>
>> mm<-rma.mv(y ~ 1, V=Vlist, random = ~ 1 |Study, struct = "UN", data =
>> base); #Vlist <- impute_covariance_matrix(vi = base$v, cluster =
>> base$Study, r = .5)
>> m<-rma(y ~ 1, v,  data = base);
>>
>> are precisely the same (although the models' summaries are different)!
>I
>> was not expecting this result.
>> Does this make sense to you? Maybe "funnel(mm)" is not taking into
>account
>> the random part of the model, I don't know... Can you please comment
>about
>> this fact?
>>
>> 2018-02-14 17:36 GMT+00:00 James Pustejovsky <jepusto at gmail.com>:
>>
>>> I think regtest() is appropriate only for effect sizes that are
>>> statistically independent. It would not be appropriate for Celiia's
>case,
>>> where there is dependence among multiple effects from a given study.
>>>
>>> Egger's regression test can be conducted simply by using the standard
>>> errors of the effect size estimates (i.e., the "sei") as a predictor
>in a
>>> meta-regression. Using cluster-robust variance estimation with this
>>> meta-regression will take care of the dependence issue. A significant
>>> coefficient on the standard error predictor would indicate funnel plot
>>> asymmetry. (Although as always, a lack of statistically significance
>does
>>> not *prove* that the funnel plot is symmetric.) When I've used this
>>> technique, I fit the meta-regression without any random effects in the
>>> model so that larger studies are given relatively more weight for
>>> estimating the meta-regression coefficients.
>>>
>>> The Henmi-Copas model is limited to univariate models. I don't know of
>a
>>> way to generalize it for multi-variate meta-analysis.
>>>
>>> James
>>>
>>> On Wed, Feb 14, 2018 at 11:21 AM, Michael Dewey
><lists at dewey.myzen.co.uk>
>>> wrote:
>>>
>>>> Dear Célia
>>>>
>>>> As far as I can see regtest can accept the yi, vi or sei parameters
>so
>>>> you should be OK there. I do not think the HC method is going to be
>easy to
>>>> do if it is even possible.
>>>>
>>>> Michael
>>>>
>>>>
>>>> On 14/02/2018 16:14, Célia Sofia Moreira wrote:
>>>>
>>>>> Hi!
>>>>>
>>>>> I am studying a pretest-posttest controlled design and I'm using
>metafor
>>>>> and clubSandwich packages. In some papers, I collected more than one
>>>>> effect
>>>>> size (from the same study sample). For this reason, I did multilevel
>>>>> modelling using rma.mv function, with
>>>>> random = ~ 1 |Study, struct = "UN".
>>>>>
>>>>> I would like to perform a publication bias analysis, using funnel
>plots,
>>>>> tests for funnel plots asymmetry, and Hemni and copas (HC) model.
>>>>> However,
>>>>> since I am using rma.mv function, I can only depict (multilevel)
>funnel
>>>>> plots; unfortunately, tests for funnel plots asymmetry and HC model
>are
>>>>> not
>>>>> available for rma.mv....
>>>>>
>>>>> Do you know any alternative way to test the asymmetry of my
>"multilevel"
>>>>> funnel plots, as well as to compare with the HC modelling?
>>>>>
>>>>> If not, can anybody please suggest other methods to assess
>publication
>>>>> bias
>>>>> in this multilevel case?
>>>>>
>>>>> Thank you very much,
>>>>> celia