[R-meta] methods for assessing publication bias while accounting for dependency

Dr. Gerta Rücker ruecker @end|ng |rom |mb|@un|-|re|burg@de
Mon Feb 28 22:44:44 CET 2022 

Hi all,

To add the Copas selection model to the models already suggested:

This model combines the usual random effects model with a selection 
model that models how the probability of publication depends on both a 
study's effect size and its standard error. In a sensitivity analysis it 
is investigated how effect estimates are expected to change with 
increasing level of selection. A goodness-of-fit test provides a 
plausible selection level, along with a corrected effect size, given the 
data.

The Copas selection model is implemented in R package metasens 
https://cran.r-project.org/web/packages/metasens/

For the implementation, see Carpenter et al. 
https://www.jclinepi.com/article/S0895-4356(08)00348-X/fulltext .

For a simulation study, see 
https://onlinelibrary.wiley.com/doi/10.1002/bimj.201000151 .

Best,

Gerta


Am 28.02.2022 um 21:44 schrieb James Pustejovsky:
> In addition to Wolfgang's and Lukasz's suggestions, I would add that I find
> the Mathur and Vanderweele approach pretty compelling. It is not exactly a
> "bias adjustment" technique (as Trim and Fill or PET/PEESE purport to be)
> but rather a sensitivity analysis, which examines hypothetical questions
> such as:
> * Supposing that statistically significant results are at most X times more
> likely to be published than non-significant results, what is the maximum
> degree of bias that would be expected in the overall average effect size
> estimate?
> * How strong would the selective publication process need to be to reduce
> the overall average effect size estimate to no more than Y?
> An interesting implication of their results is that there are scenarios
> where an overall average effect size cannot possibly be reduced to null,
> even with very extreme forms of selective publication.
>
> James
>
> On Mon, Feb 28, 2022 at 2:28 PM Lukasz Stasielowicz <
> lukasz.stasielowicz using uni-osnabrueck.de> wrote:
>
>> Dear Brendan,
>>
>> unsurprisingly Wolfgang was faster than me so I'll just add one more
>> reference (with further references) if your curious about the problems
>> of some methods (e.g. trim and fill) even in a basic two-level
>> meta-analysis:
>> Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019).
>> Correcting for Bias in Psychology: A Comparison of Meta-Analytic
>> Methods. Advances in Methods and Practices in Psychological Science,
>> 115–144. https://doi.org/10.1177/2515245919847196
>>
>>
>> One other possibility to address publication bias when dealing with
>> dependent effect sizes is to conduct a moderator analysis comparing
>> journal articles with other sources (e.g. conference proceedings,
>> dissertations). If one is willing to assume that the latter are more
>> similar to unpublished literature than journal articles then the results
>> of this moderator analysis approximate the mangnitude of publication
>> bias. Obviously, it is only some kind of sensitivity analysis and not
>> the perfect estimate of publication bias.
>>
>>
>> Best,
>> Lukasz
>> --
>> Lukasz Stasielowicz
>> Osnabrück University
>> Institute for Psychology
>> Research methods, psychological assessment, and evaluation
>> Seminarstraße 20
>> 49074 Osnabrück (Germany)
>>
>> Am 28.02.2022 um 19:45 schrieb r-sig-meta-analysis-request using r-project.org:
>>> Send R-sig-meta-analysis mailing list submissions to
>>>        r-sig-meta-analysis using r-project.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>        https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>> or, via email, send a message with subject or body 'help' to
>>>        r-sig-meta-analysis-request using r-project.org
>>>
>>> You can reach the person managing the list at
>>>        r-sig-meta-analysis-owner using r-project.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of R-sig-meta-analysis digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>      1. Re:  methods for assessing publication bias while accounting
>>>         for dependency (Viechtbauer, Wolfgang (SP))
>>>      2. Re: Heterogeneity and moderated mediation (Michael Dewey)
>>>      3. Re:  Meta-analysis of prevalence data: back-transformation
>>>         and polytomous data (Viechtbauer, Wolfgang (SP))
>>>      4. Re: Importing Correlations from PDF to table format (Kiet Huynh)
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Mon, 28 Feb 2022 13:31:46 +0000
>>> From: "Viechtbauer, Wolfgang (SP)"
>>>        <wolfgang.viechtbauer using maastrichtuniversity.nl>
>>> To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>,
>>>        "r-sig-meta-analysis using r-project.org"
>>>        <r-sig-meta-analysis using r-project.org>
>>> Subject: Re: [R-meta]  methods for assessing publication bias while
>>>        accounting for dependency
>>> Message-ID: <2377cc39202643a0ac5d87a34fce3cda using UM-MAIL3214.unimaas.nl>
>>> Content-Type: text/plain; charset="iso-8859-1"
>>>
>>> Dear Brendan,
>>>
>>> Using the 'regression method' approach could also be regarded as a form
>> of sensitivity analysis, when focusing on the model intercept as an
>> estimate of the 'adjusted' effect (as in the PET/PEESE methods). In fact,
>> if I recall the findings from various simulation studies, this seems to
>> work better than the trim and fill method.
>>> One can also aggregate the estimates to the study level (or to whatever
>> level needed so that the resulting aggregated values can be assumed to be
>> independent) and then run methods that assume independence on these
>> aggregated data (including trim and fill).
>>> Another recent method by James Pustejovsky:
>> https://www.jepusto.com/talk/stanford-qsu-2022-selective-reporting/
>>> Some other relevant readings:
>>>
>>> Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
>> Onghena, P. & Van den Noortgate, W. (2021). Detecting selection bias in
>> meta-analyses with multiple outcomes: A simulation study. The Journal of
>> Experimental Education, 89(1), 125-144.
>> https://doi.org/10.1080/00220973.2019.1582470
>>> Rodgers, M. A. & Pustejovsky, J. E. (2021). Evaluating meta-analytic
>> methods to detect selective reporting in the presence of dependent effect
>> sizes. Psychological Methods, 26(2), 141-160.
>> https://doi.org/10.1037/met0000300
>>> P.S.: Please use meaningful post titles to make the mailing list
>> archives more useful.
>>> Best,
>>> Wolfgang
>>>
>>>> -----Original Message-----
>>>> From: R-sig-meta-analysis [mailto:
>> r-sig-meta-analysis-bounces using r-project.org] On
>>>> Behalf Of Brendan Hutchinson
>>>> Sent: Friday, 25 February, 2022 14:15
>>>> To: r-sig-meta-analysis using r-project.org
>>>> Subject: [R-meta] (no subject)
>>>>
>>>> Dear mailing list,
>>>>
>>>> I have a couple of minor questions regarding methods for assessing
>> publication
>>>> bias while accounting for dependency.
>>>>
>>>> To my understanding, there is no means of running a publication bias
>> analysis,
>>>> such as trim and fill, with a multilevel meta-analytic model in R (or a
>> model in
>>>> which dependency issues need be accounted for). I am aware that one can
>> use a
>>>> regression method, such as regressing the standard error onto the
>> summary
>>>> estimate, within a multi-level model (this is fairly straightforward
>> using
>>>> rma.mv(), for example). However, what about methods for assessing the
>> robustness
>>>> of findings, if publication bias is a concern (such as trim and fill),
>> while also
>>>> accounting for dependency?
>>>>
>>>> The best I have found is a recent package "PublicationBias" by Mathur
>> and
>>>> VanderWeele (10.1111/rssc.12440).
>>>>
>>>> I am wondering if anyone has any recommendations for particular
>> methods, R
>>>> packages, or readings?
>>>>
>>>> Thanks so much!
>>>>
>>>> Brendan
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Mon, 28 Feb 2022 14:05:08 +0000
>>> From: Michael Dewey <lists using dewey.myzen.co.uk>
>>> To: Amy Zadow <Amy.Zadow using unisa.edu.au>, R meta
>>>        <r-sig-meta-analysis using r-project.org>
>>> Subject: Re: [R-meta] Heterogeneity and moderated mediation
>>> Message-ID: <e560f46d-9887-d498-8c01-fa63b87fae24 using dewey.myzen.co.uk>
>>> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
>>>
>>> It is hard to comment in detail as we do not have any information about
>>> the meta-analysis you ran. Are there two separate analyses, one for
>>> groups and one for individuals, two separate data-sets, one for groups
>>> and one for individuals, or one analysis using a multi-level
>>> meta-analysis? Presumably that is all replicated four times for each PSC
>>> (whatever that is) but that could equally be another level in the
>>> multi-level mode.
>>>
>>> Would the nature of the research environment and study design have
>>> caused you to believe that heterogeneity was expected or unlikely?
>>>
>>> Michael
>>>
>>> On 28/02/2022 06:50, Amy Zadow wrote:
>>>> Hello,
>>>>
>>>> I am seeking advice about my current results – any
>>>> comments/criticism/advice around the heterogeneity statistics would be
>>>> very helpful
>>>>
>>>> Also I would be keen to conduct a moderated mediation but not sure where
>>>> to start. Any advice/ recommended resources or code would be much
>>>> appreciated.
>>>>
>>>> Many thanks, Amy
>>>>
>>>>
>>>>    > _______________________________________________
>>>>    > R-sig-meta-analysis mailing list
>>>>    > R-sig-meta-analysis using r-project.org
>>>> <mailto:R-sig-meta-analysis using r-project.org>
>>>>    > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>>>>    >
>>>>
>>>> [[alternative HTML version deleted]]
>>>>
>>>> _______________________________________________
>>>> R-sig-meta-analysis mailing list
>>>> R-sig-meta-analysis using r-project.org <mailto:
>> R-sig-meta-analysis using r-project.org>
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>>>>
>>>>
>>>> <
>> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
>>>>       Virus-free. www.avg.com
>>>> <
>> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
>>>>
>>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>
>>>> _______________________________________________
>>>> R-sig-meta-analysis mailing list
>>>> R-sig-meta-analysis using r-project.org
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
> 	[[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-- 

Dr. rer. nat. Gerta Rücker, Dipl.-Math.

Guest Scientist
Institute of Medical Biometry and Statistics,
Faculty of Medicine and Medical Center - University of Freiburg

Zinkmattenstr. 6a, D-79108 Freiburg, Germany

Mail:     ruecker using imbi.uni-freiburg.de
Homepage: https://www.uniklinik-freiburg.de/imbi-en/employees.html?imbiuser=ruecker

More information about the R-sig-meta-analysis mailing list