[R-meta] methods for assessing publication bias while accounting for dependency

James Pustejovsky jepu@to @end|ng |rom gm@||@com
Mon Feb 28 21:44:14 CET 2022 

In addition to Wolfgang's and Lukasz's suggestions, I would add that I find
the Mathur and Vanderweele approach pretty compelling. It is not exactly a
"bias adjustment" technique (as Trim and Fill or PET/PEESE purport to be)
but rather a sensitivity analysis, which examines hypothetical questions
such as:
* Supposing that statistically significant results are at most X times more
likely to be published than non-significant results, what is the maximum
degree of bias that would be expected in the overall average effect size
estimate?
* How strong would the selective publication process need to be to reduce
the overall average effect size estimate to no more than Y?
An interesting implication of their results is that there are scenarios
where an overall average effect size cannot possibly be reduced to null,
even with very extreme forms of selective publication.

James

On Mon, Feb 28, 2022 at 2:28 PM Lukasz Stasielowicz <
lukasz.stasielowicz using uni-osnabrueck.de> wrote:

> Dear Brendan,
>
> unsurprisingly Wolfgang was faster than me so I'll just add one more
> reference (with further references) if your curious about the problems
> of some methods (e.g. trim and fill) even in a basic two-level
> meta-analysis:
> Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019).
> Correcting for Bias in Psychology: A Comparison of Meta-Analytic
> Methods. Advances in Methods and Practices in Psychological Science,
> 115–144. https://doi.org/10.1177/2515245919847196
>
>
> One other possibility to address publication bias when dealing with
> dependent effect sizes is to conduct a moderator analysis comparing
> journal articles with other sources (e.g. conference proceedings,
> dissertations). If one is willing to assume that the latter are more
> similar to unpublished literature than journal articles then the results
> of this moderator analysis approximate the mangnitude of publication
> bias. Obviously, it is only some kind of sensitivity analysis and not
> the perfect estimate of publication bias.
>
>
> Best,
> Lukasz
> --
> Lukasz Stasielowicz
> Osnabrück University
> Institute for Psychology
> Research methods, psychological assessment, and evaluation
> Seminarstraße 20
> 49074 Osnabrück (Germany)
>
> Am 28.02.2022 um 19:45 schrieb r-sig-meta-analysis-request using r-project.org:
> > Send R-sig-meta-analysis mailing list submissions to
> >       r-sig-meta-analysis using r-project.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >       https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> > or, via email, send a message with subject or body 'help' to
> >       r-sig-meta-analysis-request using r-project.org
> >
> > You can reach the person managing the list at
> >       r-sig-meta-analysis-owner using r-project.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of R-sig-meta-analysis digest..."
> >
> >
> > Today's Topics:
> >
> >     1. Re:  methods for assessing publication bias while accounting
> >        for dependency (Viechtbauer, Wolfgang (SP))
> >     2. Re: Heterogeneity and moderated mediation (Michael Dewey)
> >     3. Re:  Meta-analysis of prevalence data: back-transformation
> >        and polytomous data (Viechtbauer, Wolfgang (SP))
> >     4. Re: Importing Correlations from PDF to table format (Kiet Huynh)
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Mon, 28 Feb 2022 13:31:46 +0000
> > From: "Viechtbauer, Wolfgang (SP)"
> >       <wolfgang.viechtbauer using maastrichtuniversity.nl>
> > To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>,
> >       "r-sig-meta-analysis using r-project.org"
> >       <r-sig-meta-analysis using r-project.org>
> > Subject: Re: [R-meta]  methods for assessing publication bias while
> >       accounting for dependency
> > Message-ID: <2377cc39202643a0ac5d87a34fce3cda using UM-MAIL3214.unimaas.nl>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> > Dear Brendan,
> >
> > Using the 'regression method' approach could also be regarded as a form
> of sensitivity analysis, when focusing on the model intercept as an
> estimate of the 'adjusted' effect (as in the PET/PEESE methods). In fact,
> if I recall the findings from various simulation studies, this seems to
> work better than the trim and fill method.
> >
> > One can also aggregate the estimates to the study level (or to whatever
> level needed so that the resulting aggregated values can be assumed to be
> independent) and then run methods that assume independence on these
> aggregated data (including trim and fill).
> >
> > Another recent method by James Pustejovsky:
> https://www.jepusto.com/talk/stanford-qsu-2022-selective-reporting/
> >
> > Some other relevant readings:
> >
> > Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N.,
> Onghena, P. & Van den Noortgate, W. (2021). Detecting selection bias in
> meta-analyses with multiple outcomes: A simulation study. The Journal of
> Experimental Education, 89(1), 125-144.
> https://doi.org/10.1080/00220973.2019.1582470
> >
> > Rodgers, M. A. & Pustejovsky, J. E. (2021). Evaluating meta-analytic
> methods to detect selective reporting in the presence of dependent effect
> sizes. Psychological Methods, 26(2), 141-160.
> https://doi.org/10.1037/met0000300
> >
> > P.S.: Please use meaningful post titles to make the mailing list
> archives more useful.
> >
> > Best,
> > Wolfgang
> >
> >> -----Original Message-----
> >> From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >> Behalf Of Brendan Hutchinson
> >> Sent: Friday, 25 February, 2022 14:15
> >> To: r-sig-meta-analysis using r-project.org
> >> Subject: [R-meta] (no subject)
> >>
> >> Dear mailing list,
> >>
> >> I have a couple of minor questions regarding methods for assessing
> publication
> >> bias while accounting for dependency.
> >>
> >> To my understanding, there is no means of running a publication bias
> analysis,
> >> such as trim and fill, with a multilevel meta-analytic model in R (or a
> model in
> >> which dependency issues need be accounted for). I am aware that one can
> use a
> >> regression method, such as regressing the standard error onto the
> summary
> >> estimate, within a multi-level model (this is fairly straightforward
> using
> >> rma.mv(), for example). However, what about methods for assessing the
> robustness
> >> of findings, if publication bias is a concern (such as trim and fill),
> while also
> >> accounting for dependency?
> >>
> >> The best I have found is a recent package "PublicationBias" by Mathur
> and
> >> VanderWeele (10.1111/rssc.12440).
> >>
> >> I am wondering if anyone has any recommendations for particular
> methods, R
> >> packages, or readings?
> >>
> >> Thanks so much!
> >>
> >> Brendan
> >
> >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Mon, 28 Feb 2022 14:05:08 +0000
> > From: Michael Dewey <lists using dewey.myzen.co.uk>
> > To: Amy Zadow <Amy.Zadow using unisa.edu.au>, R meta
> >       <r-sig-meta-analysis using r-project.org>
> > Subject: Re: [R-meta] Heterogeneity and moderated mediation
> > Message-ID: <e560f46d-9887-d498-8c01-fa63b87fae24 using dewey.myzen.co.uk>
> > Content-Type: text/plain; charset="windows-1252"; Format="flowed"
> >
> > It is hard to comment in detail as we do not have any information about
> > the meta-analysis you ran. Are there two separate analyses, one for
> > groups and one for individuals, two separate data-sets, one for groups
> > and one for individuals, or one analysis using a multi-level
> > meta-analysis? Presumably that is all replicated four times for each PSC
> > (whatever that is) but that could equally be another level in the
> > multi-level mode.
> >
> > Would the nature of the research environment and study design have
> > caused you to believe that heterogeneity was expected or unlikely?
> >
> > Michael
> >
> > On 28/02/2022 06:50, Amy Zadow wrote:
> >> Hello,
> >>
> >> I am seeking advice about my current results – any
> >> comments/criticism/advice around the heterogeneity statistics would be
> >> very helpful
> >>
> >> Also I would be keen to conduct a moderated mediation but not sure where
> >> to start. Any advice/ recommended resources or code would be much
> >> appreciated.
> >>
> >> Many thanks, Amy
> >>
> >>
> >>   > _______________________________________________
> >>   > R-sig-meta-analysis mailing list
> >>   > R-sig-meta-analysis using r-project.org
> >> <mailto:R-sig-meta-analysis using r-project.org>
> >>   > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
> >>   >
> >>
> >> [[alternative HTML version deleted]]
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list
> >> R-sig-meta-analysis using r-project.org <mailto:
> R-sig-meta-analysis using r-project.org>
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
> >>
> >>
> >> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
> >
> >>      Virus-free. www.avg.com
> >> <
> http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient
> >
> >>
> >>
> >> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>
> >> _______________________________________________
> >> R-sig-meta-analysis mailing list
> >> R-sig-meta-analysis using r-project.org
> >> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >>
> >
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]

More information about the R-sig-meta-analysis mailing list