[R-meta] methods for assessing publication bias while accounting for dependency

Mon Feb 28 21:27:53 CET 2022

Dear Brendan,

unsurprisingly Wolfgang was faster than me so I'll just add one more 
reference (with further references) if your curious about the problems 
of some methods (e.g. trim and fill) even in a basic two-level 
meta-analysis:
Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). 
Correcting for Bias in Psychology: A Comparison of Meta-Analytic 
Methods. Advances in Methods and Practices in Psychological Science, 
115–144. https://doi.org/10.1177/2515245919847196

One other possibility to address publication bias when dealing with 
dependent effect sizes is to conduct a moderator analysis comparing 
journal articles with other sources (e.g. conference proceedings, 
dissertations). If one is willing to assume that the latter are more 
similar to unpublished literature than journal articles then the results 
of this moderator analysis approximate the mangnitude of publication 
bias. Obviously, it is only some kind of sensitivity analysis and not 
the perfect estimate of publication bias.

Best,
Lukasz
-- 
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Seminarstraße 20
49074 Osnabrück (Germany)

Am 28.02.2022 um 19:45 schrieb r-sig-meta-analysis-request using r-project.org:
> Send R-sig-meta-analysis mailing list submissions to
> 	r-sig-meta-analysis using r-project.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> 	r-sig-meta-analysis-request using r-project.org
> 
> You can reach the person managing the list at
> 	r-sig-meta-analysis-owner using r-project.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
> 
> 
> Today's Topics:
> 
>     1. Re:  methods for assessing publication bias while accounting
>        for dependency (Viechtbauer, Wolfgang (SP))
>     2. Re: Heterogeneity and moderated mediation (Michael Dewey)
>     3. Re:  Meta-analysis of prevalence data: back-transformation
>        and polytomous data (Viechtbauer, Wolfgang (SP))
>     4. Re: Importing Correlations from PDF to table format (Kiet Huynh)
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Mon, 28 Feb 2022 13:31:46 +0000
> From: "Viechtbauer, Wolfgang (SP)"
> 	<wolfgang.viechtbauer using maastrichtuniversity.nl>
> To: Brendan Hutchinson <Brendan.Hutchinson using anu.edu.au>,
> 	"r-sig-meta-analysis using r-project.org"
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta]  methods for assessing publication bias while
> 	accounting for dependency
> Message-ID: <2377cc39202643a0ac5d87a34fce3cda using UM-MAIL3214.unimaas.nl>
> Content-Type: text/plain; charset="iso-8859-1"
> 
> Dear Brendan,
> 
> Using the 'regression method' approach could also be regarded as a form of sensitivity analysis, when focusing on the model intercept as an estimate of the 'adjusted' effect (as in the PET/PEESE methods). In fact, if I recall the findings from various simulation studies, this seems to work better than the trim and fill method.
> 
> One can also aggregate the estimates to the study level (or to whatever level needed so that the resulting aggregated values can be assumed to be independent) and then run methods that assume independence on these aggregated data (including trim and fill).
> 
> Another recent method by James Pustejovsky: https://www.jepusto.com/talk/stanford-qsu-2022-selective-reporting/
> 
> Some other relevant readings:
> 
> Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P. & Van den Noortgate, W. (2021). Detecting selection bias in meta-analyses with multiple outcomes: A simulation study. The Journal of Experimental Education, 89(1), 125-144. https://doi.org/10.1080/00220973.2019.1582470
> 
> Rodgers, M. A. & Pustejovsky, J. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26(2), 141-160. https://doi.org/10.1037/met0000300
> 
> P.S.: Please use meaningful post titles to make the mailing list archives more useful.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>> Behalf Of Brendan Hutchinson
>> Sent: Friday, 25 February, 2022 14:15
>> To: r-sig-meta-analysis using r-project.org
>> Subject: [R-meta] (no subject)
>>
>> Dear mailing list,
>>
>> I have a couple of minor questions regarding methods for assessing publication
>> bias while accounting for dependency.
>>
>> To my understanding, there is no means of running a publication bias analysis,
>> such as trim and fill, with a multilevel meta-analytic model in R (or a model in
>> which dependency issues need be accounted for). I am aware that one can use a
>> regression method, such as regressing the standard error onto the summary
>> estimate, within a multi-level model (this is fairly straightforward using
>> rma.mv(), for example). However, what about methods for assessing the robustness
>> of findings, if publication bias is a concern (such as trim and fill), while also
>> accounting for dependency?
>>
>> The best I have found is a recent package "PublicationBias" by Mathur and
>> VanderWeele (10.1111/rssc.12440).
>>
>> I am wondering if anyone has any recommendations for particular methods, R
>> packages, or readings?
>>
>> Thanks so much!
>>
>> Brendan
> 
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 28 Feb 2022 14:05:08 +0000
> From: Michael Dewey <lists using dewey.myzen.co.uk>
> To: Amy Zadow <Amy.Zadow using unisa.edu.au>, R meta
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Heterogeneity and moderated mediation
> Message-ID: <e560f46d-9887-d498-8c01-fa63b87fae24 using dewey.myzen.co.uk>
> Content-Type: text/plain; charset="windows-1252"; Format="flowed"
> 
> It is hard to comment in detail as we do not have any information about
> the meta-analysis you ran. Are there two separate analyses, one for
> groups and one for individuals, two separate data-sets, one for groups
> and one for individuals, or one analysis using a multi-level
> meta-analysis? Presumably that is all replicated four times for each PSC
> (whatever that is) but that could equally be another level in the
> multi-level mode.
> 
> Would the nature of the research environment and study design have
> caused you to believe that heterogeneity was expected or unlikely?
> 
> Michael
> 
> On 28/02/2022 06:50, Amy Zadow wrote:
>> Hello,
>>
>> I am seeking advice about my current results – any
>> comments/criticism/advice around the heterogeneity statistics would be
>> very helpful
>>
>> Also I would be keen to conduct a moderated mediation but not sure where
>> to start. Any advice/ recommended resources or code would be much
>> appreciated.
>>
>> Many thanks, Amy
>>
>>
>>   > _______________________________________________
>>   > R-sig-meta-analysis mailing list
>>   > R-sig-meta-analysis using r-project.org
>> <mailto:R-sig-meta-analysis using r-project.org>
>>   > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>>   >
>>
>> [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org <mailto:R-sig-meta-analysis using r-project.org>
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>> <https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis>
>>
>>
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>> 	Virus-free. www.avg.com
>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>
>>
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>