[R-meta] Does trim and fill method correct for data falsification or lower quality of small studies?

[towhidi]

On 2022-05-04 16:47, Michael Dewey wrote:

> One might also point out that the methods has been criticised for 
> detecting publicaiton bias
> 
> @article{schwarzer10,
> author = {Schwarzer, G and Carpenter, J R and R\"ucker, G},
> title = {Empirical evaluation suggests {Copas} selection model
> preferable to trim--and--fill method for publication
> bias in meta--analysis},
> journal = {Journal of Clinical Epidemiology},
> year = {2010},
> volume = {63},
> pages = {282--288}
> }
> 
> Michael
> 
> On 03/05/2022 23:11, towhidi wrote: On 2022-05-03 11:45, Viechtbauer, 
> Wolfgang (NP) wrote:
> 
> Dear Ali,
> 
> Please see my responses below.
> 
> Best,
> Wolfgang
> 
> -----Original Message-----
> From: R-sig-meta-analysis 
> [mailto:r-sig-meta-analysis-bounces using r-project.org] On
> Behalf Of towhidi
> Sent: Tuesday, 03 May, 2022 1:37
> To: r sig meta-analysis list
> Subject: [R-meta] Does trim and fill method correct for data 
> falsification or
> lower quality of small studies?
> 
> Dear all,
> 
> The asymmetry in a funnel plot can be caused by factors other than
> publication bias, such as data falsification or poorer quality in
> smaller trials.
> ... or unaccounted for moderators (that are correlated with study size) 
> or more generally heterogeneity.
> 
> However, the Cochrane Handbook mentions that "the trim
> and fill method does not take into account reasons for funnel plot
> asymmetry other than publication bias".
> I searched through the handbook 
> (https://training.cochrane.org/handbook/current) and couldn't find this 
> quote. Where did you find this?
> 
> I do not understand why it cannot account for data falsification or 
> poor
> quality of small trials, assuming that these characteristics are
> associated with study size. For data falsification, the true observed
> effect size (before the fraudulent change in the data) for these 
> studies
> converges on the true underlying effect size. But the falsified data
> move these data points to the right side, and, using the trim and fill
> method, this bias is neutralized by imputing their counterparts on the
> other side.
> 'Neutralized' sounds a bit too optimistic. If a study is imputed 
> corresponding to the fraudulent study (which isn't guaranteed depending 
> on how the funnel looks in general), it is going to be placed at 'est - 
> delta', where 'est' is the pooled estimate at the end of the trim and 
> fill procedure and 'delta' is the distance between est and the 
> fraudulent study. If 'est' is larger than 0, then this would still 
> leads to some bias, but it should indeed be reduced.
> 
> Of course, the confidence intervals will be biased, because
> we are imputing data points that do not exist (which narrows the CI) 
> and
> that the bias arose from data falsification or low quality has added to
> the estimated sampling variance (which widens the CI). Also, it changes
> the weights, especially in the random-effects model.
> 
> But, isn't the point estimate a corrected estimate, assuming that data
> falsification has caused the asymmetry?
> I would say yes. A simulation study that has examined the properties of 
> various methods not only when there is publication bias but also under 
> the use of 'questionable research practices' is:
> 
> Carter, E. C., Schönbrodt, F. D., Gervais, W. M. & Hilgard, J. (2019). 
> Correcting for bias in psychology: A comparison of meta-analytic 
> methods. Advances in Methods and Practices in Psychological Science, 
> 2(2), 115-144. https://doi.org/10.1177/2515245919847196
> 
> The same argument may apply to the bias that arises from low-quality
> studies. However, if this is correct, I think that acknowledging this
> and interpreting the CIs with even more caution is more logical than
> assuming that the asymmetry is caused solely by publication bias and
> that misconduct and low quality of small studies have nothing to do 
> with
> it.
> 
> Is this correct? Or I am missing something?
> 
> Thank you.
> 
> -- Ali Zia-Tohidi MSc
> Clinical Psychology
> University of Tehran

> Dear Wolfgang,
> Thank you for your response, and thank you for the article you 
> mentioned.

> The quoted text, "the trim and fill method does not take into account 
> reasons for funnel plot asymmetry other than publication bias", was 
> from the previous version of the Cochrane Handbook 
> (https://handbook-5-1.cochrane.org/chapter_10/10_4_4_2_trim_and_fill.htm). 
> The trim and fill subsection is removed from the current version.

> Best,
> Ali



Dear Wolfgang
Thank you for elaborating on the text.

And Dear Micheal,
Thank you for your note. I intend to use different methods in my 
sensitivity analyses.

Best,
Ali


-- 
Ali Zia-Tohidi MSc
Clinical Psychology
University of Tehran