[R-meta] Does trim and fill method correct for data falsification or lower quality of small studies?

[Viechtbauer, Wolfgang (NP)]

Thanks for providing the source of this quote. 

Having read the entire text now, my interpretation of the statement ("Equally importantly, the trim and fill method does not take into account reasons for funnel plot asymmetry other than publication bias") is that this is not meant to exclude other forms of biases that could behave like publication bias (such as the use of questionable research practices), but in essence the method assumes that funnel plot asymmetry is automatically due to publication bias (or other forms of biases that behave like it), so if the asymmetry is due to something else (like an unknown moderator that is correlated with study size), it will falsely attribute that asymmetry to publication bias and falsely correct for it.

Best,
Wolfgang

>-----Original Message-----
>From: towhidi [mailto:towhidi using ut.ac.ir]
>Sent: Wednesday, 04 May, 2022 0:12
>To: Viechtbauer, Wolfgang (NP)
>Cc: r sig meta-analysis list
>Subject: Re: [R-meta] Does trim and fill method correct for data falsification or
>lower quality of small studies?
>
>On 2022-05-03 11:45, Viechtbauer, Wolfgang (NP) wrote:
>
>> Dear Ali,
>>
>> Please see my responses below.
>>
>> Best,
>> Wolfgang
>>
>>> -----Original Message-----
>>> From: R-sig-meta-analysis
>>> [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>> Behalf Of towhidi
>>> Sent: Tuesday, 03 May, 2022 1:37
>>> To: r sig meta-analysis list
>>> Subject: [R-meta] Does trim and fill method correct for data
>>> falsification or
>>> lower quality of small studies?
>>>
>>> Dear all,
>>>
>>> The asymmetry in a funnel plot can be caused by factors other than
>>> publication bias, such as data falsification or poorer quality in
>>> smaller trials.
>>
>> ... or unaccounted for moderators (that are correlated with study size)
>> or more generally heterogeneity.
>>
>>> However, the Cochrane Handbook mentions that "the trim
>>> and fill method does not take into account reasons for funnel plot
>>> asymmetry other than publication bias".
>>
>> I searched through the handbook
>> (https://training.cochrane.org/handbook/current) and couldn't find this
>> quote. Where did you find this?
>>
>>> I do not understand why it cannot account for data falsification or
>>> poor
>>> quality of small trials, assuming that these characteristics are
>>> associated with study size. For data falsification, the true observed
>>> effect size (before the fraudulent change in the data) for these
>>> studies
>>> converges on the true underlying effect size. But the falsified data
>>> move these data points to the right side, and, using the trim and fill
>>> method, this bias is neutralized by imputing their counterparts on the
>>> other side.
>>
>> 'Neutralized' sounds a bit too optimistic. If a study is imputed
>> corresponding to the fraudulent study (which isn't guaranteed depending
>> on how the funnel looks in general), it is going to be placed at 'est -
>> delta', where 'est' is the pooled estimate at the end of the trim and
>> fill procedure and 'delta' is the distance between est and the
>> fraudulent study. If 'est' is larger than 0, then this would still
>> leads to some bias, but it should indeed be reduced.
>>
>>> Of course, the confidence intervals will be biased, because
>>> we are imputing data points that do not exist (which narrows the CI)
>>> and
>>> that the bias arose from data falsification or low quality has added
>>> to
>>> the estimated sampling variance (which widens the CI). Also, it
>>> changes
>>> the weights, especially in the random-effects model.
>>>
>>> But, isn't the point estimate a corrected estimate, assuming that data
>>> falsification has caused the asymmetry?
>>
>> I would say yes. A simulation study that has examined the properties of
>> various methods not only when there is publication bias but also under
>> the use of 'questionable research practices' is:
>>
>> Carter, E. C., Schönbrodt, F. D., Gervais, W. M. & Hilgard, J. (2019).
>> Correcting for bias in psychology: A comparison of meta-analytic
>> methods. Advances in Methods and Practices in Psychological Science,
>> 2(2), 115-144. https://doi.org/10.1177/2515245919847196
>>
>>> The same argument may apply to the bias that arises from low-quality
>>> studies. However, if this is correct, I think that acknowledging this
>>> and interpreting the CIs with even more caution is more logical than
>>> assuming that the asymmetry is caused solely by publication bias and
>>> that misconduct and low quality of small studies have nothing to do
>>> with
>>> it.
>>>
>>> Is this correct? Or I am missing something?
>>>
>>> Thank you.
>>>
>>> --
>>> Ali Zia-Tohidi MSc
>>> Clinical Psychology
>>> University of Tehran
>
>Dear Wolfgang,
>Thank you for your response, and thank you for the article you
>mentioned.
>
>The quoted text, "the trim and fill method does not take into account
>reasons for funnel plot asymmetry other than publication bias", was from
>the previous version of the Cochrane Handbook
>(https://handbook-5-1.cochrane.org/chapter_10/10_4_4_2_trim_and_fill.htm).
>The trim and fill subsection is removed from the current version.
>
>Best,
>Ali
>
>--
>Ali Zia-Tohidi MSc
>Clinical Psychology
>University of Tehran