[R-meta] Publication bias/sensitivity analysis in multivariate meta-analysis

[Michael Dewey]

Just to add to Gerta's comprehensive reply.

One IPD analysis in which I was involved had a number of small studies 
which were broadly positive and one large one which was effectively 
null. The investigators were convinced that they were very unlikely to 
have missed any other studies and the most likely explanation for the 
small study effect was that the small studies were conducted by 
enthusiasts for the new therapy who often delivered it themselves 
whereas the large study involved many therapists scattered over the 
country who were more likely to represent how it would actually work if 
rolled out. I suspect similar things often happen for complex interventions.

Michael

On 15/06/2020 10:19, Gerta Ruecker wrote:
> Dear Norman, dear all,
> 
> To clarify the notions:
> 
> Small-study effects: All effects manifesting themselves as small studies 
> having different effects from large studies. The notion was coined by 
> Sterne et al. (Sterne, J. A. C., Gavaghan, D., and Egger, M. (2000). 
> Publication and related bias in meta-analysis: Power of statistical 
> tests and prevalence in the literature.
> Journal of Clinical Epidemiology, 53:1119–1129.) Small-study effects are 
> seen in a funnel plot as asymmetry.
> 
> Reasons for small-study effects may be: Heterogeneity, e.g., small 
> studies have selected patients (for example, worse health status); 
> publication bias (see below), mathematical artifacts for binary data 
> (Schwarzer, G., Antes, G., and Schumacher, M. (2002). Inflation of type 
> I error rate in two statistical tests for the detection of publication 
> bias in meta-analyses with binary outcomes. Statistics in Medicine, 
> 21:2465–2477), or coincidence.
> 
> Publication bias is one possible reason of small-study effects and means 
> that small studies with small, no, or undesired effects are not 
> published and therefore not found in the literature. The result is an 
> effect estimate that is biased towards large effects.
> 
> Sensitivity analysis is a possibility to investigate small-study 
> effects. There is an abundance of literature and methods how to do this. 
> Well-known models are selection models, e.g. Vevea, J. L. and Hedges, L. 
> V. (1995). A general linear model for estimating effect size in the 
> presence of publication bias. Psychometrika, 60:419–435 or Copas, J. and 
> Shi, J. Q. (2000). Meta-analysis, funnel plots and sensitivity analysis. 
> Biostatistics, 1:247–262.
> 
> I attach a talk with more details.
> 
> Best,
> 
> Gerta
> 
> 
> Am 15.06.2020 um 02:28 schrieb Norman DAURELLE:
>> Hi all, I read this thread, and the topic interests me, but I didn't 
>> quite understand your answer :when you say " Publication bias is a 
>> subset of small study effects where you know the
>> aetiology of the small study effects. If you do not then it is safer to
>> refer to small study effects. "
>> I don't really understand what you mean.I thought publication bias 
>> meant that the studies included in a sample of study didn't really 
>> account for the whole range of possible effect sizes (with their 
>> associated standard error).Is that not what publication bias refers to 
>> ? And if it is, how does it also correspond to the definition you gave 
>> ?Thank you !Norman.
>> ----- Mail d'origine -----
>> De: Michael Dewey <lists using dewey.myzen.co.uk>
>> À: Huang Wu <huang.wu using wmich.edu>, r-sig-meta-analysis using r-project.org
>> Envoyé: Sun, 14 Jun 2020 12:54:30 +0200 (CEST)
>> Objet: Re: [R-meta] Publication bias/sensitivity analysis in 
>> multivariate meta-analysis
>>
>> Dear Huang
>>
>> Comments in-line
>>
>> On 13/06/2020 20:57, Huang Wu wrote:
>>> Hi all,
>>>
>>> Greetings. I have some questions about publication bias/sensitivity 
>>> analysis. First, are publication bias and sensivity analysis the same 
>>> thing? If not, how are they different?
>> Publication bias is a subset of small study effects where you know the
>> aetiology of the small study effects. If you do not then it is safer to
>> refer to small study effects. A sensitivity analysis could be almost
>> anything but usually it manes fitting the model to one or more data-sets
>> similar to the original one. Examples are leave-one-out analysis, or
>> using only a subset of supposed higher quality studies.
>>
>>> Second, I saw people use funnel plot, fail-safe N, Egger�s regression 
>>> test to test publication bias 
>>> (http://www.metafor-project.org/doku.php/features), are these methods 
>>> applicable to multivariate meta-analysis?
>> Yes they are.
>>
>> Thanks.
>>> Third, what do you recommend to do publication bias/sensivity 
>>> analysis in multivariate meta-analysis? Thanks
>>>
>> I think what analysis you do will depend on the scientific question.
>>
>> Michael
>>
>>> Best wishes
>>> Huang
>>>
>>> Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for 
>>> Windows 10
>>>
>>>
>>> [[alternative HTML version deleted]]
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> R-sig-meta-analysis mailing list
>>> R-sig-meta-analysis using r-project.org
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>>

-- 
Michael
http://www.dewey.myzen.co.uk/home.html