[R-meta] publication bias

[Michael Dewey]

Dear Teresa

I accidentally deleted your e-mail so this only goes to the list

I think there are a number of important points to raise here.

All cut-offs are arbitrary so if you have a particularly informative one 
with 8 studies then show it and conversely.

There has been considerably criticism in recent years of funnel plots

@article{lau06,
    author = {Lau, J and Ioannidis, J P A and Terrin, N and
       Schmid, C H and Olkin, I},
    title = {The case of the misleading funnel plot},
    journal = {British Medical Journal},
    year = {2006},
    volume = {333},
    pages = {597--600}
}
@article{terrin05,
    author = {Terrin, N and Schmid, C H and Lau, J},
    title = {In an empirical evaluation of the funnel plot,
       researchers could not visually identify publication bias},
    journal = {Journal of Clinical Epidemiology},
    year = {2005},
    volume = {58},
    pages = {894--901}
}


If you do use a regression type test bear in mind that all the usual 
cautions about regression hold. If there is an isolated small study with 
all the others clustering near the top of the plot then the small study 
will have a disproportionate effect on the regression.

Michael

=================== Teresa's original e-mail said ============


I am wondering whether evaluation of publication bias through visual 
funnel plot inspection and performing Egger’s regression test is 
meaningful with less than 10 studies (Suggested cutoff by Sterne et al. 
( 2011).
To give a little bit more context: I have run a total of 5 meta-analyses 
depending on the outcome of a metabolit (either measured throughout the 
day: analyses on morning concentration, afternoon concentration and 
evening concentration, or measured after stress induction: analyses on 
physical stress, social stress and metabolic stress). Some studies 
provided values for more than one outcome (e.g. metabolit concentration 
in the morning and after physical stress induction, or metabolit 
concentration after a physical and after a social stressor) and were 
included in more than one analysis.
The 3 analyses on concentration throughout the day include 56, 19 and 11 
studies, respectively. The two analyses on stress include 8, 4 and 8 
studies, respectively.
My thesis advisor suggested to construct funnel plots and perform 
Egger’s test for each individual study, but I have quite frequently read 
that assessment of publication bias is not advisable with less than 10 
studies in an analysis. So, when I would apply this rule of thumb, I 
would assess publication bias only in the 3 analyses on metabolit 
concentration throughout the day. In some publications I have seen that 
they assessed potential of publication with less than 10 studies (either 
by setting another cutoff of for example 5 studies or by stating 
something like “the results have to be interpreted with caution as only 
few studies were included”- however in my opinion such a note seems a 
little weird in my case as I would show the reader 3 funnel plots and 
then question the results and interpretability).


-- 
Michael
http://www.dewey.myzen.co.uk/home.html