[R] Waaaayy off topic...Statistical methods, pub bias, scientific validity

[Peter Langfelder]

>From a purely statistical and maybe somewhat naive point of view,
published p-values should be corrected for the multiple testing that
is effectively happening because of the large number of published
studies. My experience is also that people will often try several
statistical methods to get the most significant p-value but neglect to
share that fact with the audience and/or at least attempt to correct
the p-values for the selection bias.

That being said, it would seem that biomedical sciences do make
progress, so some of the published results are presumably correct :)

Peter

On Thu, Jan 6, 2011 at 9:13 PM, Spencer Graves
<spencer.graves at structuremonitoring.com> wrote:
>      Part of the phenomenon can be explained by the natural censorship in
> what is accepted for publication:  Stronger results tend to have less
> difficulty getting published.  Therefore, given that a result is published,
> it is evident that the estimated magnitude of the effect is in average
> larger than it is in reality, just by the fact that weaker results are less
> likely to be published.  A study of the literature on this subject might
> yield an interesting and valuable estimate of the magnitude of this
> selection bias.
>
>
>      A more insidious problem, that may not affect the work of Jonah Lehrer,
> is political corruption in the way research is funded, with less public and
> more private funding of research
> (http://portal.unesco.org/education/en/ev.php-URL_ID=21052&URL_DO=DO_TOPIC&URL_SECTION=201.html).
>  For example, I've heard claims (which I cannot substantiate right now) that
> cell phone companies allegedly lobbied successfully to block funding for
> researchers they thought were likely to document health problems with their
> products.  Related claims have been made by scientists in the US Food and
> Drug Administration that certain therapies were approved on political
> grounds in spite of substantive questions about the validity of the research
> backing the request for approval (e.g.,
> www.naturalnews.com/025298_the_FDA_scientists.html).  Some of these
> accusations of political corruption may be groundless.  However, as private
> funding replaces tax money for basic science, we must expect an increase in
> research results that match the needs of the funding agency while degrading
> the quality of published research.  This produces more research that can not
> be replicated -- effects that get smaller upon replication.  (My wife and I
> routinely avoid certain therapies recommended by physicians, because the
> physicians get much of their information on recent drugs from the
> pharmaceuticals, who have a vested interest in presenting their products in
> the most positive light.)
>
>
>      Spencer
>
>
> On 1/6/2011 2:39 PM, Carl Witthoft wrote:
>>
>> The next week's New Yorker has some decent rebuttal letters.  The case is
>> hardly as clear-cut as the author would like to believe.
>>
>> Carl