[R] normality testing with nortest

[Duncan Murdoch]

On 5/22/2006 5:25 AM, Peter Dalgaard wrote:
> Uwe Ligges <ligges at statistik.uni-dortmund.de> writes:
> 
> 
>>> A = 0.1846, p-value = 0.9059
>>>
>>>> ad.test(rnorm(100,mean=5,sd=3))
>>> ...
>>> A = 0.5138, p-value = 0.1887
>>>
>>>     I mistakenly had thought the p-values would be more stable since I 
>>> am artificially creating a random normal distribution.  Is this expected 
>>> for a normality test or is this an issue with how rnorm is producing 
>>> random numbers?  I guess if I run it many times, I would find that I 
>>> would get many large values for the p-value?
>> Well, as many large values as small values, 5% significant differences 
>> for the 5% level....
>>
>> The following looks alright:
>>
>> hist(replicate(1000,  ad.test(rnorm(100,mean=5,sd=3))$p.value))
> 
> We see this misunderstanding worryingly often. Worrying because it
> reveals that a fundamental aspect of statistical inference has not
> been grasped: that p-values are designed to be (approximately)
> uniformly distributed and fall below any given level with the stated
> probability, when the null hypothesis is true.

I think it's the fallacious belief that the p-value measures the 
probability that the null hypothesis is true.  This is currently 
misunderstanding #1 in the Wikipedia entry for P-values.

(Google had me worried:  I searched for "probability that the null 
hypothesis is true" and found

> P-value - Wikipedia, the free encyclopedia
> The p-value is the probability that the null hypothesis is true, justifying the "rule" of considering as significant p-values closer to 0 (zero). ...
> en.wikipedia.org/wiki/P-value - 17k - Cached - Similar pages

This quote is preceded by:  "All of the following [...] statements are 
false:" Context is important! :-)

The vast majority of hits to that search also pointed out that this 
interpretation was incorrect.  A couple of counterexamples were a 
"research methods" page at a department of psychology, and
another at a medical school.  I'll send a copy of this note to people there.

Duncan Murdoch

> 
> There is no mechanism to give you "fewer significant" or "more stable"
> p-values, and a p-value close to one is no better an indication of a
> true null hypothesis than one of 0.5 or 0.25.
>