[R] t-test behavior given that the null hypothesis is true

[Duncan Murdoch]

On 13-01-09 8:24 AM, Pavlos Pavlidis wrote:
> Hi Ted,
> thanks for the reply. I use a similar code which you can see below:

You've got a typo.

>
> k <- 10000
> c <- 6
> rv <- array(NA, dim=c(k, c) )
> for(i in 1:k){
>    rv[i,] <- rnorm(c, mean=0, sd=1)
> }
>
> rv.t.pvalues <- array(NA, k)
>
> for(i in 1:k){
>    rv.t.pvalues[i] <- t.test(rv[i, 1:(c/2)], rv[i, (c/2+1):c],
> equal.var=TRUE, alternative="two.sided")$p.value
> }
>

That should be var.equal=TRUE, not equal.var=TRUE.  You're using Welch's 
approximation, and it is not an exact test.

Duncan Murdoch

> hist(rv.t.pvalues)
>
> The histogram is this one:
> *http://tinyurl.com/histogram-rt-pvalues-pdf
>
> *
> *all the best
> idaios
> *
>
>
> On Wed, Jan 9, 2013 at 12:29 PM, Ted Harding <Ted.Harding at wlandres.net>wrote:
>
>> On 09-Jan-2013 08:50:46 Pavlos Pavlidis wrote:
>>> Dear all,
>>> I observer a strange behavior of the pvalues of the t-test under
>>> the null hypothesis. Specifically, I obtain 2 samples of 3
>>> individuals each from a normal distribution of mean 0 and variance 1.
>>> Then, I calculate the pvalue using the t-test (var.equal=TRUE,
>>> samples are independent). When I make a histogram of pvalues
>>> I see that consistently the bin of the smallest pvalues has a
>>> lower frequency. Is this a known behavior of the t-test or it's
>>> a kind of bug/random number generation problem?
>>>
>>> kind regards,
>>> idaios
>>
>> Using the following code, I did not observe the behavious you describe.
>> The histograms are consistent with a uniform distribution of the
>> P-values, and the lowest bin for the P-values (when the code is
>> run repeatedly) is not consistently lower (or higher, or anything
>> else) than the other bins.
>>
>> ## My code:
>> N <- 10000
>> Ps <- numeric(N)
>> for(i in (1:N)){
>>    X1 <- rnorm(3,0,1) ; X2 <- rnorm(3,0,1)
>>    Ps[i] <- t.test(X1,X2,var.equal=TRUE)$p.value
>> }
>> hist(Ps)
>> ################################################
>>
>> If you would post the code you used, the reason why you are observing
>> this may become more evident!
>>
>> Hoping this helps,
>> Ted.
>>
>> -------------------------------------------------
>> E-Mail: (Ted Harding) <Ted.Harding at wlandres.net>
>> Date: 09-Jan-2013  Time: 10:29:21
>> This message was sent by XFMail
>> -------------------------------------------------
>>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>