[R] [EXT] Re: A very small p-value

[Eik Vettorazzi]

Hi,
Stepping briefly outside the R context, I noticed a statistical point in 
the text you linked that, in my opinion, isn't quite right. I believe 
there's a key misunderstanding here: The statement that the z-test does 
not depend on the number of cases is incorrect. The p-value of the 
z-test is —just like other tests— very much dependent on the sample 
size, assuming the same mean difference and standard deviation.
The text you linked is actually calculating an Effect Size, which is 
(largely) independent of the sample size. Effect Size answers the 
question of how "relevant" or "large" the difference between groups is. 
This is fundamentally different from testing for "significant" differences.
Specifically, the crucial 1/\sqrt{n} term, which is necessary for 
calculating the standard error of the mean difference, seems to be 
missing from the presented formula for the z-score. I just wanted to 
quickly point this out.

Best regards

Am 27.10.2025 um 14:12 schrieb Petr Pikal:
> Hallo
> 
> The t test is probably not the best option in your case. With 95
> observations your data behave more like a population and you  may get
> better insight using z-test. See
> https://toxictruthblog.com/avoiding-little-known-problems-with-the-t-test/
> 
> Best regards.
> Petr
> 
> 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> Neobsahuje
> žádné viry.www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> 
> so 25. 10. 2025 v 11:46 odesílatel Christophe Dutang <dutangc using gmail.com>
> napsal:
> 
>> Dear list,
>>
>> I'm computing a p-value for the Student test and discover some
>> inconsistencies with the cdf pt().
>>
>> The observed statistic is 11.23995 for 95 observations, so the p-value is
>> very small
>>
>>> t_score <- 11.23995
>>> n <- 95
>>> print(pt(t_score, df = n-2, lower=FALSE), digits=22)
>> [1] 2.539746620181247991746e-19
>>> print(integrate(dt, lower=t_score, upper=Inf, df=n-2)$value, digits = 22)
>> [1] 2.539746631161970791961e-19
>>
>> But if I compute with pt(lower=TRUE), I got 0
>>
>>> print(1-pt(t_score, df = n-2, lower=TRUE), digits=22)
>> [1] 0
>>
>> Indeed, the p-value is lower than the epsilon machine
>>
>>> pt(t_score, df = n-2, lower=FALSE) < .Machine$double.eps
>> [1] TRUE
>>
>> Using the square of t statistic which follows a Fisher distribution, I got
>> the same issue:
>>
>>> print(pf(z, 1, n-2, lower=FALSE), digits=22)
>> [1] 5.079493240362495983491e-19
>>> print(integrate(df, lower=z, upper=Inf, df1=1, df2=n-2)$value, digits =
>> 22)
>> [1] 5.079015231299358486828e-19
>>> print(1-pf(z, 1, n-2, lower=TRUE), digits=22)
>> [1] 0
>>
>> When using the t.test() function, the p-value is naturally printed :
>> p-value < 2.2e-16.
>>
>> Any comment is welcome.
>>
>> Christophe
>>
>>> R.version
>>                 _
>> platform       aarch64-apple-darwin20
>> arch           aarch64
>> os             darwin20
>> system         aarch64, darwin20
>> status
>> major          4
>> minor          5.1
>> year           2025
>> month          06
>> day            13
>> svn rev        88306
>> language       R
>> version.string R version 4.5.1 (2025-06-13)
>> nickname       Great Square Root
>> -------------------------------------------------
>> Christophe DUTANG
>> LJK, Ensimag, Grenoble INP, UGA, France
>> ILB research fellow
>> Web: http://dutangc.free.fr
>> -------------------------------------------------
>>
>>
>>          [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> https://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help using r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide https://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-- 
Eik Vettorazzi

Universitätsklinikum Hamburg-Eppendorf
Institut für Medizinische Biometrie und Epidemiologie

Christoph-Probst-Weg 1
4. Obergeschoss, Raum 04.1.021.1

20246 Hamburg

Telefon: +49 (0) 40 7410 - 58243
Fax:     +49 (0) 40 7410 - 57790

Web: www.uke.de/imbe



--

_____________________________________________________________________

Universitätsklinikum Hamburg-Eppendorf; Körperschaft des öffentlichen Rechts; Gerichtsstand: Hamburg | www.uke.de
Vorstandsmitglieder: Prof. Dr. Christian Gerloff (Vorsitzender), Joachim Prölß, Prof. Dr. Blanche Schwappach-Pignataro, Corinna Wriedt
_____________________________________________________________________