[Rd] Integer overflow in rhyper.c

[Jonathan Blood]

Hello R-Devel,

I have identified an integer overflow bug in rhyper that occurs when the
total population (`m + n`) exceeds `.Machine$integer.max`, provided that
the parameters themselves are still 32-bit integers and the number of draws
(`k`) is small.

The Issue:

In src/nmath/rhyper.c, internal calculations involving the total population
size appear to cast to 32-bit signed integers. This causes a wrap-around to
negative values when `m + n >= 2^31`, triggering a "SHOULD NOT HAPPEN"
warning in the `afc()` factorial helper and causing the function to always
return 0.

Interestingly, the bug is sensitive to the value of `k`, suggesting it only
affects one specific algorithmic branch (likely the H2PE algorithm setup).

Minimal Reproducible Example:

```r
set.seed(1)

# 1. Sum of population is exactly 2^31, k < 19
# This triggers warnings and incorrectly returns 0
rhyper(1, 2^30, 2^30, 18)
#> Warning in rhyper(1, 2^30, 2^30, 18): rhyper.c: afc(i), i=-2147483648 <
0 --
#> SHOULD NOT HAPPEN!
#> [1] 0

# 2. Subsequent calls with identical parameters no longer trigger the
warning,
# but consistently return 0
sum(rhyper(1e5, 2^30, 2^30, 18))
#> [1] 0

# 3. Same population, but k >= 19 behaves correctly (likely different
branch)
rhyper(1, 2^30, 2^30, 19)
#> [1] 12

# 4. Reducing population sum below 2^31 behaves correctly
rhyper(1, 2^30 - 1, 2^30 - 1, 18)
#> [1] 9

# 5. Parameters large enough to be treated as doubles behave correctly
rhyper(1, 2^31, 2^31, 18)
#> [1] 7

sessionInfo()
#> R version 4.5.3 (2026-03-11 ucrt)
#> Platform: x86_64-w64-mingw32/x64
#> Running under: Windows 11 x64 (build 22631)
#>
#> Matrix products: default
#>   LAPACK version 3.12.1
#>
#> locale:
#> [1] LC_COLLATE=English_United States.utf8
#> [2] LC_CTYPE=English_United States.utf8
#> [3] LC_MONETARY=English_United States.utf8
#> [4] LC_NUMERIC=C
#> [5] LC_TIME=English_United States.utf8
#>
#> time zone: America/New_York
#> tzcode source: internal
#>
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base
#>
#> loaded via a namespace (and not attached):
#>  [1] digest_0.6.38     fastmap_1.2.0     xfun_0.54         glue_1.8.0

#>  [5] knitr_1.50        htmltools_0.5.8.1 rmarkdown_2.30
 lifecycle_1.0.5
#>  [9] cli_3.6.5         reprex_2.1.1      withr_3.0.2
compiler_4.5.3
#> [13] rstudioapi_0.17.1 tools_4.5.3       evaluate_1.0.5    yaml_2.3.10

#> [17] rlang_1.1.7       fs_2.0.1
```

Regards,

Jon Blood

	[[alternative HTML version deleted]]