[Rd] Integer overflow in rhyper.c

[Martin Maechler]

>>>>> Jonathan Blood 
>>>>>     on Fri, 24 Apr 2026 08:21:03 -0400 writes:

    > Hello R-Devel,
    > I have identified an integer overflow bug in rhyper that occurs when the
    > total population (`m + n`) exceeds `.Machine$integer.max`, provided that
    > the parameters themselves are still 32-bit integers and the number of draws
    > (`k`) is small.

    > The Issue:

    > In src/nmath/rhyper.c, internal calculations involving the total population
    > size appear to cast to 32-bit signed integers. This causes a wrap-around to
    > negative values when `m + n >= 2^31`, triggering a "SHOULD NOT HAPPEN"
    > warning in the `afc()` factorial helper and causing the function to always
    > return 0.

    > Interestingly, the bug is sensitive to the value of `k`, suggesting it only
    > affects one specific algorithmic branch (likely the H2PE algorithm setup).

    > Minimal Reproducible Example:

    > ```r
    > set.seed(1)

    > # 1. Sum of population is exactly 2^31, k < 19
    > # This triggers warnings and incorrectly returns 0
    > rhyper(1, 2^30, 2^30, 18)
    > #> Warning in rhyper(1, 2^30, 2^30, 18): rhyper.c: afc(i), i=-2147483648 <
    > 0 --
    > #> SHOULD NOT HAPPEN!
    > #> [1] 0

    > # 2. Subsequent calls with identical parameters no longer trigger the
    > warning,
    > # but consistently return 0
    > sum(rhyper(1e5, 2^30, 2^30, 18))
    > #> [1] 0

    > # 3. Same population, but k >= 19 behaves correctly (likely different branch)
    > rhyper(1, 2^30, 2^30, 19)
    > #> [1] 12

    > # 4. Reducing population sum below 2^31 behaves correctly
    > rhyper(1, 2^30 - 1, 2^30 - 1, 18)
    > #> [1] 9

    > # 5. Parameters large enough to be treated as doubles behave correctly
    > rhyper(1, 2^31, 2^31, 18)
    > #> [1] 7

    > sessionInfo()
    > #> R version 4.5.3 (2026-03-11 ucrt)
    > #> Platform: x86_64-w64-mingw32/x64
    > #> Running under: Windows 11 x64 (build 22631)

....

    > Regards,
    > Jon Blood


Thank you, Jon.

I've finally addressed this and fixed it;  the internal check
needed    n+m > MAX_INT  instead of only checking the n and m.

  --> Fixed in svn rev 90223

I'm sorry this was not addressed earlier.

Best regards,
Martin