[R-pkg-devel] [External] Re: Test fails on M1 Mac on CRAN, but not on macOS builder

[Pein, Florian]

Dear Duncan and Simon,
thank you both very much for you help.

I can make the test more informative and also break it down into substeps. But I am unsure whether CRAN policies allow to use their system for such testing steps. I rather think not. Though I must say that I still do not know how to otherwise fix the error. Can anyone, ideally a CRAN maintainer, confirm that this is okay?

Regarding the tolerance, the test compares to small integers. In this specific situation both sides are 5L on my local system. The tolerance is there to ensure that
ncol(compareStat) * testalpha is not something like 4.999999999 due to floating point approximations and we end up with 4L when as.integer() is applied. This is not happening in the concrete example, since ncol(compareStat) * testalpha = 5.67. I am very sure that the error is on the left hand side and compare is larger than 5.

In fact, tol in
rejected[, i] <- teststat[i, ] > ret[i] + tol
may need to be larger, since teststat contains quite large values. I think there is a realistic chance that a floating point error occurs at this point. But once again, I do not want to send a random guess to CRAN when I cannot test whether this has fixed the problem or not. I have tested the old code with --disable-long-double and compiler flags such as -ffloat-store and -fexcess-precision=standard and it works. I do not know to what degree this ensures that it works on all systems.

Many thanks and best wishes,
Florian

________________________________
From: Simon Urbanek <simon.urbanek using R-project.org>
Sent: 22 May 2023 00:25
To: Pein, Florian <f.pein using lancaster.ac.uk>
Cc: r-package-devel using r-project.org <r-package-devel using r-project.org>
Subject: [External] Re: [R-pkg-devel] Test fails on M1 Mac on CRAN, but not on macOS builder

This email originated outside the University. Check before clicking links or attachments.

Florian,

looking at the notes for 2.1-4 it says the tolerance has the wrong sign, i.e. you're adding it to the value on both sides of the interval (instead of subtracting for the lower bound). In your latest version the tolerances get added everywhere so that makes even less sense to me, but then I don't know what you actually intended to be completely honest. All I say, simply make sure you get the logic for the tolerance intervals right.

Cheers,
Simon


> On 19/05/2023, at 9:49 PM, Pein, Florian <f.pein using lancaster.ac.uk> wrote:
>
> Dear everyone,
> my R package stepR (https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fpackages%2FstepR%2F&data=05%7C01%7Cpein%40live.lancs.ac.uk%7C6cc9896064a34616d3c108db5a52aa15%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638203083314419558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=J1MeMbuYtFfl6Qc6N142pgiychUS1I5Tdc%2Bz005lsLc%3D&reserved=0<https://cran.r-project.org/web/packages/stepR/>) fails the CRAN package checks on M1 Mac, but the error does not occur on the macOS builder (https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmac.r-project.org%2Fmacbuilder%2Fsubmit.html&data=05%7C01%7Cpein%40live.lancs.ac.uk%7C6cc9896064a34616d3c108db5a52aa15%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638203083314419558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BRu4Nser4a1YggooWKHc%2BFewcIRbL3d08%2BeO8Ovv7N0%3D&reserved=0<https://mac.r-project.org/macbuilder/submit.html>). So, I am unable to reproduce the error and hence unable to fix it (starring at the code did not help either).
>
> The relevant part is
>
> * checking tests ...
>  Running ��testthat.R�� [35s/35s]
> [36s/36s] ERROR
> Running the tests in ��tests/testthat.R�� failed.
>
>> test_check("stepR")
>  [ FAIL 1 | WARN 0 | SKIP 23 | PASS 22741 ]
>
>  ���� Failed tests ��������������������������������������������������������������������������������������������������������������������������������
>  �w�w Failure ('test-critVal.R:2463:3'): family 'hsmuce' works �w�w�w�w�w�w�w�w�w�w�w�w�w�w�w�w�w�w�w�w
>  compare <= as.integer(ncol(compareStat) * testalpha + tolerance) is not TRUE
>
>  `actual`:   FALSE
>  `expected`: TRUE
>  sqrt
>  Backtrace:
>      �g
>   1. �|�wstepR (local) testVector(...) at test-critVal.R:2463:2
>   2.   �|�wtestthat::expect_true(...) at test-critVal.R:50:2
>
>  [ FAIL 1 | WARN 0 | SKIP 23 | PASS 22741 ]
>  Error: Test failures
>  Execution halted
>
>
> Has anyone an idea how to tackle this problem?
>
> The test code is long (a full version is available on CRAN). The following is the code part that I think is relevant (once again I cannot reproduce the error, so I am also unable to give a minimal reproducible example, I can only guess one):
>
> library(stepR)
> library(testthat)
>
> testn <- 1024L
> teststat <- monteCarloSimulation(n = 1024L, r = 100L, family = "hsmuce") # essentially a matrix with values generated by rnorm()
> testalpha <- 0.0567
> tolerance <- 1e-12
>
> ret <- critVal(n = 1024L, penalty = "sqrt", output = "vector", family = "hsmuce", alpha = testalpha, stat = teststat)
>
> statVec <- as.numeric(teststat)
> tol <- min(min(diff(sort(statVec[is.finite(statVec)]))) / 2, 1e-12) # different to the CRAN version to be robust to two values very close to each other
> rejected <- matrix(FALSE, ncol(teststat), nrow(teststat))
> compare <- integer(ncol(teststat))
>
> for (i in 1:nrow(teststat)) {
>  rejected[, i] <- teststat[i, ] > ret[i] + tol
> }
>
> for (i in 1:ncol(teststat)) {
>  compare[i] <- max(rejected[i, ])
> }
> compare <- sum(compare)
> expect_true(compare <= as.integer(ncol(teststat) * testalpha + tolerance), info = "sqrt")
>
> # version with an additional tolerance (suggested when the test failed on CRAN, but it does not help either)
> # both sides are small intgers, so it should not be needed
> expect_true(as.integer(compare + tolerance) <= as.integer(ncol(teststat) * testalpha + tolerance) + 0.5, info = "sqrt")
>
>
> I am not sure how to approach this problem, so any suggestions are very much welcomed.
> Many thanks and best wishes,
> Florian Pein
> (Lancaster university)
>
>
>       [[alternative HTML version deleted]]
>
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=05%7C01%7Cpein%40live.lancs.ac.uk%7C6cc9896064a34616d3c108db5a52aa15%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638203083314419558%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BvR2s9hArL7SXRBV67ABwDHLS%2BhI9cRtdig7IlLftpo%3D&reserved=0<https://stat.ethz.ch/mailman/listinfo/r-package-devel>


	[[alternative HTML version deleted]]