[R-pkg-devel] CRAN submission error when running tests in testthat

[Ivan Krylov]

On Thu, 25 Nov 2021 12:33:01 +0100
Gábor Csárdi <csardi.gabor using gmail.com> wrote:

> Are you sure about this? I don't think testthat hides any output. R
> CMD check on the other hand AFAICT only shows the last 13 lines by
> default. See the _R_CHECK_TESTS_NLINES_ environment variable in
> https://cran.r-project.org/doc/manuals/r-devel/R-ints.html

Indeed you're right, R CMD check truncating the output is the main
problem here. With a full traceback, it would at least be possible to
see which dependency crashes R on CRAN Macs. I should have been more
clear in my e-mail.

By "testthat hiding output" I mean that there's no *.Rout[.fail] files
for every file in tests/testthat/*.R with line-by-line output of the
test code. In cases when R itself crashes in the middle of the test run,
these *.Rout.fail files could be useful to find out which line caused
the crash. With testthat, crashing the R process during tests results
in a single testthat.Rout.fail:

> > library(testthat)
> > library(package_name)
...
> > test_check("package_name")
> 
>  *** caught segfault ***
> address (nil), cause 'memory not mapped'
> 
> Traceback:
>  1: .C("foo")
>  2: eval(code, test_env)
...
> 23: test_dir("testthat", package = package, reporter = reporter,     ..., load_package = "installed")
> 24: test_check("package_name")
> An irrecoverable exception occurred. R is aborting now ...

Here, the traceback makes the answer obvious, but it's not always so.
With serious heap corruption, it's possible to get just:

> > test_check("package_name")
> free(): invalid pointer
> Aborted

With no indication of the call or the file causing the crash.

You are right that this is not the main problem here: if the test
output wasn't truncated by R CMD check, we would know the origin of the
crash, even without testthat.

-- 
Best regards,
Ivan