[R-pkg-devel] Submitting a package whose unit tests sometimes fail because of server connections

[Dirk Eddelbuettel]

On 16 April 2019 at 11:40, Will wrote:
| Some things I have considered include:
| 
|    1. Skipping all unit tests on CRAN (using something like
|    *testtht::skip_on_cran*). This would immediately fix the problem, and as
|    a mitigating factor we report automated test result and coverage on the
|    package's GitHub page (https://github.com/ropensci/suppdata).
|    2. Using HTTP-mocking to avoid requiring a call to a server during tests
|    at all. I would be uncomfortable relying solely on this for all tests,
|    since if the data hosters changed things we wouldn't know. Thus I would
|    still want the Internet-enabled tests, which would also have to be turned
|    off for CRAN (see 1 above). This would also be a lot of additional work.
|    3. Somehow bypassing the requirement for the unit tests to all pass
|    before the package is checked by the CRAN maintainers. I have no idea if
|    this is something CRAN would be willing to do, or if it is even possible.
|    It would be the easiest option for me, but I don't want to create extra
|    work for other people!
|    4. Slowing the tests with something like *Sys.sleep*. This might work,
|    but would slow the tests massively and so might that cause problems for
|    CRAN?
| 
| Does anyone have any advice as to which of the above would be the best
| option, or who I should email directly about this?

5. Run a hybrid scheme where you have multiple levels:

   5.1 Do what eg Rcpp does and only opt into 'all tests' when an overall
       variable is set; that variable can be set conveniently in .travis.yml
       and conditionally in your test runner below ~/tests/

       That way you can skip tests that would fail.

   5.2 Do a lot of work and wrap 3. above into try() / tryCatch() and pass
       if _your own aggregation of tests_ passes a threshold.

       Overkill to me.

   5.3 Turn all tests on / off based on some other toggle. I.e. I don't think
       I test all features of RcppRedis on CRAN as I can't assume a redis
       server, but I do run those tests at home, on Travis, ...

Overall, I would recommend to 'keep it simple & stupid' (KISS) as life is to
short to argue^Hdebate this with CRAN. And their time is too precious so we
should try to make their life easier.

Dirk

-- 
http://dirk.eddelbuettel.com | @eddelbuettel | edd using debian.org