[R-pkg-devel] Submitting a package whose unit tests sometimes fail because of server connections

[Paul Gilbert]

5.4  In the spirit of simple & stupid you can also use the built in 
mechanism for doing this: organize some of your tests in subdirectories 
like inst/testWithInternet, inst/veryLongTests, 
inst/testsNeedingLicence, inst/testsNeedingSpecialCluster, etc. CRAN 
will only run the tests in the tests/ directory, but you can check them 
yourself  using

R CMD check  --test-dir=inst/testWithInternet   whatever.tar.gz

 > In a separate response On 4/16/19 2:06 PM, Steven Scott wrote:
 >  Just don't include the live fire stuff in the package.

Please do not do this. If you omit tests from your package then it 
cannot be properly checked by other people.

Paul Gilbert

On 4/16/19 2:16 PM, Dirk Eddelbuettel wrote:
> 
> On 16 April 2019 at 11:40, Will wrote:
> | Some things I have considered include:
> |
> |    1. Skipping all unit tests on CRAN (using something like
> |    *testtht::skip_on_cran*). This would immediately fix the problem, and as
> |    a mitigating factor we report automated test result and coverage on the
> |    package's GitHub page (https://github.com/ropensci/suppdata).
> |    2. Using HTTP-mocking to avoid requiring a call to a server during tests
> |    at all. I would be uncomfortable relying solely on this for all tests,
> |    since if the data hosters changed things we wouldn't know. Thus I would
> |    still want the Internet-enabled tests, which would also have to be turned
> |    off for CRAN (see 1 above). This would also be a lot of additional work.
> |    3. Somehow bypassing the requirement for the unit tests to all pass
> |    before the package is checked by the CRAN maintainers. I have no idea if
> |    this is something CRAN would be willing to do, or if it is even possible.
> |    It would be the easiest option for me, but I don't want to create extra
> |    work for other people!
> |    4. Slowing the tests with something like *Sys.sleep*. This might work,
> |    but would slow the tests massively and so might that cause problems for
> |    CRAN?
> |
> | Does anyone have any advice as to which of the above would be the best
> | option, or who I should email directly about this?
> 
> 5. Run a hybrid scheme where you have multiple levels:
> 
>     5.1 Do what eg Rcpp does and only opt into 'all tests' when an overall
>         variable is set; that variable can be set conveniently in .travis.yml
>         and conditionally in your test runner below ~/tests/
> 
>         That way you can skip tests that would fail.
> 
>     5.2 Do a lot of work and wrap 3. above into try() / tryCatch() and pass
>         if _your own aggregation of tests_ passes a threshold.
> 
>         Overkill to me.
> 
>     5.3 Turn all tests on / off based on some other toggle. I.e. I don't think
>         I test all features of RcppRedis on CRAN as I can't assume a redis
>         server, but I do run those tests at home, on Travis, ...
> 
> Overall, I would recommend to 'keep it simple & stupid' (KISS) as life is to
> short to argue^Hdebate this with CRAN. And their time is too precious so we
> should try to make their life easier.
> 
> Dirk
>