[R-pkg-devel] Submitting a package whose unit tests sometimes fail because of server connections

[Iñaki Ucar]

On Tue, 16 Apr 2019 at 19:57, Will <will.pearse using gmail.com> wrote:
>
> Hello everyone,
>
> I'm sorry to bother you, but I would like some help getting a package (
> *suppdata*; https://github.com/ropensci/suppdata) available through CRAN.
> The package downloads supplementary materials from papers and data
> repositories on the basis of their DOI, making reproducible analyses a
> little easier to share and create.
>
> The package's unit tests involve downloading data, and when multiple
> requests for the same resource are sent from a single machine (as seems to
> be done by CRAN's testing servers) the data hosts will sometimes refuse the
> connection and so the package's tests will fail. I emphasise that the tests
> pass at Travis-CI (https://travis-ci.org/ropensci/suppdata) and OpenCPU (
> https://ropensci.ocpu.io/suppdata/info); I am confident this is a
> connection issue but I would be grateful to be shown I am wrong. I am not
> sure how to solve this problem, and was hoping you might have some advice.
> Some things I have considered include:
>
>
>    1. Skipping all unit tests on CRAN (using something like
>    *testtht::skip_on_cran*). This would immediately fix the problem, and as
>    a mitigating factor we report automated test result and coverage on the
>    package's GitHub page (https://github.com/ropensci/suppdata).

This doesn't seem a good idea.

>    2. Using HTTP-mocking to avoid requiring a call to a server during tests
>    at all. I would be uncomfortable relying solely on this for all tests,
>    since if the data hosters changed things we wouldn't know. Thus I would
>    still want the Internet-enabled tests, which would also have to be turned
>    off for CRAN (see 1 above). This would also be a lot of additional work.

What about mocking on CRAN and run Internet-enabled tests otherwise?

>    3. Somehow bypassing the requirement for the unit tests to all pass
>    before the package is checked by the CRAN maintainers. I have no idea if
>    this is something CRAN would be willing to do, or if it is even possible.
>    It would be the easiest option for me, but I don't want to create extra
>    work for other people!

I think it's acceptable to skip a test if something is not available,
but not the majority of them, for packages like this which basically
is about downloading stuff from APIs. Mocking on CRAN, as said before,
is the way to go here.

>    4. Slowing the tests with something like *Sys.sleep*. This might work,
>    but would slow the tests massively and so might that cause problems for
>    CRAN?

This is not a good idea either.

-- 
Iñaki Úcar