[Rd] CRAN test / avoidance

[Paul Gilbert]

( subject changed from Re: [Rd] R-devel Digest, Vol 115, Issue 18 )

I have the impression from this, and previous discussions on the 
subject, that package developers and CRAN maintainers are talking at 
cross-purposes. Many package maintainers are thinking that they should 
be responsible for choosing which tests are run and which are not run by 
CRAN, whereas CRAN maintainers may want to run all possible tests 
sometimes, or a trimmed down set when time constraints demand this. With 
good reason, CRAN may want to run all possible tests sometimes. There 
are too many packages on CRAN that remain there because they don't have 
any testing or vignettes, and very few examples. Encouraging more of 
that is a bad thing.

If I understand correctly, the --as-cran option was introduced to help 
developers specify options that CRAN uses, so they would find problems 
that CRAN would notice, and correct before submitting. The Rd 
discussions of this have morphed into a discussion of how package 
developers can use --as-cran to control which tests are run by CRAN.

I tend to be more sympathetic with what I call the CRAN maintainer view 
above, even though I am a package developer. I think packages should 
have extensive testing and that all the tests should go in the source 
package on CRAN, so the testing is available for CRAN and everyone else. 
(Although, it is sometimes not clear if CRAN maintainers like me doing 
this, because they are torn between time demands and maintaining quality 
- that is part of the confusion.)

The question becomes: how does information get passed along to indicate 
things that may take a long time to run. The discussion so far has 
focused on developers setting, or using, some flags to indicate tests 
and examples that take a long time. Another option would be to have the 
check/build process generate a file with information about the time it 
took to run tests, vignettes, and examples, probably with some 
information about the speed of the machine it was run on. Then CRAN and 
anyone else that wants to run tests can take this information into 
consideration.

Paul

On 12-09-19 10:08 AM, Terry Therneau wrote:
>> In general, as a package user, I don't want people to be able to
>> suppress checks on CRAN.  I want things fixed.
>>
>> So I am pretty sure there won't ever be a reliable "CRAN-detector" put
>> into R.  It would devalue the brand.
>>
>> Duncan Murdoch
>
> My problem is that CRAN demands that I suppress a large fraction of my
> checks, in order to fit within time constraints.  This leaves me with 3
> choices.
>
> 1. Add lines to my code that tries to guess if CRAN is invoker.  A cat
> and mouse game per your desire above.
>
> 2. Remove large portions of my test suite.  I consider the survival
> package to be one of the pre-eminent current code sets in the world
> precisely because of the extensive validations, this action would change
> it to a second class citizen.
>
> 3. Add a magic environment variable to my local world, only do the full
> tests if it is present, and make the dumbed down version the default.
> Others who want to run the full set are then SOL, which I very much
> don't like.
>
> I agree that CRAN avoidence, other than the time constraint, should be
> verboten.  But I don't think that security through obscurity is the
> answer.  And note that under scenario 3, which is essentially what is
> currently being forced on us, I can do such micshief as easily as under
> number 1.
>
> Terry Therneau
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel