[Rd] if(--as-cran)?

[Duncan Murdoch]

On 04/09/2012 3:44 PM, Terry Therneau wrote:
>
> On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
> > On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
> >> On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.duncan at gmail.com> wrote:
> >>
> >>
> >> >On 04/09/2012 8:20 AM, Terry Therneau wrote:
> >> >>
> >> >> On 09/04/2012 05:00 AM, r-devel-request at r-project.org wrote:
> >> >> > The issue is not just about "CRAN" vs "off CRAN".
> >> >> > It is good to think about a more general scheme of
> >> >> > "light testing" vs "normal testing" vs "extensive testing",
> >> >> > e.g., for the situation where the package implements
> >> >> > (simulation/bootstrap/ ..) based inference, and the developer
> >> >> > (but not only) should be able to run the extensive tests.
> >> >> >
> >> >> > Martin
> >> >>
> >> >> I agree with Martin. A mechanism to specify testing level would be the
> >> >>best.
> >> >> Then CRAN can choose to set that variable to "3" say, with level 1 for
> >> >>extensive and 2 for
> >> >> usual.
> >> >> I'm quite willing to put up with the nuisance of print()
> >> >>enclosures. I prefer it to
> >> >> having yet another way to subvert the evaluation model.
> >> >>
> >> >> I'm a believer in testing everything possible in my packages, and
> >> >>wear it it as a badge
> >> >> of honor that the survival package has 4 lines of R code in the tests
> >> >>directory for every
> >> >> 3 in the R directory. But CRAN only needs to run a small subset of
> >> >>this.
> >> >
> >> >We have a mechanism to specify testing level: the --as-cran flag. We
> >> >could presumably make it more elaborate by adding other flags, or option
> >> >levels, or whatever.
> >> >
> >> >What I think we shouldn't do is try to create an R-level test that says
> >> >
> >> > if (testingLevel() > 3) {
> >> > doSomething
> >> >}
> >> >
> >> >because tests can be turned on and off, individually. If testingLevel 3
> >> >specified tests (A, B, C), then is our testingLevel higher if we are
> >> >running tests (A, B, D, E, F, G)? Why not just test for the presence of
> >> >whichever test is most relevant to that particular code block, e.g.
> >> >
> >> > if ("D" %in% tests()) {
> >> > doSomething
> >> >}
> >>
> >>
> >> I would prefer the testingLevel() approach of the "D" %in% tests()
> >> approach, since testingLevel() provides a natural way to add successively
> >> greater test details without having to dig into the code to determine the
> >> set of tests.
> >
> > I don't see how you could possibly calculate a single number in a reasonable way. What is
> > the number that should be returned for (A, B, D, E, F, G)?
> >
> > Duncan Murdoch
>
> Duncan is leapfrogging ahead to another level that I hadn't thought of.  An example would
> be to divide my survival package as "cox", "parametric", "survfit", "all", some of whaich
> overlap. Interesting idea, but beyond what I'd use.  When I'm focused on a detail I run
> the test of interest directly, not through CMD check.  For me low, med, high intensity
> suffices with as-cran invoking the first level and high including those that take an
> exceptionally long time.
>    If you went to an A, B, C, ... approach what would be the as-cran default?
>
> Of course there is then the furthest level, which I recently instituted for survival due
> to the large number of dependencies: before posting a change download all the other
> dependent packages and run their tests too.  It's an overnighter, and in that case I'd
> want level=high.  Forewarned is forearmed.

I don't think I'm leapfrogging, I think you're asking about something 
new.  The thread was started by Spencer asking how to determine within 
an example whether he was being tested by CRAN, because he didn't want 
to run a long test there (it caused a timeout).  This is an instance of 
the general problem of tailoring example and test code to the tests 
being run.

The testingLevel() function is supposed to be a way to know that a 
certain level of testing is being done, to allow such tailoring. 
However, I don't think it's practical.  I think you can ask whether a 
specific test is being run (my "D" %in% tests() example), but you can't 
reasonably convert the set of tests chosen by a tester into a single number.

What I think you and Greg are talking about is something different. You 
are asking that we set up more suites of tests, corresponding to 
numerical levels.  Currently we have two suites:  the default, and the 
--as-cran suite.  But we also have completely customized suites, set by 
users who want to check specific things.  They can do that the way you 
do (by calling the tests explicitly), or by setting environment 
variables (as described in the Tools chapter of the R Internals manual).

Duncan Murdoch