[Rd] Running package tests and not stop on first fail

[Hervé Pagès]

On 11/05/2016 01:53 PM, Martin Maechler wrote:
>>>>>> Oliver Keyes <ironholds at gmail.com>
>>>>>>     on Fri, 4 Nov 2016 12:42:54 -0400 writes:
>
>     > On Friday, 4 November 2016, Martin Maechler
>     > <maechler at stat.math.ethz.ch> wrote:
>
>     >> >>>>> Dirk Eddelbuettel <edd at debian.org <javascript:;>>
>     >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>     >>
>     >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>     >> My > proposed name '--no-stop-on-error' was a quick shot;
>     >> if | > somebody has a more concise or better "English
>     >> style" > wording | (which is somewhat compatible with all
>     >> the other > options you see | from 'R CMD check --help'),
>     >> | please > speak up.
>     >>
>     >> > Why not keep it simple?  The similar feature this most
>     >> > resembles is 'make -k' and its help page has
>     >>
>     >> > -k, --keep-going
>     >>
>     >> > Continue as much as possible after an > error.  While
>     >> the target that failed, and those that > depend on it,
>     >> cannot be remade, the other dependencies of > these
>     >> targets can be processed all the same.
>     >>
>     >> Yes, that would be quite a bit simpler and nice in my
>     >> view.  One may think it to be too vague,
>
>     > Mmn, I would agree on vagueness (and it breaks the pattern
>     > set by other flags of human-readability). Deep familiarity
>     > with make is probably not something we should ask of
>     > everyone who needs to test a package, too.
>
>     > I quite like stop-on-error=true (exactly the same as the
>     > previous suggestion but shaves off some characters by
>     > inverting the Boolean)
>
> Thank you, Brian, Dirk and Oliver for these (and some offline)
> thoughts and suggestions!
>
> My current summary:
>
> 1) I really don't want a  --<option-key>=value
>    but rather stay with logical/binary variables that "express
>    themselves"... in the same way I strongly prefer
>
>        if (A_is_special)       ....
>    to
>        if (A_special == TRUE)  ....
>
>    for a logical variable A_* .   Yes, this is mostly a matter
>    of taste,.. but related to how R style itself "works"
>
> 2) Brian mentioned that this is only about ./tests/ tests which
>    are continued, not about the Examples which are treated separately.
>    That's why we had contemplated additionally using 'tests' (because that's
>    the directory name used for unit/regression/.. tests) in the option
>    name.
>
> Even though Brian is correct, ideally we *would* want to also influence the
> examples' running to *not* stop on a first error..   However that would
> need more work, reorganizing how the examples are run and that may not be
> worth the pain.   However it should be considered a goal in the long run.

My name is Hervé, and I was not suggesting that what happens with the
examples should be changed. I was just preaching consistency (again
sorry) between what happens with the examples and what happens with
the tests. Why not simply change the latter? Do we really need an option
to control this? The behavior was changed for the examples a couple of
years ago and nobody felt the need to introduce an option to control
this at the time.

>
> After all that, I tend to remain with the original proposed name. It is at
> least easy to pronounce and spell correctly...

Unless --no-stop-on-error controls both (i.e. examples and tests), and 
both behave the same way by default, this is a misnomer.

If this option controls the tests only, it should be more something like
--no-stop-on-test-error. If in the long run, another option is added to
control the examples, then could be --no-stop-on-example-error.

But again, maybe all this is not needed at all. Maybe changing what
happens with the tests would be enough.

Cheers,
H.

>
> Martin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Hervé Pagès

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319