[R-pkg-devel] Problems with too much testing

[Steven Scott]

One of the basic principles of testing is "test interface, not
implementation."  Tests that violate this principle become "change detector
tests" instead of "correctness tests" and essentially prevent any
improvements to the code.  In C++ objects have "public" and "private"
methods, and the testing diktat is to test the public ones.  I hit this
issue a lot at Google when internal clients wrote change detector tests
effectively fencing in my tools.  My solution was to point out the
problematic tests and ask them not to do that.

The intent of R as a language is to be fast, lightweight, and flexible.  In
the interest of keeping it that way for users, it has grown large and
cumbersome for package authors.  If we try to tack on a "public/private"
object layout I'm afraid that would be yet another burden on package
authors.  The most obvious place I see to do this is in the NAMESPACE file,
which is already an ugly beast.  So while I agree with Duncan's
characterization of the problem, to my ears the "social solution" is the
cleanest one available, if we're talking about formal mechanisms.

Informally, both C++ and python use underscores in variable names to
suggest private data members.  You could just prefix "PRIVATE" to the name
of all the data members you don't want tested, where "PRIVATE" might become
"." or "_", or some other character.  I would argue against using
underscores, but I think I've already lost that battle.

On Fri, Apr 16, 2021 at 7:53 AM J C Nash <profjcnash using gmail.com> wrote:

> I'm generally in accord with Duncan on this. There are inevitably
> situations where general
> rules don't apply. Our challenge is to find practical ways to keep the
> overall workload of
> all participants in the process to a minimum.
>
> JN
>
> On 2021-04-16 10:18 a.m., Duncan Murdoch wrote:
> > On 16/04/2021 9:49 a.m., J C Nash wrote:
> >> Another approach is to change the responsibility.
> >>
> >> My feeling is that tests in the TESTING package should be modifiable by
> the maintainer of
> >> the TESTED package, with both packages suspended if the two maintainers
> cannot agree. We
> >> need to be able to move forward when legacy behaviour is outdated or
> just plain wrong. Or,
> >> in the case that I find affects me, when improvements in iterative
> schemes change iterates
> >> slightly. My guess is that Duncan's example is a case in point.
> >>
> >> I doubt this will ever occur, as it doesn't seem to be the R way.
> However, I do know that
> >> improvements in methods are not going to CRAN in some cases.
> >
> > In the cases I've been involved with the authors of the testing package
> have accepted suggested changes when I've made
> > them:  I think that's also part of "the R way".  However, this takes
> time for both of us:  I need to understand what
> > they are intending to test before I can suggest a change to it, and they
> need to understand my change before they can
> > decide if it is acceptable, or whether further changes would also be
> necessary.
> >
> > Github helps a lot with this:  if the testing package is there, I can
> quickly reproduce the issue, produce a fix, and
> > send it to the author, who can tweak it if I've set things up properly.
> >
> > For the kinds of changes you're making, I suspect relaxing a tolerance
> would often be enough, though if you switch
> > algorithms and record that in your results, the testing package may need
> to replace reference values.  I think I'd be
> > uncomfortable doing that.
> >
> > Duncan Murdoch
> >
>
> ______________________________________________
> R-package-devel using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>

	[[alternative HTML version deleted]]