[R-pkg-devel] Problems with too much testing

[Duncan Murdoch]

On 16/04/2021 9:49 a.m., J C Nash wrote:
> Another approach is to change the responsibility.
> 
> My feeling is that tests in the TESTING package should be modifiable by the maintainer of
> the TESTED package, with both packages suspended if the two maintainers cannot agree. We
> need to be able to move forward when legacy behaviour is outdated or just plain wrong. Or,
> in the case that I find affects me, when improvements in iterative schemes change iterates
> slightly. My guess is that Duncan's example is a case in point.
> 
> I doubt this will ever occur, as it doesn't seem to be the R way. However, I do know that
> improvements in methods are not going to CRAN in some cases.

In the cases I've been involved with the authors of the testing package 
have accepted suggested changes when I've made them:  I think that's 
also part of "the R way".  However, this takes time for both of us:  I 
need to understand what they are intending to test before I can suggest 
a change to it, and they need to understand my change before they can 
decide if it is acceptable, or whether further changes would also be 
necessary.

Github helps a lot with this:  if the testing package is there, I can 
quickly reproduce the issue, produce a fix, and send it to the author, 
who can tweak it if I've set things up properly.

For the kinds of changes you're making, I suspect relaxing a tolerance 
would often be enough, though if you switch algorithms and record that 
in your results, the testing package may need to replace reference 
values.  I think I'd be uncomfortable doing that.

Duncan Murdoch