[R-pkg-devel] Problems with too much testing

[J C Nash]

I'm generally in accord with Duncan on this. There are inevitably situations where general
rules don't apply. Our challenge is to find practical ways to keep the overall workload of
all participants in the process to a minimum.

JN

On 2021-04-16 10:18 a.m., Duncan Murdoch wrote:
> On 16/04/2021 9:49 a.m., J C Nash wrote:
>> Another approach is to change the responsibility.
>>
>> My feeling is that tests in the TESTING package should be modifiable by the maintainer of
>> the TESTED package, with both packages suspended if the two maintainers cannot agree. We
>> need to be able to move forward when legacy behaviour is outdated or just plain wrong. Or,
>> in the case that I find affects me, when improvements in iterative schemes change iterates
>> slightly. My guess is that Duncan's example is a case in point.
>>
>> I doubt this will ever occur, as it doesn't seem to be the R way. However, I do know that
>> improvements in methods are not going to CRAN in some cases.
> 
> In the cases I've been involved with the authors of the testing package have accepted suggested changes when I've made
> them:  I think that's also part of "the R way".  However, this takes time for both of us:  I need to understand what
> they are intending to test before I can suggest a change to it, and they need to understand my change before they can
> decide if it is acceptable, or whether further changes would also be necessary.
> 
> Github helps a lot with this:  if the testing package is there, I can quickly reproduce the issue, produce a fix, and
> send it to the author, who can tweak it if I've set things up properly.
> 
> For the kinds of changes you're making, I suspect relaxing a tolerance would often be enough, though if you switch
> algorithms and record that in your results, the testing package may need to replace reference values.  I think I'd be
> uncomfortable doing that.
> 
> Duncan Murdoch
>