[Rd] Routine and Deep testing with R CMD check

[Prof Brian Ripley]

On Wed, 11 Jun 2008, Bill Dunlap wrote:

> On Wed, 11 Jun 2008, Kevin R. Coombes wrote:
>
>> [1] Because of the need for a nightly build of BioConductor, the tests
>> (in the ./tests directory) of a package that run routinely as part of "R
>> CMD check" must complete in five minutes.
>> [2] Nontrivial regression testing of complex algorithms on real data can
>> easily take longer than five minutes.
>> [3] Maintaining and improving code that works on various kinds of
>> "omics" data is greatly facilitated by the inclusion of nontrivial
>> regression tests.
>>
>> Of course, points [1] and [3] are incompatible in the current setup.
>> Both could, however, be accommodated by changing the way "R CMD check"
>> runs test scripts. There are at least two ways this could be accomplished.
>
> Another approach is to have check use the new-to-2.8.0 function
> setSessionTimeLimit() to let the checker put a time limit on
> the tests.  In a time-constrained environment you could ask check
> to spend no more than 5 minutes running the tests but you could
> also set the limit to 3 hours or Inf or anything else.
>
> This would only be useful if the quicker tests were done first,
> so you'd have to alphabetize your test files.

Or to use environment variable(s) to select which tests to do (which is 
what I had started to write a reply to suggest when I got diverted to 
something more urgent).  E.g. 'foreign' has a slow test that needs 
Internet access, and it will become optional via an environment variable 
in the next release.

BTW, some of us run checks with a Unix-level time limit set: there are a 
few CRAN packages that infinite-loop on 64-bit systems so it is a 
necessary precaution.

> It might be nice if check could print the time it took to do
> each test.

That's an existing request for various parts of the checking procedure. 
When the time to run a package check jumps up, it is sometimes tedious to 
find out where the time went.  But the tests are currently scripted by 
make, so difficult to produce timings portably.

> Bill Dunlap

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595