[Rd] Routine and Deep testing with R CMD check

[Kevin R. Coombes]

Hi,

I recently provoked some discussion over on the BioConductor developer 
list about testing and R CMD check. In brief, the salient points that 
arose from the discussion are:

[1] Because of the need for a nightly build of BioConductor, the tests 
(in the ./tests directory) of a package that run routinely as part of "R 
CMD check" must complete in five minutes.
[2] Nontrivial regression testing of complex algorithms on real data can 
easily take longer than five minutes.
[3] Maintaining and improving code that works on various kinds of 
"omics" data is greatly facilitated by the inclusion of nontrivial 
regression tests.

Of course, points [1] and [3] are incompatible in the current setup. 
Both could, however, be accommodated by changing the way "R CMD check" 
runs test scripts. There are at least two ways this could be accomplished.

One possibility would be to define a separate directory as part of the 
package structure to store the more complex, longer, "deep" tests. That 
is, one might have
	package/R
	package/man
	package/tests
	package/deeptests  [or some better name]
Then the default for "R CMD check" would be to run all the .R scripts in 
tests but none of the .R scripts in deep tests. However, one would also 
have to add an option, perhaps something like
	R CMD check --run-deeptests [package]
to run the more elaborate test scripts.

The second possibility would be to keep all the tests in the same 
(existing ./tests) directory but include a standard file in that 
directory to indicate which scripts are more complex.  This would be a 
little trickier to design, since it would be nice to keep the existing 
behavior for someone who ignores the new structure. One, could however, 
allow for something like

----------------------------
## CONTROLTESTS

complex: test1.R, test3.R
simple:  test2.R, test4.R
-----------------------------

The default (in the absence of A CONTROLTESTS file or for scripts in the 
directory that are not listed in the file) is to assume "simple", which 
would then get the existing behavior without changes to existing packages.

How hard would it be to implement one of these approaches for R CMD check?

  -- Kevin