[R-sig-QA] Validation of an R installation

[Douglas Bates]

I am visiting Martin Maechler at ETH in Zurich, Switzerland for a month. 
  During this time I will also visit Tony Rossini and others at Novartis 
in Basel for a day.  One of the topics that I will discuss with Tony and 
have discussed extensively with Martin is ways in which we can help 
those useRs who face questions of whether R is "validated".  There was 
some discussion on R-Help about this recently and there has also been 
discussion in the past both on- and off-list.

Those who have compiled R from the sources may realize that "make check" 
and "make check-all" apply an extensive and ever-growing series of test 
cases to a version of R.  The same is true of R CMD check for packages.

In general a binary version of R or binary versions of packages do not 
include the tests directory.  There is no need - the tests are run at 
the time that the binary version is created.  However, it would be 
reasonable for a useR to have a mechanism to perform such tests on 
his/her installed version of R when desired, even if the version is from 
a binary installation.  One way we felt this could be achieved is to 
create a 'validate' package that contains the files used for the tests. 
  A useR can install the package then run the tests.  I believe that 
this would do substantially more validation of R that what is done for 
commercial statistical packages.

I have also proposed to Martin that we produce a document that describes 
the software development process for R, including the QA tools that are 
applied throughout the development process.

Others, especially Henric Nilsson, Frank Harrell Jr., Tony Rossini, and 
Jim Garrett, have proposed producing a document similar to the "white 
papers" produced by commercial companies specifically addressing the FDA 
guidelines in Title 21, Code of Federal Regulations, Part 11.  We hope 
that such efforts could be facilitated by what we propose.

Martin has proposed in private mail to Henric that we use the R-SIG-QA 
email list to communicate about this effort.  I think it would be an 
excellent use of this list and I hope others agree.

I include a list of topics that we felt might be discussed in a document 
about the development process for R core.  Suggestions and comments on 
this list and on the general idea of a validate package are welcome.

		   Quality assurance methods for R

			     R Core Team

Composition of R-Core Team

Access to the source code repository

Versions of R available - R-release, R-patched, R-devel

Routine tests applied during the development process

Release process
    Tests performed during release

Building R from source code

  configuration process (autoconf)
  range of tests covered by 'make check'
  range of tests covered by 'make check-all'

Binary installations

  Windows

    Configuration
    Checks performed on binary builds

  OS X for the Macintosh

    Configuration
    Checks performed on binary builds

Checking of packages on CRAN

  Daily package checking

Special capabilities

  Memory leak detection through valgrind

"War stories"

  Bugs found in other software (e.g. Lapack)