[R] SAS or R software

[Henric Nilsson]

Marc Schwartz said the following on 2004-12-18 01:19:

> As you are likely aware, other statistically relevant issues are
> contained in various ICH guidance documents regarding GCP considerations
> and principles for clinical trials:
> 
> http://www.ich.org/UrlGrpServer.jser?@_ID=475&@_TEMPLATE=272

ICH E9 states that (p. 27):
"The computer software used for data management and statistical analysis 
should be reliable, and documentation of appropriate software testing 
procedures should be available."

Some commercial software vendors (SAS, Insightful, and StatSoft) offer 
white papers stating that their software can work within an 21 CFR Part 
11 compliant system.

http://www.sas.com/industry/pharma/develop/papers.html

http://www.insightful.com/industry/pharm/21cfr_part11_Final.pdf

http://www.statsoft.com/support/whitepapers/pdf/STATISTICA_CFR.pdf

Some commercial vendors (SAS and Insightful) also offers tools for 
validation of the installation and operation of the software. SAS has

http://support.sas.com/documentation/installcenter/common/91/ts1m3/qualification_tools_guide.pdf

and S-PLUS has validate().

As a statistical consultant working within the pharamceutical industry, 
I think that our clients find the white papers being some kind of 
quality seal. It signals that someone has actually thought about the 
issues involved, written a document about it, and even stated that it 
can be done. Of course, there's a lot of FUD going on here. But if our 
lives can be made simpler by producing similar white papers and QA 
tools, why not?

(But for some people, only SAS will do:
Last week we were audited on behalf of a client. One of the specific 
issues discussed were validation and the Part 11 compliance of S-PLUS. 
In this specific trial, data are to be transferred from Oracle Clinical 
-> SAS -> SPLUS, and they auditors were really worried about the first 
and last link of that chain. Finally, they suggested using only SAS... 
And in this particular case, Part 11 is really a non-issue since 
physical records exists (i.e. case report forms) and all final S-PLUS 
output and code will also be stored physically (i.e. print-outs) -- no 
need for electronic signatures here!)

> There is also a general guidance document for computer systems used in
> clinical trials here:
> 
> http://www.fda.gov/ora/compliance_ref/bimo/ffinalcct.htm
> 
> Though it is to be superseded by a draft document here:
> 
> http://www.fda.gov/cder/guidance/6032dft.htm 

 From the introduction (p. 2):
"This document provides guidance about computerized systems that are 
used to create, modify, maintain, archive, retrieve, or transmit 
clinical data required to be maintained and/or submitted to the Food and 
Drug Administration (FDA)"

The `retrieve' part is certainly applicable. If we regard R as 
off-the-shelf software, the guidance says (p. 11):
"For most off-the-shelf software, the design level validation will have 
already been done by the company that wrote the software. Given the 
importance of ensuring valid clinical trial data, FDA suggests that the 
sponsor or contract research organization (CRO) have documentation
(either original validation documents or on-site vendor audit documents) 
of this design level validation by the vendor and would itself have 
performed functional testing (e.g., by use of test data sets) and 
researched known software limitations, problems, and defect corrections. 
Detailed documentation of any additional validation efforts performed by 
the sponsor or CRO will preserve the findings of these efforts.

In the special case of database and spreadsheet software that is: (1) 
purchased off-the-shelf, (2) designed for and widely used for general 
purposes, (3) unmodified, and (4) not being used for direct entry of 
data, the sponsor or contract research organization may not have 
documentation of design level validation. FDA suggests that the sponsor 
or contract research organization perform functional testing (e.g., by 
use of test data sets) and research known software limitations,
problems, and defect corrections.

In the case of off-the-shelf software, we recommend that the following 
be available to the Agency on request:

* A written design specification that describes what the software is 
intended to do and how it is intended to do it;

* A written test plan based on the design specification, including both 
structural and functional analysis; and

* Test results and an evaluation of how these results demonstrate that 
the predetermined design specification has been met."

I think the guidance is quite clear here. We must prove to the FDA, at 
their wish, that the software used is working properly. In order to do 
this, we seem to need documents describing the development process and 
the QA tools used by R Core. An idea of what we'll need may be found in 
the `Computer Systems Validation in Clinical Research - A Practical 
Guide (Edition 1)' at

http://www.acdm.org.uk/public/publications/publications.htm

Especially section 2.4, 5 + subsections, 8 + subsections, and 9.7 + 
subsections seem relevant. (I've ordered the 2nd edition, but it hasn't 
arrived yet.)


Henric