[R] OT Futility Analysis

[Marc Schwartz (via MN)]

On Wed, 2006-02-22 at 11:57 -0500, Kevin E. Thorpe wrote:
> Thank you Spencer and Steve for your helpful comments.  If I may, I
> would like to elaborate on some of the points you raise.

Kevin,

I am not sure if you received any offlist replies to your post. Given
the subject matter, I had considered that you might have.

You might find the following thread over in the MedStats group to be of
interest:

http://groups.google.com/group/MedStats/browse_frm/thread/144c97dc5cfc4f00?tvc=1

It discusses some of the issues of early stopping, in this particular
case due to "running out of funds". Some of the points raised below are
addressed in that thread.

MedStats, BTW, would be a good forum to consider for your query.

> Stephen A Roberts wrote:
> > I would take the line that if they hadn't pre-specified any stopping
> > rules, the only reason to stop is safety or new external data. I
> > would be very suspicious of requests from the steering committee to
> > stop for futility - they should be blinded so why are they thinking
> > futility unless results have leaked? I would argue that they are
> > obliged to finish the trial once they start.
> 
> In general I agree with this.  In this case the request for a futility
> analysis came from the sponsor (a drug company).  It is a classic case
> of company B buys company A and wnats to stop R&D on company A's drugs.
> Therefore the company was looking for a reason to stop.  Now that they
> will stop producing the drug used in the trial, recruitment will end
> before reaching its target.  Now the Steering Committee's point of
> view is that if there is any reasonable hope, they would find some
> other way to continue recruitment.  I am confident that results have
> not leaked.  I am well aquainted with the data management and blinding
> procedures in place for the trial.

Has the decision to cease production already been made or is the sponsor
still open to being "sold" on the idea of keeping the study going,
pending the outcome of your further work?

If production of the study treatment has already ceased, the ability of
the SC to make a business case to the sponsor may be a forgone
conclusion if there is insufficient product available to continue.

> > This is an example of the need to sort out these things in advance -
> > look up the stuff from the UK DAMOCLES project. The recent book
> > edited by DeMets et al (Data Monitoring in Clinical Trials: A Case
> > Studies Approach) is a good read on these sorts of issues and I think
> > there is a more statistical book from the same group of authors.
> 
> Thanks for the reference.  My library has it, so will give it a look.
> 
> > As far as software is concerned, futility analysis and conditional
> > power are simply standard analyses with made up data and more-or-less
> > justifiable assumptions.
> 
> I am also interested if there are good alternatives to conditional
> power for this type of scenario.
> 
> > Steve.
> > 
> > 
> > 
> >> 
> >> What does this particular Steering Committee think a "futility 
> >> analysis" is?  Do they have any particular reference(s)?  What do
> >> you find in your own literature review?
> >> 
> >> If it were my problem, I think I'd start with questions like that. 
> >> Your comments suggested to me a confounding of technical and
> >> political problems.  The politics suggests the language you need to
> >> use in your response.  Beyond that, I've never heard before of a
> >> "futility analysis", but I think I could do one by just trying to
> >> be clear about the options the Steering Committee might consider
> >> plausible and then comparing them with appropriate simulations --
> >> summarized as confidence intervals, as you suggest.
> 
> I did ask REPEATEDLY for guidelines from the steering committee, but
> none came or are likely to come.  In fact, they wanted me to come up
> with the recommendation, which I find entirely inappropriate, but here
> I am.  So, I don't think I'm confounded between techincal and political.

>From what I have seen of the regulatory guidance documents, the SC
should not provide you with any guidelines and the analysis should be
done independent of their input, since their input may be biased in
favor of the new drug. As I note below, the mere fact that the SC is
arguing in favor of continuing the study would suggest the possibility
of a priori bias. This is critical to consider, since there are no
pre-specified stopping rules.

In addition, these should not be done by you in isolation either and the
other clinical members of the DMC/DSMB should materially contribute to
the process. They should be just as clinically competent as any members
of the SC relative to putting forth reasonable assumptions upon which to
base any analyses.

> Basically, they want to stop if there is a low chance of rejecting the
> null hypothesis.  This is often referred to as conditional power or
> stochastic curtailment.  I recently saw a paper by Scott Emerson
> pointing out some problems (interpretation, relation to unconditional
> power).
> 
> As far as references, I have used a book by Jennisen and Turnbull in
> the past, but, as I recall, with the exception of stochastic
> curtailment, it assumes the trial was designed with group sequential
> methods.  I have also just found a 1988 Biometrics paper by Lan and
> Wittes on the B-value which I will read.

J&T has a case study with survival analysis as I recall. I don't have it
at hand at the moment.

Another paper that you might find helpful is:

Modifying The Design of Ongoing Trials Without Unblinding
Gould and Shih
Statist. Med., 17, 89 — 100 (1998)

I happened to locate a copy here during a Google search:

http://www2.umdnj.edu/~shihwj/papers/Gould%20and%20Shih%2098.pdf


There is also a Powerpoint presentation by Hung et al that you might
find of interest here:

http://www.fda.gov/cder/Offices/Biostatistics/Hung_etal_6/Hung_etal_6.PPT


BTW, though timing does not help you, there is a new book in process by
Proschan, Lan and Wittes due in June here:

http://www.springer.com/sgw/cda/frontpage/0,11855,4-10134-22-96964889-0,00.html


Two other documents that would be of value here, in a general guideline
sense, are the FDA's draft guidance on DMC's for sponsors (yes, I noted
you are "north of the border"):

http://www.fda.gov/cber/gdlns/clindatmon.htm

and of course ICH E9:

http://www.ich.org/cache/compo/475-272-1.html#E9

Both of which touch on the areas of interim analyses and early stopping.


It seems to me there there is a tension between the sponsor and the SC. 

In the former case, the sponsor is looking to make a business decision,
based upon what will clearly be less than perfect data, where
"scientific integrity" is or may be compromised. 

On the other hand, the SC is looking to argue for keeping the study
going and that presumably is based upon some a priori insight into the
likelihood of a favorable outcome for the new drug.

If there is to be a mid-course correction in the study (even though
there were no pre-specified stopping rules), there is a risk that the
mere decision to continue the study will somehow bias the future conduct
of it, given the present dynamics and "who knows what" about the
decision making process.

Some of the questions to be considered are what assumptions do you make
in the course of your assessment relative to future data and the need
for adjustments, if any, to the primary hypotheses of the study
(including alpha levels) given the knowledge that becomes available
during the un-planned interim analysis.

At the end of the day, you will need to consider the requirement to make
adjustments in the current study protocol, requiring re-submission and
re-approval by the requisite regulatory authority. It therefore would be
worthwhile to consider proactive communications with the regulatory
contacts for the study and discuss this situation with them to get their
"buy in" on any proposed approaches before taking any further steps.

<snip other content>

HTH,

Marc Schwartz