Stopping Trials for Futility Ranjit Lall (May 2009)

Stopping Trials for Futility

Ranjit Lall

(May 2009)

OVERVIEW

• Summary of a 1 day workshop on “Stopping Trial for Futility”

• Funded by the RSS/NIHR HTA/MRC

• 11th November 2008 in London

• Speakers:

Jon Nicholl (ScHARR, University of Sheffield, Chair person)John Whitehead (Lancaster University)Andy Grieve (King’s College London)Karl-Fredrik Burman (Director of Statistical Science, Astra Zeneca)

WHAT IS FUTILITY?

• The term ‘futility’ is used to refer to the inability of the trial to achieve its objectives

• On-going monitoring of trials for safety, treatment efficacy or futility

• Early interims – review will focus more on safety, quality of conduct and trial integrity rather than efficacy.

• Later interims - may include formal efficacy or futility analysis

WHAT IS FUTILITY?

• A futility analysis causes a clinical trial to be stopped as soon as it becomes clear that a negative outcome is inevitable and thus it is no longer worthwhile continuing the trial to its completion.

PUBLICLY FUNDED TRIALS

• Early phase trials are supported by MRC

• Later phase trials supported by HTA

• HTA budget = £88 million pa• Nearly all spent on trials

• Typical design of phase III trials

– parallel group RCT, multi-centre, 3-5 years, £1million - £2.5 million, N = 400- 4000

FUTILITY DESIGNS

• Futility analyses are common in pharmaceutical trials

• NO HTA funded trials have planned a futility analysis

• Questions:

- Why is this?- Should futility analyses be designed into HTA trials?- If so, which trials?- And when and how should they be done?

Should HTA do futility analyses?

YES –

(i) As pointed out by Ware, Muller and Braunwald (1985),

“……………..early termination for futility could reduce enormous expenditure of resources, human and financial, involved in the conduct of trials that ultimately provide a negative answer regarding the value of the medical innovation….”.

(ii) It is ethically wrong to continue to recruit patients to trials with little hope of achieving helpful results

•


NO –

(i) All well conducted trials provide valuable evidence

(ii) The cost of designing trials with a planned futility analysis would outweigh any savings

•


• IT DEPENDS –

(i) In placebo controlled trials, futility suggests no worthwhile benefit (i.e. little chance of finding evidence that the intervention is better than placebo)

(ii) In head-to-head trials, treatment will be far from futile (so is futility analyses inappropriate in head-to –head trials)?

(iii) Futility analyses should be conducted in large expensive trials in which early stopping could save substantial resources

(iv)Futility analysis most useful where there are several treatments being compared, and there is a desire to eliminate one (for better resource use)

Extension grants

• 54% of MRC/HTA trials seek extensions

• Usually due to long set up/slow recruitment

• 55% of trials with extensions still don’t achieve N

• Cost of extensions for HTA = £50K - £1 million

• Current HTA ask for a ‘futility analysis’ in terms of conditional power on application.

• Almost never done as intended or refused

Fixed Interim analyses

• Focus Type I error (i.e. Rejecting the H0 when its true)

• Methods used include

- Pocock (1977)

- O’Brien and Fleming (1979)

- Haybrittle and Peto (1976)

- Lans and DeMets alpha spending function (1983, 1989)

Futility Analysis (cont’d)

• Two fundamental approaches:

(i) Using the above group sequential methods to compute the stopping boundaries for futility (in a similar way to computing stopping boundaries for efficacy)

• The boundaries can be defined to allow for early stopping for futility only, or to allow for early stopping for either futility or efficacy


(ii) Conditional power

• The power of a trial tells whether a clinical trial is likely to have high probability to detect a pre-defined treatment effect of interest.

• Very low power implies that a trial is unlikely to reach statistical significance even if there is a true effect.

• One should never begin a trial with low power.

• However, sometimes low power becomes apparent only after a trial is well under way.


(ii) Conditional power: based on the probability of a getting a statistically significant final result, conditional on the data collected so far (in the interim)

• Stochastic curtailment – a frequentist method• Predictive power – partially Bayesian methods• Predictive probability – fully Bayesian methods


(a) Stochastic curtailment:

• At planning stages, sample size based on (a) Difference - d; (b) power (80% or 90%); (c) type I error rate

• At the interim, recalculate the power to detect a difference d.

• Calculations involve the observed data and an assumption regarding the distribution of the future data

• This method assumes that the observed difference between the treatments at the interim stage is the true treatment difference, d.


(a) Stochastic curtailment:

• Detect e.g.<20% power no point continuing;

• Detect e.g.>50% power then continue.

• This cut-off is pre-specified (choice would depend on the risk/benefit considerations)

• Take other information into account before deciding on stopping or not extending.


(b) Predictive power:

• Conditional power calculations use the observed data and an assumed future treatment effect

• Not sure what assumptions to make

• Predictive power approach addresses this by averaging the conditional power function over a range of treatment difference, with weights based on the observed treatment effect


(b) Predictive probability:

• Prior probability: based on the interim data and the prior distribution of the parameter for the treatment effect;

• Statistical model: mathematical model which expresses the relationship of the parameters and the data

• Posterior probability: Probability of a clinically important treatment effect


• If early results show:

• Intervention is better than expected

conditional power will be high

• Intervention is worse that expected

conditional power will be low (unless your sample size increases)


The two most challenging aspects:

• the selection of optimal stopping thresholds;• the timing of the analysis

both of which require the balancing of various risks.

Ask the audience

(questions for the workshop)

Should we plan to do futility analysis in publicly-funded trials ?

0% 0%0%0%

0

5

1. Yes – for all trials (built into design)

2. Yes – but only for extension requests

3. Never

4. Undecided

Should we plan to do futility analysis in publicly-

funded trials?

1. Yes – for all trials (built into

design)

1. Yes – but only for extension

requests

1. Never

2. Undecided

21%

53 %

9%

17%

Yes – for all Yes but Never Undecided Appropriate only for Trials (built extension into design) requests

Should we plan to do futility analysis in publicly-

funded trials?

• May not be appropriate for all trials, but justification for doing/not doing should be clearly laid out in the application

• Should be made as a requirement by funders (and the funders should clearly lay out what is required by them)

• Clear distinction needs to be made between trial futility (e.g. lack of recruitment-operational aspect) and treatment futility (lack of efficacy - statistical)

In what circumstances are they appropriate?

0% 0%0%

1. Only in placebo-controlled trials

2. In both placebo-controlled and head-to head trials

3. Undecided

0

5


1. Only in placebo-controlled

trials

2. In both placebo-

controlled and head

to-head trials

3. Undecided

Only in In both placebo- undecided appropriate controlled and placebo-controlled head-to-head trials trials

29%

44%

27%


• It may be difficult to predict in advance

• Futility analyses may not be very appropriate for very large

pragmatic trials, where other information (other than efficacy) is

important

Should funding bodies insist upon futility analyses before granting extensions?

0% 0%0%

1. Yes

2. No

3. Undecided

0

5


1. Yes

2. No

3. Undecided

Yes No Undecided

47%

27% 25%


Should not be insisted upon, but if investigator refuses to do futility then justification should be given

What is futile?How low can the conditional power be before stopping a trial

0% 0% 0% 0% 0%0%0%0%0%

1. 1%

2. 5%

3. 10%

4. 15%

5. 20%

6. 30%

7. 40%

8. 50%

9. It depends 0

5

What is futile?

How low can the conditional power be before stopping a trial?

1. 1%

2. 5%

3. 10%

4. 15%

5. 20%

6. 30%

7. 40%

8. 50%

9. It depends

1% 5% 10% 15% 20% 30% 40% 50% It depends

0% 0% 5% 5% 5% 5% 5%

2%

72%

What is futile?

How low can the conditional power be before stopping a trial?

• Futility analysis should be focused on effect size (rather than

conditional power)

• A trial may be considered as futile in its ability to recruit patients,

but would add information to a evidence based data

Thank you !!

Stopping Trials for Futility Ranjit Lall (May 2009)

Documents

Transcript of Stopping Trials for Futility Ranjit Lall (May 2009)