1

Changing Trial Designs on the Fly

Janet WittesStatistics Collaborative

ASA/FDA/Industry Workshop September 2003

2

Context

Trial that is hard to redo• Serious aspect of serious disease• Orphan

3

Statistical rules limiting changes

To preserve the Type I error rate

To protect study from technical problems arising from operational meddling

4

Challenge

senserigor

5

6

Challenge

senselessrigor mortis

7

Scale of rigor

Over rigid Rigorous Prespecified methods for change – preserves Unprespecified but reasonable change Invalid analysis

• responders analysis• outcome-outcome analysis• completers

8

Consequences

No change during the study

OR

Potential for the perception that change caused by effect

9

Prespecified changes

Sequential analysis Stochastic curtailing Futility analysis Internal pilot studies Adaptive designs Two-stage designs

10

Problems

Technical Solved

Operational Risks accepted

Efficiency Understood

11

Add a DMC

What if it acts inconsistently with guidelines?

Something really unexpected happens?• DMC initiates change• Steering Committee initiates change

12

Reasons for unanticipated changes

Unexpected high-risk group Changed standard of care Statistical method defective Too few endpoints Assumptions of trial incorrect Other

13

Examples

1. Too much censoring; DMC extends trial

2. Boundary not crossed but DMC stops3. Unexpected adverse event4. Statistical method defective5. Event rate too low; DMC changes

design

14

#1 Endpoint-driven trial

Trial designed to stop after 200 deaths Observations different from expected

• Recruitment• Mortality rate

At 200 deaths, fu of many people<2 mo DMC: change fu to minimum 6 mo P-value: 0.20 planned; 0.017 at end

15

#2. Boundary not crossed

Endpoint• Primary: 7 day MI• Secondary: one-year mortality

Very stringent boundary

16

What DMC sees

Very strong result at 7 days No problem at 1 year Clear excess of serious adverse events

17

Haybittle-Peto bound (10%)

0

1

2

3

4

5

6

7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BoundsObserved

18

Haybittle-Peto bound (30%)

0

1

2

3

4

5

6

7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BoundsObserved

19

Haybittle-Peto bound (50%)

0

1

2

3

4

5

6

7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BoundsObserved

20

Haybittle-Peto bound (70%)

0

1

2

3

4

5

6

7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BoundsObserved

21

Haybittle-Peto bound (70%)

0

1

2

3

4

5

6

7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

BoundsObservedOB

22

#3. Unexpected adverse event: PERT study of the WHI

Prespecified boundaries forBenefit HarmHeart attack StrokeFracture PEColon cancer Breast cancer

23

Observations

Benefit Harm----- Stroke

Fracture PEColon cancer Breast cancer

Heart attack

24

Actions

Informed the women about increased risk of stroke, heart attack, and PE

Informed them again Stopped the study

25

#4. Statistical method defective

Neurological disease 20 question instrument Anticipated about 20% would not come Planned multiple imputation- results:

• Scale: 0 to 80• Value for ID 001: 30 38 ? 42 28 ?• MI values: -22, 176

26

#5. Too few endpoints

Example: approved drug Off-label use associated with AE

• Literature: SOC event rate: 20 percent

Non-inferiority design - = 5 Sample size: 800/group

27

Observation

400 people randomized 0 events What does the DMC do?

28

Choices

Continue to recruit 1600 Stop and declare no excess Choose some sample size Tell the Steering Committee to

choose a sample size What if n=1? 2? 5? 10?

29

Conclusions

Ensure that DMC understands role Separate decision-making role of

DMC and Steering Committee Distinguish between reasonable

changes on the fly and “cheating” Expect fuzzy borders

30

Technical

Changing plans can increase Type I error rate• We need to adjust for multiple looks• How do we adjust for changes?

31

Operational

Unblind assessments

Subtle change in procedures

In clinical trials, the FDA and SEC