Activating & Opening

Oncology Clinical Trials:

Process & Timing Analysis

David M. Dilts, PhD, MBADirector & Professor, Engineering Management Program, School of Engineering

Professor, Owen Graduate School of Management

Co-Director, Center for Management Research in Healthcare (cMRHc.org)

Alan B. Sandler, MDAssociate Professor of Medicine, Division of Hematology/Oncology

Medical Director, Thoracic Oncology Program

Director, Vanderbilt-Ingram Cancer Center Affiliates NetworkCo-Director, Center for Management Research in Healthcare (cMRHc.org)

Thank you to the study sites

Method

Part I: Process Mapping– Extensive visits at each site to document processes, loops and

decisions:

l Say…..: What participants say is done

l Should: What policies and procedures say should be done

l Do…...: What trial chart reviews shows was done

– Creation of process map

Part II: Process Timing– Identify calendar time for total process and major steps, and

potential influencers of the time

Part III: Accrual Data– Investigate actual accrual results of trials

• Dilts DM and Sandler AB (2006) “The Invisible Barriers to Opening Clinical Trials, J Clinical Oncology, 24(28): 4545-52

• Dilts DM et. al (2006) “Processes to Activate Phase III Clinical Trials in a Cooperative Oncology Group: The Case of

Cancer and Leukemia Group B,” J Clinical Oncology, 24(28): 4553-57.

• Dilts DM et. al (2008) “Development of Clinical Trials in a Cooperative Group Setting: The Eastern Cooperative

Group,” Clinical Cancer Research, 14(11):3427-33

Investigator-Initiated Trials High Level Process Map

Set-up Steps Trial Steps

Open Trial

(n=37) (n=15)

Comparison of Median Development TimeDevelopment vs. Operational Time by Phase*

* Sample: All ECOG Phase II and III studies activated between 1/2000-7/2006 and closed to accrual (n=52)

43.9%

56.1%

54.1%

45.9%

CIRB Review

111 days

Activation

7 days

Concept Development

193 days

Concept

Review

126 days

Concept Review

16 days

Concept Voting

2 days

Concept Approval

7 days

Study Team Teleconference

16 days

Grant Development

222 days

FDA Review

100 days

Regulatory Affairs Development

350 days

Forms / Database Development

434 days

CDE Compliance Review

240 days

Protocol Develoment

477 days

Protocol Review

277 days

Major Processing Activities

35 ft x 5 ft in 8pt

font

Process Flows for a Phase III

Cooperative Group Trial Creation

50 ft x 5 ft in 8pt

font

45 ft x 5 ft in 8pt

font

Process Steps for Phase III

Cooperative Group Trial Activation*

CRM (old)Steering Comm.

(new)

Process Steps* >481 >370 >300 >303

Working Steps >420 >317 >248 >244

Decision Points 61 42 52 59

Processing Loops 26 29 20 22

Stopping Points 13 16 11 12

* The process steps reported only show one loop in the process. Actual development frequently includes multiple loops

Level-0 Process Maps - CCC

P ro to c o l F in a liz a t io n

L O I a n d P ro to c o l D e v e lo p m e n t ( in c lu d in g In d u s tr y S p o n s o r

re v ie w )

P re l im in a r y B u d g e t A s s e s s m e n t

In fo r m e d

C o n s e n t D e v e lo p m e n t

F o rm a l

B u d g e t D e v e lo p m e n t

C o n tra c ts

N e g o t ia t io n s

R e g u la to ry

R e q u ire m e n ts

IR B

R e v ie w

F o rm s

D e v e lo p m e n t

F in a l C o n tra c t

S ig n in g

S tu d y

A c t iv a t io n

P R C R e v ie w

P O D R e v ie w

F D A

R e v ie w

37 ft x 3.5 ft in 8pt font

10

Process CountsComprehensive Cancer Centers

CCC 1* CCC 2 CCC 3 CCC 4

Process Steps 117 374 345 316

…Working Steps 64 292 272 263

…Decision Points 53 61 62 53

Processing Loops - 31 27 31

Stopping Points 19 21 11 18

11

Opening a Phase III Cooperative

Group Trial at a CCC

50 ft x 5 ft in 8pt

font

37’1’ x 3’6” ft in 8pt

font

CCC

45 ft x 5 ft in 8pt

font

12

Process Steps for Opening a

Phase III Cooperative Group Trial1

1. Representative cooperative group and Comprehensive Cancer Center2. Process steps reported only show one loop in the process. Actual development frequently includes multiple loops

Cooperative

Group

CTEP

/CIRBCCC Total

Process Steps >458 >216 >136 >810

…Working Steps >399 >179 >74 >652

…Decision Points 59 37 62 158

Potential Loops2 26 15 27 68

No. of Groups Involved 11 14 13 38

13

Reviews Required to Develop a

Cooperative Group Phase III Trial

CTCG CTEP CCC CCOP/Affiliates Others

Scientific Review Disease Site Committee Steering / CRM Protocol Review Feasibility Review Industry Sponsor

Executive Committee PRC Site Surveys

Protocol Reviews (2-4) CTEP Final

Data Management CRF Reviews (2-4) CDE Review

Database Review

Saftey / Ethics Informed Consent Local IRB Informed Consent CIRB

Regulatory Regulatory Review PMB Review FDA

RAB Review

Contracts / Grants Budget Industry Sponsor

Language

Study Start-up Start-up Review Start-up Review Start-up Review

Table 2. Types of Reviews Required to Develop a CTCG Trial: Categorized by Stakeholders

Abbreviations: CCC, Comprehensive Cancer Centers; CCOP, Community Clinical Oncology Program; CDE, Common Data Element; CIRB, Central Institutional Board Review, CRF, Case Report Form; CRM, Concept Review Meeting; CTCG, Clinical Trials Cooperative Group; CTEP, Cancer Therapy Evaluation Program; FDA, Food and Drug Administration; PMB, Pharmaceutical Management Branch, PRC, Protocol Review Committee; RAB, Regulatory Affairs Branch

14

Conclusion:

w Redundancy and Scope Creep

– Scope creep: when one group or organization

expands the scope of its authority or power

w Example: how many different scientific

reviews does a study require?

– Theory: more reviews = better study

– Practice: more reviews = slower opening

trials, no evidence of improvement

l Inspect in quality

15

How many of these steps add

value?

50 ft x 5 ft in 8pt

font

37’1’ x 3’6” ft in 8pt

font

CCC

45 ft x 5 ft in 8pt

font

16

Non-Value Added Statistics

at one CCC

Participants ---------Steps---------- Outcomes

Level 0 (Macro Process

Level)

Primary Other Value

Added

Non-

Value

Added

%

Value

Added

Decision

Points Decline

Accept

Or N/A

CCC 11 16 15 5 75% 13 2 1

Affiliates Main Office 3 - 7 3 70% 3 2 1

Affiliate Member Sites

(range of 3 sites)

3 1-8 6-15 1-5 75% 4-13 4-6 1

Sub-Process Level

IRB 7 3 12 26 32% 23 9 3

IRB Amendments 6 - 14 13 52% 13 6 3

SRC 4 - 13 11 54% 10 3 1

Regulatory and Clinical

Research Center

6 - 13 15 46% 7 1 1

17

Conclusion:

w There are far too many non-value

added (NVA) steps

w Value-Added steps

– Those that impact the quality, safety or

effectiveness of the study

w Note: if a study does not achieve

minimum accrual goals, then all steps

are NVA

18

Activation & Opening TimePhase III Cooperative Groups Trials

Group/CCC n Median Min Max

CALGB 13 784 537 1130

ECOG 28 808 435 1604

CCC 1 58 120 27 657

CCC 2 3 252 139 315

CCC 3 4 122 81 179

CCC 4 178 116 21 836w Notes:

– From initial conception at CTCG to activation; from receipt by CCC to opening– Time is calendar days, not work days– These are lower bounds because we only investigated survivors – Total time to open a study is the addition of Group time + CCC time

19

Total Time to Open a Phase III

Cooperative Group Study

Cooperative Group & CTEP/CIRB Processes

Median: 116 to 252 days*

Range: 21-836 days

Median: 784 to 808 days*

Range: 435-1604 days

Comprehensive Cancer Center

Processes

* Depending upon site, based on the Phase III trials studied

Protocol Finalization

LOI and Protocol Development (including Industry Sponsor

review)

Preliminary Budget Assessment

Informed Consent

Development

Formal Budget

Development

Contracts Negotiations

Regulatory Requirements

IRB Review

Forms Development

Final Contract

Signing

Study Activation

PRC Review

POD Review

FDA Review

Total Median Time from idea to opening~920 days (2.5 years)Range: 456 – 2440 days (1.25 - 6.7 yrs)

20

Development time for a phase III trial

from CTEP concept receipt to Activation

Figure 4. Development time for a phase III trial from CTEP concept receipt to Activation. Box ranges (H-Spread) indicate lower 25th and upper 75th percentile of the sample.

T-bars indicate the 95.0% confidence intervals by year. Dots indicate trial development time outside the bounds of the CI. Year indicates the year that the trial was activated.

Development time does not include the time involved to develop the concept prior to CTEP submission.

21

Phase III Activation Time

Time

Pre-Executive Committee Submission Development Time

Post-Executive Committee Submission Development Time

~600 Days (Median)

~800 Days (Median)

Sample: 26 Phase III CALGB and ECOG trials investigated in-depth

22

Where is the hold up?Total Time-Formal

Submission

Concept Review

Protocol Dev.

Protocol Review CIRB

CTEP Final Approval

Pre-Activation

295 26% 16% 13% 23%

330 20% 6% 2% 34% 19% 2%

407 22% 19% 16% 23% 2% 5%

424 36% 14% 11% 23% 0% 14%

465 21% 7% 46% 8% 7% 5%

483 59% 8% 12% 12% 3% 5%

490 22% 27% 14% 8% 0% 3%

491 53% 10% 5% 25% 0% 2%

524 26% 18% 29% 18% 1% 7%

551 14% 18% 23% 33% 1% 7%

560 25% 2% 10% 19% 15% 22%

580 6% 31% 16% 35% 3% 1%

589 42% 12% 15% 18% 2% 7%

596 34% 4% 6% 26% 22% 0%

614 4% 21% 29% 15% 1% 9%

668 15% 26% 35% 13% 2% 5%

758 3% 37% 32% 15% 2% 9%

829 10% 27% 36% 2% 20%

840 13% 24% 35% 16% 2% 7%

877 8% 30% 35% 2% 3%

885 35% 9% 11% 9% 1% 23%

930 31% 26% 6% 17% 8% 8%

931 12% 19% 11% 14% 7% 32%

1063 19% 52% 19% 8% 0% 1%

1248 45% 23% 10% 7% 5% 4%

1549 6% 35% 35% 18%

23

Conclusion:

w There is no Silver Bullet with regard to

time

w There is no single, clearly identified

bottleneck process

l Particularly, not IRB or FDA

w There are floating bottlenecks, or rate

limiting factors

24

Example Flow: E1301

Concept Review Days

Protocol Review Days

Total Days

Study Chair 49 122 171

Co-Operative Group 59 340 399

CTEP 98 184 282

CIRB n/a 123 123

Total 206 769 975

25

Reviews Requiring ResponseCalendar Days of Reviews and Group response by review type* for Phase III

Cooperative Group Studies (n=28 studies) activated from 2000 - 2005

* Reviews listed are only are partial list of required reviews. Other reviews including RAB, PMB, and CTSU are required but

were not available at the time of data collection. ** Group response time to industry cooperation not available

*** Recorded time for amendments only include study amendments prior to study activation

Review Time Group Response Time Total Time

Reviewer n median min max median min max median min max

Concept

CRM CTEP 14 60.0 15 104 71.5 1 368 126 16 411

CEP CTEP 4 48.0 19 66 35.5 22 84 83.5 41 150

Concept Re-review CTEP 3 6.0 1 6 17.0 1 56 23 7 57

Industry ** Industry 14 32.5 1 168

Protocol

PRC CTEP 33 32.0 5 69 32.0 1 188 63 8 230

Protocol Re-review CTEP 22 7.5 1 84 8.5 1 266 17 2 315

CIRB

CIRB CIRB 43 29.0 5 55 21.0 2 83 51 10 112

Re-review after CIRB CTEP 19 12.0 1 32 17.0 1 140 34 2 144

Amendment ***

Protocol Re-Review CTEP 2 9.0 1 17 5.5 5 6 14.5 6 23

CIRB CIRB 10 12.0 2 34 29.5 3 67 40.5 11 101

Re-review after CIRB CTEP 1 1.0 1 1 22.0 22 22 1 23 23

26

Conclusion:

w Only by working together can major

improvements be made

27

Discrete Event Simulation Model

of CALGB

28

High Level Process Flow for

Phase III Studies

29

Current

Simulation

Cooperative

Group

Improvement

CTEP

Improvements

CTEP and

Cooperative

Group

Improvements

Description As-Is performanceImproved Selective Study

Crit eria

Improved Review

Performance

Simult aneous

Improvments

n* 99 103 108 121

mean 837.29 791.51 653.01 252.78

min 222 209.07 214.93 214.35

max 3857.89 3576.71 4292.98 311.96

st. dev 760.75 729.3 670.63 24.14

Timing

Distribution

* Simulation period defined over a period of 5 years (1825 Calendar Days)

* Note: Axes on the Timing Distribution Graphs are different

Simulation Results of Working Together

30

Actual Accrual Per Trial RangesComprehensive Cancer Centers1

Accrual Per Trial

CCC 1 CCC 2 CCC 3 CCC 4 Total

N 148 323 104 323 898

0 20.9% 26.9% 26.9% 34.4% 28.6%

1-4 33.0% 32.3% 30.3% 31.3% 30.8%

5-10 19.3% 16.1% 22.7% 18.0% 18.6%

11-15 11.0% 7.3% 8.4% 4.3% 7.7%

16-20 3.7% 3.7% 3.4% 5.3% 4.1%

>20 12.4% 15.0% 7.6% 6.8% 10.1%

1Excludes pediatric studies

31

Accrual at CCCs: Cooperative v. non-Cooperative Group Trials

Cooperative Group Trials CCC 1 CCC 2 CCC 3 CCC 4Total/

Average

N 37 166 61 130 394

0 27.0% 38.6% 29.5% 46.9% 38.8%

1 to 4 37.8% 38.6% 39.3% 38.5% 38.6%

5 or more 35.1% 22.9% 31.1% 14.6% 22.6%

Non-Cooperative Group Trials CCC 1 CCC 2 CCC 3 CCC 4

Total/ Average

N 111 157 43 193 504

0 18.9% 14.6% 23.3% 25.9% 20.6%

1 to 4 30.6% 22.9% 9.3% 26.4% 24.8%

5 or more 50.5% 62.4% 67.4% 47.7% 54.6%

ASCO Poster 2008

32

All therapeutic, non-pediatric, Phase I, I/II, II, and III oncology trials evaluated by

CTEP that with submitted protocol opened to patient accrual, and subsequently

closed to accruals between January 1, 2000 and December 31, 2007 in the United States. Starting date is initial LOI/concept receipt to CTEP, end date is trial open for

accrual.

33

Final Accrual Performance* of CTEP-

Sponsored Phase III Trials

Opened and Closed between 2000 – 2007 (n=61)

•Accrual Performance is calculated based on the percentage of final accruals of studies completely closed to accrual compared to the projected minimum accrual goals as stated at the time of activation

•Pediatric studies are excluded from the sample

63.9% do not achieve minimum projected accruals at closure

5,776 patients accrued to these studies

49.2% do not achieve 25% of minimum projected accruals at

closure

36.1% meet or exceed minimum

projected accruals

34

Impact of Development Time on Accrual

CTEP Therapeutic Studies

All therapeutic, non-pediatric, Phase I, I/II, II, and III oncology trials evaluated by CTEP that began trial development with concept

submission, opened to patient accrual, and subsequently closed to accruals between January 1, 2000 and December 31, 2007 in the

United States. Starting date is initial LOI/concept receipt to CTEP, end date is trial open for accrual. N=553

35

“What one postpones, one actually

abandons”--Peter Drucker “The Effective Executive”, p. 110

36

Recommendations

w Immediate (Quick-Fix)

w Mid-range

w Long term

37

Immediate Recommendations

wImmediately start collecting & analyzing data

wSay what you mean & mean what you say

w“Just Say No”– Eliminate “entitlement culture”

wSet expiry dates:– If a study has not opened in a year, why continue

it?

wStop tweaking– “Two strikes and you’re out”

38

Analyzing & Collecting Data

w Analyzing Existing Data

– What are the characteristics of the studies with low accruals

and/or long development times?

– How many studies have not achieved their original

timelines?

– Plot time-in & time-out for all studies

w Collecting & analyzing additional data

– Identify & count reasons for:

l loops (revisions), study closures, or any specific reasons

for rework (runs chart)

l What “stips” must be done the PI & which can be done

by others

39

Say what you mean &

mean what you say

w Use standard and consistent terminology

– Approval versus Final-Approval versus Final-Final Approval

w Use schedules that are schedules, priorities that are

priorities and “no’s” that are “no’s”– If 50% of studies are “Priority 1”, then all studies are

“Priority 1”

– If the only penalty for being late is getting more time, then

why do something on time?

– If a study can be rejected at concept, protocol, and group but

still be opened, does “no” mean “no”?l 14 disapproved concepts had protocols created, 11 resulted in active

protocols

l 17 of withdrawn concepts had protocols created, 8 of which resulted

in active protocols

40

Just say “No”Controlling the Inflow

w Study Period 2000-2005 Averages/year

w Inflows ~43.6 LOI’s/yr

– (70.7% eventually accepted)

w Work-In-Process ~75 concepts or

protocols

w Outflow 18 protocols /yr

w If no new studies are accepted, it would

take 4.16 years to clear the queue

41

Besides too much work,

what do more concepts do?

wAttention Deficit

“What information consumes is rather

obvious: it consumes the attention of its

recipients. Hence a wealth of information

creates a poverty of attention, and a need to

allocate that attention efficiently among the

overabundance of information sources that

might consume it.”

Herbert Simon,

Nobel Laureate and Turing Prize Winner

42

“A ‘No’ uttered from deepest conviction is

better and greater than a ‘Yes’ merely

uttered to please, or what is worse, to avoid

trouble.”

--- Mahatma Gandhi

43

Medium Term

Recommendations

w Eliminate redundant, non-value added steps in the

entire process

– Cooperative Groups, CTEP, CIRB, & Comprehensive

Cancer Centers

w Benchmark other NIH Institutes &

Pharmaceutical firms

w Develop and utilize standards

– i.e., vary in critical scientific issues, not in

administrative processes

w Triage concepts using scientific merit and

operational complexity

44

Triaging Concepts:A Technique for Determining Entrance

Scientific

Merit

High

Low

High Low

Operational

Complexity

Fill

45

“An analysis of executive contributions

comes up with an embarrassing richness of

important tasks; any analysis of executives’

time discloses an embarrassing scarcity of

time available for work that really

contributes.”--Peter Drucker “The Effective Executive”, p. 100

46

Long Term Recommendations

w Change culture:

– From one of a feeling of entitlement

– Build a Mass Customization Process instead

of an Engineered-To-Order Process

– Start with a clean sheet of paper

w Use Focused Phase III Teams

47

Use Focused Phase III teams

w Create a dedicate team to rapidly develop Phase III concepts

w Team should include:– Co-Operative Group members

l Local Study Champion

l Statistician

l Protocol Specialist

l Regulatory Affairs

l Grants & Contracts

l Forms Developer

– CTEP members

l Statistician

l CDE

l Former CIRB Member

– Pharmaceutical Representative

w Goal: Rapidly, cooperatively, and with high quality, activate a

Phase III protocol in 90 days

48

“No task is completed until it has become

part of organizational action and behavior”

--Peter Drucker “The Effective Executive”, p. 109

“Unless a decision has ‘degenerated into work’ it is

not a decision; it is at best a good intention.”-- Peter Drucker “The Effective Executive”, p. 114

49

Thank you

www.cMRHc.org

50

Additional Potential Rules for

Identifying Priorities

w Pick for the future, not the past

w Focus on opportunities, not problems

w Choose your own portfolio, not what

others do

w Aim high, not safe & easy

51

Development TimeNew-To-The-World Goods or Services

41.7

24

56.4

144

0

20

40

60

80

100

120

140

160

1995 2004

Mo

nth

s

New-to-the-world products or services Pharmaceuticals

▼42%

▲155%

• Adams M, Boike D. Product Development and Management Association Foundation's Comparative Performance

Assessment Study: PDMA Foundation; 2004;

• Slater EE. Today's FDA. N Engl J Med 2005;352:293-7.

52

CTEP Acceptance Success Rates

by Trial Phase

Phase I I/II II III Other Pilot Total

Concepts n 1421 490 2649 513 25 52 5150

Gone Forward 33.4% 29.8% 35.3% 55.0% 28.0% 48.1% 36.3%

In Review 2.4% 3.5% 1.4% 1.9% 8.0% 3.8% 2.0%

Withdrawn / Replaced 13.7% 6.3% 13.6% 21.2% 24.0% 11.5% 13.7%

Disapproved 50.5% 60.4% 49.6% 21.6% 40.0% 36.5% 47.9%

Protocols n 388 166 1,031 192 150 72 1,999

Gone Forward 88.9% 80.7% 86.9% 83.3% 74.7% 81.9% 85.3%

In Review 4.6% 6.0% 4.3% 8.3% 8.7% 6.9% 5.3%

Withdrawn / Replaced 5.9% 9.6% 6.8% 6.3% 10.0% 6.9% 7.1%

Disapproved 0.5% 3.6% 2.0% 2.1% 6.7% 4.2% 2.3%

For all protocols received by CTEP, 1/2000 to 12/2007

53

Time From Concept Receipt to ActivationPhase III Therapeutic Studies activated through CTEP 1/2000 – 6/2007†

by Cooperative Group

Group Number

Median

Days**

Min

Days

Max

Days

CALGB 21 532 229 1526

ECOG 27 635 274 1532

SWOG 27 597 342 1706

COG 34 719 203 1908

GOG 9 402 298 665

NSABP 13 691 317 1655

RTOG 16 432 264 989

Other* 18 583 372 1697

Total/ Overall 165 594 203 1908

*Other: ACOSOG, IBCSG, NCCTG, NCIC, NCIMB, TX035

† these dates do not include the days for concept development

& approval at the cooperative group

** Concept approval time represented 8% to 39% of the days,

depending upon Group

54

YearSample Size

(trials)

Median Development Time (d)

Minimum Development Time (d)

Maximum Development Time (d)

IQR

2000 27 533 203 1114 402 - 661

2001 15 563 312 1706 427 - 943

2002 18 541 370 1110 464 - 724

2003 21 587 321 1908 415 - 889

2004 24 655 229 1423 427 - 854

2005 18 496 264 1142 369 - 659

2006 22 678 329 1655 542 - 896

2007 22 832 495 1776 680 - 1039

Total 167 602 203 1908 454 - 861

Table 2: Phase III CTEP Sponsors Clinical Trials by Activation Year. Sample size includes all CTEP-sponsored phase III clinical trials

activated from January 2000 to December 2007. Development time is defined from the receipt of the concept by CTEP to the time the

trial is activated for patient accruals. refers to the 25th and 75th percentile of the trials within each year and total years.

Development time for a phase III trial

from CTEP concept receipt to Activation

55

One Critical Step:

Executive Committee Approval

“[Once you select a clinical trial to pursue] it is like

pushing a boulder off a cliff…it is impossible to stop”

56

Besides too much work,

what do more concepts do?

wReduction in Attractiveness of Choice

“In the last few years, there has been an increase in

literature that indicates that people often have real

difficulty making choices, especially when those choices

involve tradeoffs (Luce, 1998; Luce, Payne, & Bettman,

1999). Moreover, the difficulties increase as the number of

attractive options increase (Iyengar & Lepper, 2000;

Schwartz, 2004). Everything suffers from comparison

(Brenner, Rottenstreich, & Sood, 1999). Multiple

attractive options may even lead to paralysis.”

Keys & Schwartz, “’Leaky’ Rationality”, Persp. On Psy. Sci

2(2):162-180, 2007