EXL Clinical Quality Oversight Forum

Managing Risk and Ensuring Effective

Oversight in Outsourced Clinical Trials

Clinical Quality Oversight Forum

2

Panel Members

Mitch Katz Executive Director Medical Research Operations, PURDUE PHARMA LP

David Marks Executive Vice President, Quality Management, RESEARCH PHARMACEUTICAL SERVICES, INC. (RPS)

David Nickerson Senior Director, Clinical Quality Management, PFIZER


3

Today’s Topics

•  Background: Challenges in quality management

•  Part One: Proactive quality management and effective oversight

•  Part Two: Risk sharing •  Part Three: Risk assessment and risk management


Background and Overview

5

Mission: To accelerate the development of best practice approaches and industry standards for the proactive quality

management of outsourced trials Lead Sponsors and CROs in optimizing their approaches to proactive quality management with an emphasis on bringing them into greater alignment.

Avoca Quality Consortium

6

Avoca Quality Consortium Pharma/Biotech Industry Participation to Date

**Roche is also Consortium Members

7

Avoca Quality Consortium CRO Industry Participation

8

Are there quality issues with outsourced

clinical trials?

Challenges in Quality Management

9

Challenges in Quality Management Disconnect between sponsors and CROs regarding perceptions of

quality delivered by CROs

Sponsor: Overall, how satisfied are you with the quality delivered by your

Clinical Service Providers (including but not limited to CROs)?

2%

45%

30%

20%

3% 17%

70%

10% 3%

CRO: Overall, how satisfied are you with the quality that your company

delivers for its sponsors?

N=245* N=200*

5%

13%

8%

8%

15%

15%

40%

10%

3%

11%

27%

95%

100%

67%

77%

83%

64%

58%

20%

60%

71%

57%

66%

60%

20%

15%

15%

20%

20%

20%

14%

23%

16%

10%

8%

6%

7%

20%

10%

14%

17%

7%

2%

0% 20% 40% 60% 80% 100%

A

B

C

D

E

F

G

H

I

J

K

-All-

Very satisfied Somewhat satisfied Neither satisfied nor dissatisfied

Somewhat dissatisfied Very dissatisfied

*2013 Avoca Quality Consortium Assessment

10

Overview: Challenges in Quality Management

“Reactive” vs “proactive” approaches to quality management: a look at whether quality issues exist

Areas with relatively high satisfaction:

●  Compliance with SOPs and other written procedures

●  Data quality and integrity

●  Audit plans and execution

Areas with relatively high dissatisfaction:

●  Oversight of third party vendors

●  Governance of quality

●  Communications surrounding quality

●  Availability of quality personnel for projects

●  Efficiency/timeliness in achieving clean data

●  Adherence to monitoring plan

3.3 – 3.5 (on scale of 1 to 5)

2.9 – 3.2 (on scale of 1 to 5)

11

Quality by Task: Comparison of Sponsor and CRO Results (2012 Avoca Quality Consortium Assessment)

Sponsors: On average, how satisfied are you with

your clinical service providers’…

CROs: On average, how would you rate your

company's…

Compliance with SOPs and other written procedures 3.5 3.7

Data quality and integrity 3.4 3.8

Audit plans and execution 3.3 3.6

Adequacy of Monitoring Plan 3.2 3.5

Document control 3.2 3.3

Adherence to Monitoring Plan 3.2 3.4

Efficiency/timeliness in achieving clean data 3.2 3.5

Monitoring of protocol compliance 3.1 3.7

Availability of quality personnel for my projects 3.1 3.2

Management of protocol compliance 3.1 3.6

Staff training 3.0 3.3

Site training 3.0 3.4

Governance of quality (e.g. accountability, management system, leadership support) 2.9 3.4

Oversight of third party vendors (e.g. labs, IVRS vendors, etc.) 2.9 3.3

Communications surrounding quality 2.9 3.4

12

Research Finding:

Strong correlation between taking

proactive approaches to quality management

and satisfaction with the quality of deliverables.

Sponsor Proactivity Drives Satisfaction

12

0

1

2

3

4

5

0 1 2 3 4 5

Sa*sfac*on

with

the "quality"

delivered

by clinical se

rvice providers

The (sponsor) teams with which I work iden*fy all processes/deliverables for which quality expecta*ons of clinical service

providers need to be set

0

1

2

3

4

5

0 1 2 3 4 5

Sati

sfac

tion

wit

h th

e "q

ualit

y"

deliv

ered

by

clin

ical

ser

vice

pr

ovid

ers

The (sponsor) teams with which I work effectively communicate their expectations regarding quality to their

clinical service providers

13

Clarity of expectations: scores range between 2.4 on the low end to 4.8 on the high end (based on a scale of 1 to 5)

Avoca Relationship Assessments

Clarity of Roles and Expectations vs. Sponsor’s Overall Satisfaction: Across Strategic Partnerships*

*p<.05

Statistically significant correlation across strategic

partnerships:

Clarity of expectations, roles, and responsibilities, and

overall satisfaction with the work performed.

14

Data suggests room for improvement in proactive approaches

Many sponsors companies have reported difficulty in operationalizing approaches for

identifying and communicating expectations. The (sponsor) teams with which I work effectively communicate their

expectations regarding quality to their clinical service providers.

15

Operationalizing Proactive Communication

16


17


Construction Industry ü  Has operationalized

proactive, systematic and integrated communication strategies

Although buildings are more

complex and sophisticated, building failure is exceedingly rare:

(.00002 percent/year)

“The biggest cause of serious failure in this business is a failure of communication.”

Finbarr O’Sullivan, Project Executive, Moriarty & Associates, Project Executive, Russia Wharf office building on Boston Waterfront

18

Leuchten’s Art Studio/Garage

Subset of problems to date: ü  Foundation footings

poured based on old set of drawings

ü  Scheduling snafu’s between builder and roofer

ü  Failed inspection due to issues with electrical and plumbing

Root causes of issues: Problems

with proactive communication,

collaboration and clarity of expectations

19

Themes to explore today

•  Proactive communication •  Collaboration •  Clarity of expectations

...and impact on quality of outsourced clinical trials


Part One:

Proactive quality management and effective oversight

21

•  Research Topics –  Micromanagement –  Documentation around roles and responsibilities –  Overseeing work at a strategic level rather than tactical –  Definition of roles to minimize duplication of effort –  Efficiency in use of resources –  Use of technology to promote efficient oversight practices

Proactive Quality Management and Effective Oversight `

22

What do you think?

Polling question for audience: Sponsors: My company is efficient in the use of resources applied to outsourced projects. CROs: Sponsors that I work with are efficient in the use of resources applied to outsourced projects. A.  Strongly agree B.  Agree C.  Neither agree nor disagree D.  Disagree E.  Strongly disagree

23

21%

13%

5%

6%

6%

2%

30%

42%

34%

34%

28%

18%

34%

21%

34%

34%

29%

31%

11%

20%

18%

18%

32%

34%

4%

5%

8%

8%

6%

15%

The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of

deliverables meets expectations.

My company clearly documents CRO oversight practices, roles, and responsibilities.

I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic"

rather than detailed level.

My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort.

My company is efficient in the use of resources applied to the oversight of outsourced trials.

The technology systems used by my company and its CRO partners promote efficient oversight practices.

Sponsor Perceptions of Oversight Practices Mean N

3.5 219

3.4 219

3.1 203

3.1 218

2.9 218

2.6 215

Perceptions of Oversight Practices Consortium Sponsor Findings

5 Strongly Agree

4 3 Neither Agree nor Disagree

2 1 Strongly Disagree

24

8%

5%

3%

4%

3%

4%

47%

34%

30%

14%

23%

22%

32%

34%

33%

44%

35%

29%

11%

23%

25%

32%

31%

29%

3%

5%

10%

5%

8%

16%

The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets

expectations.

The sponsors with which I work clearly document CRO oversight practices, roles, and responsibilities.

The technology systems used by my company and its sponsor partners promote efficient oversight practices.

The sponsors with which I work are efficient in the use of resources applied to the oversight of outsourced trials.

The sponsors with which I work clearly define the roles of internal and CRO staff so as to minimize duplication of

effort. There is a direct positive relationship between the

intensity of sponsor oversight and the quality of CRO deliverables.

CRO Perceptions of Sponsor Oversight Practices Mean N

3.5 157

3.1 153

2.9 159

2.8 152

2.8 149

2.7 158

Perceptions of Oversight Practices CRO Findings

5 Strongly Agree

4 3 Neither Agree nor Disagree

2 1 Strongly Disagree

25

Statements Evaluated by Sponsors Sponsor Average

CRO Average Statements Evaluated by CROs

The operational teams involved in overseeing our CROs consistently micromanage to ensure that the quality of

deliverables meets expectations. 3.5 3.5

The sponsors with which I work consistently micromanage to ensure that the quality of deliverables meets expectations.

My company clearly documents CRO oversight practices, roles, and responsibilities. 3.4 3.1 The sponsors with which I work clearly document CRO

oversight practices, roles, and responsibilities.

My company clearly defines the roles of internal and CRO staff so as to minimize duplication of effort. 3.1 2.8 The sponsors with which I work clearly define the roles of

internal and CRO staff to minimize duplication of effort.

I personally have received high-quality deliverables from CROs, even when overseeing their work at a "strategic"

rather than detailed level. 3.1 2.7

There is a direct positive relationship between the intensity of sponsor oversight and the quality of CRO deliverables.

My company is efficient in the use of resources applied to the oversight of outsourced trials. 2.9 2.8 The sponsors with which I work are efficient in the use of

resources applied to the oversight of outsourced trials.

The technology systems used by my company and its CRO partners promote efficient oversight practices. 2.6 2.9 The technology systems used by my company and its

sponsor partners promote efficient oversight practices.

N≥203 N≥149

Perceptions of Oversight Practices 5 = Strongly Agree, 1 = Strongly Disagree

Perceptions of Oversight Practices Sponsor and CRO Findings

26

Best Practices in Quality Management

Operationalizing approaches to

setting and communicating

expectations drives consistency and

increases quality

27

Operationalizing Proactive Approaches to Quality Management and Oversight

Governance / Organiza*onal Construct

Technical / Project

Oversight

Processes

Communica*on

Oversight Leadership

Requirements

Metrics / Analy*cs / Technology

Roles / Responsibili*es

Proac*ve Risk Management / Con*ngencies

Effective Oversight

Define

Plan

Lead

Decide

Measure

Deliver

Proactive Quality

Management

28

Verbatim Comments

“[Sponsors don’t give us] ownership of decisions that we should have. This delays our deliverables which Sponsors will hold us accountable for in the end.” [CRO]

“CRO's don't like to problem solve and offer suggestions. They'd rather just do as they are told.” [Sponsor]

“One Sponsor that I work with is new to fully outsource trials, so at the beginning of our working relationship, they micromanaged us. This has slowly relaxed as we showed that we were consistently meeting project deliverables. “ [CRO]

Panel Discussion

Part Two:

Risk-sharing

31

Risk-sharing between Sponsors and Providers

Types of Risk-sharing

● Provider bonuses for achieving milestones and targets

● Provider penalties for missing milestones and targets

● Guarantees of continued work/revenue stream in exchange for provider commitments

● Provider stake in outcome of study

(e.g., company stock)

32

0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%

Provider penalties for failure to achieve milestones/targets

Provider bonuses for achievement of milestones/targets

Guarantee of continued work/revenue stream in exchange for provider commitments

None of the above

Strategic partnerships/alliances (N=205) Preferred provider relationships (N=194) Transactional relationships (N=217)

Risk-Sharing Models Used by Type of Relationship

Risk-Sharing Model Usage Sponsor Findings

Risk-sharing models are used more commonly in strategic partnerships or preferred provider agreements than in transactional relationships.

33

What is your experience?

Polling question for audience: With which risk-sharing method have you had the most positive experience? A.  Provider bonuses for achieving milestones and targets

B.  Provider penalties for missing milestones and targets

C.  Guarantees of continued work/revenue stream in exchange for provider commitments

D.  Provider stake in outcome of study (e.g., company stock)

34

41%

30%

25%

50%

62%

21%

48%

61%

52%

47%

34%

55%

10%

9%

23%

3%

4%

24%







Experience/Satisfaction with Risk-Sharing Models Used

Primarily Positive

A mix of Positive and Negative

Primarily Negative

Risk-Sharing Satisfaction Sponsor and CRO Findings

N

87

94

104

92

74

95

Spon

sors

CR

Os Consor*um sponsors report more posi*ve experiences with the use of provider bonuses

compared to provider penal*es.

35

41%

30%

25%

50%

62%

21%

48%

61%

52%

47%

34%

55%

10%

9%

23%

3%

4%

24%







Experience/Satisfaction with Risk-Sharing Models Used

Primarily Positive

A mix of Positive and Negative

Primarily Negative

Risk-Sharing Satisfaction Sponsor and CRO Findings

N

87

94

104

92

74

95

Spon

sors

CR

Os

Similar results are reported by sponsors and CROs in regard to use of provider penalties for failure to achieve milestones/targets.

Panel Discussion

Part Three:

Risk Assessment and Risk Management

38

37%

18%

20%

13%

12%

Usage Frequency of Systematic Risk Assessment Processes

Systematic Risk Assessment Frequency Sponsor and CRO Findings

51-75% of trials

>75% of trials

25-50% of trials

1-24% of trials

Never

52%

21%

13%

13% 1%

Sponsors CROs

N=174 N=135

39

Use of Systematic Risk Assessment

“Please briefly describe the conditions under which a systematic risk assessment process is used.” – Verbatim Responses

●  “Not consistent”

●  “Don’t know”

●  “Often done ‘in silos’”

●  Only…

╸ During RFP process

╸ At governance level

╸ Ad hoc

╸ At program level, not study level

╸ For audit strategy

╸ Upon sponsor request

╸ For key clients

╸ When resource limitations are a big concern

●  Done, but not necessarily “systematic and rigorous”

╸ Too high level and “standard”

╸  copy and paste from previous studies

╸  no time to think about study-specific nuances

40

Risk Assessment and Risk Management

“We do not consistently approach the work from a risk assessment perspective and we continually put ourselves in the position of being surprised by sponsor questions, demands or comments.” [CRO]

41

Insights from Avoca Data

5 = Very Satisfied, 1 = Very Dissatisfied CONSORTIUM GROUP AVERAGES

Performance Aspect In-house Teams

CRO Partners

CRO Self Assessment

Proactive identification of potential risks 3.5 2.8 3.5

Proactive risk analysis and evaluation 3.3 2.7 3.4

Compilation of risk-related trial information during a trial (observations, trends, etc.) 3.2 2.6 3.2

Frequency of review of risk-related trial information 3.3 2.7 3.3

Rigor of review of risk-related trial information 3.1 2.5 3.1

Communications regarding risk-related trial information 3.3 2.8 3.2

Appropriateness of measures suggested or taken in reaction to risk-related information 3.4 3.0 3.3

Overall performance on risk assessment and management related activities 3.2 2.7 3.3

Low to middling scores suggest

knowledge gaps in key areas related to

risk identification, analysis and management

Performance Satisfaction on Select Risk Assessment Variables

42

Systematic Risk Assessment Results Sponsor and CRO Findings

More efficient use of resources for your company and/or your partner?

Increased quality?

Q: In general, have your risk assessment and management approaches resulted in…

N

204

159

203

159

9%

16%

20%

28%

39%

40%

36%

43%

17%

16%

8%

6%

35%

28%

36%

23%

Sponsors

CROs

Sponsors

CROs

Yes Sometimes No Don’t Know / Too Soon to Tell

43

Systematic Risk Assessment Results Sponsor and CRO Findings

More efficient use of resources for your company and/or your partner?

Increased quality?

Q: In general, have your risk assessment and management approaches resulted in…

N

204

159

203

159

9%

16%

20%

28%

39%

40%

36%

43%

17%

16%

8%

6%

35%

28%

36%

23%

Sponsors

CROs

Sponsors

CROs

Yes Sometimes No Don’t Know / Too Soon to Tell

44

What is your experience?

Polling question for audience: How would you rate your understanding of best practices in risk assessment and management in clinical trials? A.  Very strong understanding

B.  Good understanding

C.  Fair understanding

D.  Poor understanding

45

Understanding of Best Practices in Risk Assessment/Management

8%

39% 42%

11% Very strong understanding

Good understanding

Fair understanding

Poor understanding

How would you rate your understanding of best practices in risk assessment and management in clinical trials?

9%

49%

35%

7% Very strong understanding

Good understanding

Fair understanding

Poor understanding

N=207* N=159*

Sponsor: CRO:

*2013 Avoca Quality Consortium Assessment

46

“Team members inability to understand the big picture of what their contribution means to the broader study and what the broader study means to the overall strategy for the product is a big contributor to failed risk management.” [Sponsor]

“We don't proactively manage risk effectively. Instead we micromanage our CROs in the name of sponsor oversight, which is not a good use of anyone's time or resources.” [Sponsor]

Verbatim Comments

Panel Discussion

Thank you!

Patricia Leuchten [email protected]

Mitch Katz

[email protected]

David Marks [email protected]

Dave Nickerson

[email protected]

EXL Clinical Quality Oversight Forum

Health & Medicine

Transcript of EXL Clinical Quality Oversight Forum