Measuring the Impact of Risk-Based Monitoring · 2020-03-26 · Risk Based Monitoring continues to...

Tuesday 24March2020

9:00am-10:30 am (Eastern)

Measuring the Impact of Risk-Based Monitoring

Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 2

2

Julie Day

• TransCelerate BioPharma Inc.

• Program Director

Jenni Sullivan

TransCelerate BioPharma Inc.

Program Manager

Gloria McHugh

TransCelerate BioPharma Inc.

Project Manager

TransCelerate Team

Please report feedback or technical issues to: [email protected]

mailto:[email protected]


Expert Presenters

3

Samantha Hewlett

• Risk-Based Study Execution Capability Lead, Amgen

• RBM Team Member

Rodrigo Perez Marentes

Clinical Studies Unit Risk-Based Monitoring Business Lead, Sanofi

RBM Team Member

Mary Arnould

Director, Clinical Science Management Operations & RBM Lead, Astellas

RBM Team Member


Logistics for this Webinar

4

All participants will be muted for this call.

For audio: Connect to audio to listen to presentations via your computer or phone

You can reduce control panel for a better view of

presentation by clicking the orange and red arrow

To submit a question to the presenters:Type your question in the Questions panel and click Send.

NOTE: this webinar may be recorded in whole or in part.

If you do not consent to be recorded, please send a note

using the Questions panel.


Logistics for this Live Polling

5

To help make this session interactive and facilitate sharing of useful and

appropriate information, we will conduct live polls throughout the session.

Your participation is completely voluntary and all responses will be

anonymous. Answers will be shared only as a percentage of respondents.

Instructions: When the

blue poll question

appears, enter your

response and submit.


Ground Rules

6

We want to make this discussion helpful and answer as many of your questions as we can, so some quick ground rules:

• Participation is voluntary, as is using TransCelerate assets / tools

– The responsibility for compliance with laws and regulations is owned by solution adopter.

• You don’t have to identify what company you work for

• Things we would ask you not to discuss:

– What vendors / sites / CROs you are using or not using

– Any issues you have with any vendors / sites / CROs

– Your long-term development plans– Anything related to costs

• We can’t answer questions about:– Vendors– Costs of using / implementing TransCelerate assets / tools – Which member companies are using the assets / tools

Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 7Copyright ©2017 TransCelerate BioPharma Inc., All rights reserved. * Confidential - NOT FOR DISTRIBUTION *

AGENDA

1) Overview of TransCelerate [Julie Day, TransCelerate ] (5 mins)

2) RBM Metrics: Measuring the Impact of Risk Based Monitoring [Sam Hewlett, Amgen] (15 mins)

3) Member Company Perspectives with RBM Metrics (30 mins)

▪ Challenges in Implementing RBM Metrics-Experience from 2 Member Companies [Mary Arnould, Astellas]

▪ How Amgen Has Evolved Their Metrics [Sam Hewlett, Amgen]

▪ What’s Next for Risk Based Monitoring [Rodrigo Perez, Sanofi]

4) Q&A (focused on today’s webinar metrics topics) [Sam Hewlett, Amgen] (40 mins)

7

1. Overview of

TransCelerate

Julie Day•TransCelerate BioPharma Inc.

•Program Director


Our Shared Vision:

To improve the health of people around

the world by accelerating and simplifying

the research and development of

innovative new therapies.

TransCelerate:

A Not-for-Profit Entity Created to Foster Collaboration

Copyright ©2020 TransCelerate BioPharma Inc., All rights reserved.

2 1 MEMBERCOMPANIES

Current state of organization

Today

25+including 5

pharmacovigilance initiatives

INITIATIVES

focus on preclinical research

ENHANCING INDUSTRY COLLABORATION

With an effective and proven governance

structure have increased the ease and desire to

collaborate

FACILITATING FUTURE PLATFORM

TRIALS

12+ initiatives deliver solutions that facilitate future platform

trials

BREADTH & DEPTH

Over 60 solutions being delivered across 25+ initiatives, across 3 strategic

priorities

TransCelerate Founded

2012

5 INITIAL INITIATIVES

10 MEMBERCOMPANIES

10

platform to enable data sharing

BioCelerate Founded

2016

Bayer most recent

member

Copyright ©2020 TransCelerate BioPharma Inc., All rights reserved.

The Reach of our Global Membership is Expanding

11

Membership is available to biopharmaceutical research and development organizations

that engage in innovative discovery, development and manufacturing of new medicines*.

* to be eligible for membership, companies must meet

specified eligibility criteria.

Thousands of people have contributed to the design,

development and deployment of TransCelerate solutions

2. RBM Metrics: Measuring the

Impact of Risk Based Monitoring

Samantha Hewlett

Risk-Based Study Execution Capability Lead, Amgen

RBM Team Member


Timeline of Risk Based Monitoring Initiative

PAST PRESENT FUTURE

2012-2017

Active Phase: Deliver & Measure

• 8 Methodology Framework

Papers

• Multiple Supporting Tools and

Resources for Industry Use

• TransCelerate collects, blinds and

aggregates RBM metrics data

from member companies.

Implementation increases most

significantly from 2015-2016.

2018-2019

Mature Phase: Facilitate Adoption

2012TransCelerate forms. One of its first initiatives is Risk Based

Monitoring, in response to FDA guidance.

• Focus on Facilitating Awareness, Adoption and Best Practice Sharing

• Ongoing High Visibility Engagements across Industry (Healthcare Authorities, Conferences, Site Advocacy Groups, Contract Research Organizations)

• Perception is that RBM is improving quality and efficiency, however quantifying the magnitude of improvement remains challenging

2019TransCelerate Metrics Survey. 18 member

companies are invited to participate in an anonymous survey to better understand

how the value of RBM implementation is defined and assessed.

2020 on …

Risk Based Monitoring continues to

evolve. Multiple factors including

ICH E6, E8 revisions, evolving

quality management systems, and

technology advances have the

potential to impact Risk Based

Monitoring

2020 and beyond…

Evolution


Collecting RBM Data Quarterly | Methods

• Per FDA suggestion, data was collected from participating

member companies quarterly from 2013-2017, resulting in

>1000 observations.

• Guided by a defined set of RBM metrics, member

companies voluntarily reported data to TransCelerate on

clinical trials where they were implementing RBM.

• Member companies were asked to rate change over time

for each metric as “better,” “worse” or “about the same”

(as compared to each company’s internal baseline

expectation).

• Data was blinded, aggregated and reported by

TransCelerate each quarter.

METRICS

Average number of major/critical audit findings

Percentage of unreported, confirmed SAEs as

compared to total SAEs as discovered through

any method

Number of Significant Protocol Deviations

Average Monitoring (all types) cost per site

Average interval between on-site monitoring visits

per site

Median number of days from issue open to close

Median number of days from patient visit to eCRF

data entry

Median number of days from query open to close

In 2018 & 2019, we have further aggregated the data to analyze risk based monitoring trends and produce cumulative observations.


Cumulative Data Observations (Relative %)

• Audit Findings: Quality appears to increase

in more mature studies.

• SAE Reporting: Early on in studies, there

appears to be an initial rise in SAE

reporting, but as the study progresses, SAE

reporting appears to improve, overall.

• Significant Protocol Deviations:

Compliance appears to improve during

the study; however, data becomes too

limited in later maturities to prove this.

• Overall Monitoring Cost & On-Site Visit

Interval: Cost and On-site Visit Intervals are

highly correlated.

• Issue Open to Close, eCRF Data Entry,

Query Open to Close: Those implementing

took care to focus on data flow,

recognizing it is critical to success.

• All 8 metrics showed at least 50% “better”

• >70% were “about the same” or “better”


Understanding Current State | What do RBM metrics look like today?

2019

RBM Metrics Survey

of Member Companies

(n=18)

Objectives:

(1) analyze current use of the original metrics

(2) assess how benefits of RBM are currently measured

(3) determine whether new metrics have been developed by members to further define the value of RBM

Results:

(1) Interpretation of experiences of member companies

(2) Not “one-size-fits-all”


Understanding Current State2019 Member Company Survey Results

Key Decision Drivers Behind RBM Adoption

Both Quality/Compliance and

Efficiency/Cost are primary

drivers for implementing an

RBM model.

1 Participants could select only 1 response



Types of Trials in Scope of RBM

❖ Implementation of RBM is now seen across

all phases of Interventional Clinical Trials.

❖ 14 companies report utilizing RBM in all

phases of their studies currently. The others

have taken various, staggered

approaches.

❖ Member companies approaches to

implementation have evolved over time

to expand to current state.

14

21 1

0

2

4

6

8

10

12

14

16

All Clinical Trials Phase I/II/III

Only

Phase II/III

Only

None

(not

implemented

RBM)

# of Member Companies



Components Implemented with RBM

• Risk assessments, reduced Site Data

Verification (SDV) and a more remote

approach to analysis and monitoring are

the major components consistently seen

across companies who have adopted a

RBM model.

• The original focus areas of the

TransCelerate RBM model (e.g., site

focused) appear to be well-established.

• The risk-based approach appears to be

spreading to other, related areas (e.g. data

cleaning).

2 Participants could select more than one response



How long did it take to see/measure the impact(s) of RBM

implementation?

1 Participants could select only 1 response

➢ Whilst it appears that RBM contributed

positively to the development process

within 1-3 years of implementation,

few companies are tangibly

measuring its direct impact through

metric analysis, with many are still

developing their metrics and

collecting data.

➢ While quality is a key decision driver

for implementation, organizations

struggle to measure quality directly

and note significant change

management efforts are necessary to

support the process.


Are the original metrics being used to

evaluate the implementation of RBM?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Use of RBM Metrics

Currently Use Previously Used Never Used

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Usefullness of RBM Metrics

Extremely Useful Somewhat Useful Neutral Not Very Useful

Are the original metrics

considered useful?

Audience Poll

Question: Has your company implemented

metrics to assess the value of RBM? (select one)

❑ Yes

❑ No

❑ I’m not sure

Audience Poll

Question: How helpful have your company metrics

been in measuring the value of RBM? (select one)

❑ Very helpful

❑ Somewhat helpful

❑ Not very helpful

❑ Not at all helpful

❑ Neutral, a mix, or un-determined at this time

3. Member Company

Perspectives with RBM Metrics

Challenges in Implementing

RBM Metrics-Experience from 2

Member Companies

Mary Arnould

Director, Clinical Science Management Operations & RBM Lead, Astellas

RBM Team Member


TransCelerate Metrics

Metric Category


Quality

Percentage of unreported, confirmed SAEs as

compared to total SAEs as discovered through any

method



Cycle TimeMedian number of days from patient visit to eCRF data

entry


Average interval between on-site monitoring visits per

site Cost



A Tale of Two Companies Implementation of RBM Metrics

Company A

• Large pharma

• Insourced central and site monitoring

• Insourced site monitoring

• Primary driver for implementation was quality

• Implementation was “all in” from outset

• RBM Metrics developed from the outset and tracked quarterly

Company B

• Medium pharma

• Insourced central monitoring

• Outsourced site monitoring

• Primary driver for implementation was quality

• Implementation began with pilots and progressed to full implementation

• RBM Metrics not initiated until full implementation was well underway

Company Characteristics:

Both companies sought to develop all 8 TransCelerate metrics and

developed a non-RBM comparison for each metric


The “Easy” Metrics

Median number of days from patient visit to eCRF data entry



The “Challenging” Metrics


• Are all studies audited?

• Have audit plans changed over time?

• Are audits consistently performed across phases and TAs?


• Have categorizations of PDs changed over time?

• Data source may present challenges if PDs collected by vendor partners

• Study specific PD identification or management may impact overall metric


• Data sources may present challenges if collected by vendor partners


The “Challenging” Metrics

Average interval between on-site monitoring visits per site

• Relies on CTMS data for monitoring visit dates

• Has direct impact on Monitoring Cost

• Helpful to visualize utilization of on-site and remote visits if using adaptive approach


• Actual costs may not reflect potential cost savings because of difficult to control variables (e.g. Portfolio characteristics, vendor relationships, Increasing cost over time)

• Easier to estimate when all monitoring activities are performed by sponsor


The “Difficult” Metric

Percentage of unreported, confirmed SAEs as compared to total SAEs as discovered through any method

• Difficulty exists in measuring the unknown

• Relies on effective SDR-what is impact when SDR is performed on less than 100% of data?

• Two examples:

• Responses to question on monitoring visit report related to sufficient SAE reporting

• Measuring SAE reporting time (Site awareness date to reporting date) as a surrogate indicator


Conclusion

Common challenges:

• Access to the data

• Establishing and maintaining the non-RBM baseline

• Number of non-RBM studies will decrease over time as number of RBM studies increases

• Visualization tool-Tableau, Excel, others?

• Are these the metrics that best reflect the value of RBM to the organization?

Evolution of Amgen

Metrics Over Time

Samantha Hewlett


RBM Team Member


Evolution of Metrics Over Time

I’m looking in generally

the right area

Precise detection



▪ Amgen implemented first version of RBM in 2012

▪ Primarily focussed on reduced SDR/SDV with enhanced data review

▪ Supported by many of the metrics already discussed – data entry, query resolution,

protocol deviations and SDV backlog

▪ Metrics were reviewed bi-weekly by CRAs and drove actions according to pre-defined

guidance

Challenge

▪ Not reproduceable

▪ No ability to trend data

▪ Not flexible

▪ Very granular in nature



Centralized Statistical

Monitoring (CSM) detects

sites with unusual clinical &

operational data and flags

potential issues for GCTMs to

investigate

Central Monitors review

site operational Key Risk

Indicators (KRIs) and

recommend actions to CRAs

Central Monitoring Data Monitoring

Enhanced Data Review Levels of data review, including

ongoing, real-time data review

by Medical Data Reviewers

Site Monitoring

Flexible Monitoring

Schemas based on risk &

site performance

Risk-Based Source

Data Verification and

increased focus on Source

Data Review

Risk ManagementCT-RACT used to identify and manage risks & critical processes

RBM


Evolution of Metrics Over Time - Conclusion

▪ Internal and external factors WILL continue to change

▪ Sophistication in data analysis is evolving at a rapid rate

▪ New data sources continue to be introduced

▪ We can not afford to be complacent and must continue to ensure that

even with the “need for speed” data integrity is maintained

WHY?


Evolution of Metrics Over Time - Conclusion

▪ Internal and external factors WILL continue to change

▪ Sophistication in data analysis is evolving at a rapid rate

▪ New data sources continue to be introduced

▪ We can not afford to be complacent and must continue to ensure that

even with the “need for speed” data integrity is maintained

WHY – because patients matter

4. Future of RBM Metrics

Rodrigo Perez Marentes

Clinical Studies Unit Risk-Based Monitoring Business Lead, Sanofi

RBM Team Member


Current Challenges

<50% of MC use historical controls/baseline data to demonstrate the value of RBM implementation

Measuring RBM value is not necessarily one-size-fits-all. Sponsors prefer fit for purpose methods.

Multiple changes are introduced in trial conduct in parallel – Masks true impact of RBM on each trial.

Limitations to consistently define, measure and benchmark quality in a scalable manner

Challenges to quantifying the value of RBM and other quality-focused initiatives.

The existing measurement processes prevent us from quantifying the benefits of RBM accurately.


How Are RBM Metrics Evolving?

Proxy Metrics

The original RBM metrics were proxy metrics that measured for quality but did not always directly measure the impact attributable to RBM

Original Metrics

New Metrics

Several MCs have defined new/novel Key Performance Indicators

(KPIs) to measure the value of RBM.

KPIs

Tailor Made Metrics

Best practices appear to include defining measures tailor-made to the chosen RBM model and the individual organization’s operational design.

Other quality

initiatives

The art and

science of

measuring &

demonstrating the benefits of

RBM are still

evolving


What’s Next For Risk Based Monitoring Metrics?

QTLs

Regulatory Inspections

ICH E6, E8 Revisions

Technology

Guidance needed

Insight & experience needed

Continued evolution

Continued industry communications to share

learningsIncreasing levels of data intelligence & capability

for feedback loops

2020 on… RBM Continues to evolve.

Multiple factors including ICH E6, E8

revisions, evolving quality management

systemes, and technology advances

have the potential to impact RBM.


These Metric Narratives illustrate the expectationsof RBM’s impact as well as alternative observations

RANK PROCESS COMPLIANCE DATA

Extremely Useful

• Time from data “cut” to Action3

• Ratio of on-site to off-site monitoring visits

• SDR & SDV backlog4

• % queries resolved in 7 days3

• % pages submitted in 7 days3

• Important Protocol Deviation

Incidence3

• Missed Assessments2

• Dosing Deviation

Incidence3

Somewhat Useful

• Reports for centralized monitoring – user statistics to indicate

frequency and duration of use3

• Ratio of Data correction XX days after initial data entry3

• RBM user satisfaction survey3

• Query rates2

• Qualitative interviews with HQ trial teams2

• Survey for use and usefulness of site risk indicator report2

• TMF Compliance2 • SAE/AE rates2

• Query rate (per 1000

data points)3

Neutral• Action item aging3

• External data review status2

• CAPAs close on time (site)3

• CAPA # overall3

• eTMF status2

• Ratio of number of AE emerging

per subject4

• Query aging3

• Ratio of number of AE

emerging per subject4

Not very Useful• On-site vs remote visit ratio: Ratio calculated as number on-

site to remote visits4

• Ratio of Missing data for the

primary endpoint4

• Ratio of Missing data for

the primary endpoint4

KEY PERFORMANCE INDICATORS (KPIs)

Examples of new KPIs for RBM developed by Member Companies. The 9 MCs that responded provided 28 new/novel KPIs and ranked them by usefulness.

1 These are the direct answers from survey respondents, we have not summarized, word-smithed, or tried to categorize them. Some information may be contradictory since it was submitted by different companies. 2 Qualitative Indicator 3 Quantitative Indicator 4 Duplicate


What’s next for Sanofi?

Sanofi’s Perspective – Future of metrics

➢ Continuous improvement mode - Performance Metrics to be fine tuned and

re-defined.

➢ Set new target of monitoring visit adaptive approach - Shift from “one size

fits all” to “fit for purpose”.

➢ Process simplification for metrics calculation methodology – to be tackled

by department.

➢ Leverage Technology: Automation of performance indicators measurement

– still too manual and not consistent.

➢ Implementation of QTLs – Participation in TransCelerate QTLs workstream.

➢ From a targeted RBM into a more holistic approach of global Risk Based

Quality Management.

Audience Poll

Question: What do you think is THE MOST

IMPORTANT issue facing RBM implementers today? (select one)

❑ Continued technological advancements to support RBM

❑ Inspection Findings feedback from marketing applications

❑ More guidance from Regulatory Authorities

❑ QTLs

❑ Other

Q & A or Panel DiscussionType your questions into the Questions panel on your

GoToWebinar screen, click “Submit”

Please note that Q & A session are not recorded.

Please stay on the line to participate in the live Q & A.

Samantha Hewlett


RBM Team Member

Please focus your

questions on today’s

Metrics topic

Audience Poll

How much value did this Webinar provide?

No ValueA Little Value

Moderate Value

A Lot of Value

Visit us, for more information:www.TransCelerateBioPharmaInc.com

Watch our “About Us” Video

Sign up for our Newsletter, Accelerate to Innovate

@TransCelerat

e

TransCelerate

BioPharma Inc.

Thank you

http://www.transceleratebiopharmainc.com/

https://www.youtube.com/watch?v=cn8xZqTq_Ug

Measuring the Impact of Risk-Based Monitoring · 2020-03-26 · Risk Based Monitoring continues to...

Documents

Transcript of Measuring the Impact of Risk-Based Monitoring · 2020-03-26 · Risk Based Monitoring continues to...