Risk Based Monitoring Presenters: Duncan Hall, CEO Triumph Research Intelligence Tammy Finnigan, COO Triumph Research Intelligence

Presentation Roadmap

1. Introductions and background

2. What is RBM really all about?

3. The value of getting it right, the risks with getting it wrong

4. The TRI approach to RBM

5. The TRI solution to RBM

6. The Future of Visual OPRA

DH

Triumph Background

Triumph Consultancy Services Formed in 2002 Purely life sciences focused consultancy Specialists in clinical systems design and implementation, business process definition, optimization and automation

Triumph Research Intelligence Formed in 2013 Aim is to develop an operational platform (Visual OPRA) designed specifically for the identification, management and reporting of site quality Provision of both platform and supporting services

DH

RBM: WHAT IS IT REALLY ABOUT? Triumph Research Intelligence

TF

RBM – What Is It All About?

R B M Risk • A Forecast • Signal • Quantifiable

Monitoring • Site based • Remote • Central • Statistical

Informs

TF

RBM – What Are We Assessing?

Reporting diligence

o What is the likelihood that an important event will be reported?

Data quality

o A measure of the variability of the data

o A measure of probability that the data is an accurate reflection of the real world

Protocol compliance

o The sites ability to comply with the protocol

o Direct or indirect

BEHAVIOUR An assessment of the sort of behaviour which is likely to result in risks to quality in one or more of the following categories:

TF

RBM: VALUE OF GETTING IT RIGHT Triumph Research Intelligence

DH

What Does Getting It Right Look Like?

SIGNAL AND NOISE Getting it right means that we have isolated the signal from the noise. We have identified those sites which exhibit behaviours likely to have a negative impact on quality.

Operationally o We know which sites and even which patients to focus our monitoring effort on

o We know what behavioural aspects are impacting quality

o We know what corrective actions to take

Statistically o We have an accurate measure of the site quality risk

o We are able to rank sites in order of risk to ensure effective prioritisation of action

Regulatory o We have a record of our assessments of risk

o We have a record of the actions we took

o We have evidence of the efficacy of our actions DH

What Does Getting It Wrong Look Like?

NOISE, LOTS OF IT We have either failed to identify the true signal, and are simply creating noise, or we create so much noise the signal is masked.

Operationally o We spend a lot of time looking at data – some of it very pretty!

o We confuse performance with quality and start targeting low or high performers

o We give unclear messages to sites and see little or no sign of quality improvement

Statistically o We look for outliers using inappropriate statistical methods (%, deviations, limits)

o We assess subjective or cleaned data sources

Regulatory o Inadequate monitoring and oversight

o Higher chance of inspection findings

DH

RISK DETECTION & TECHNOLOGY Triumph Research Intelligence

DH

Solution Process

Knowledge based used to inform study design and

select M/RIs

Confirmation of availability and quality of

data in warehouse Study specific ETL built

Data consumed by Quality Risk Engine and

optimised for visualisation

Visualisations generated for each QRI, knowledge

based used to spot patterns / behaviours

Sites ranked in terms of risk and suggested

actions for breached thresholds defined

through medical review and knowledge base

CRAs / CDMG informed of observations and required actions

Intervention made with site, and efficacy of

intervention measured over agreed number of

assessment periods

Knowledge based and quality oversight records

updated

DH

Quality Risk Indicators

Core set of Quality risk visualisations defined

Algorithms designed to represent QRIs on funnel plots

Each indicator allows data review as well as visualisation (important for assessment of data variability)

Dynamic axis and funnel plot materialisation

Data series show / hide functionality

QRIs are not standalone, but related and will give rise to specific patterns of data which we call ‘risk signatures’.

Risk signatures allow a more specific, targeted response, allowing a more valuable site / CRA interaction.

QUALITY RISK INDICATORS WHEN COMBINED ALLOW CREATION OF RISK SIGNATURES

Quality risk indicators should: • Indicate the probability that a site is a quality risk, NOT

site / CRA performance • Be based on objective data, not subjective (CRA /

operational data) • Be two dimensional to take account of data volume (error

rate is inversely proportional to volume) • Be specific and sensitive – separate signal from noise • Be easy to interpret

DH

Data Volume – Dynamic Thresholds

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 2 4 6 8 10 12 14 16 18 20

Failu

re /

10

su

bje

cts

# subjects

Confidence increases with volume

Combining multiple risk indicators per site to create an aggregate risk score allows earlier identification of risk, with a lower subject / visit count

Normalizing data by volume

DH

Risk Profiling – Medical History

Hovering over a site will show a pop up with the site

identifier and data specific to that site for easy risk

identification

DH

Risk Profiling – BP Number Preference

In the case of BP readings, we are looking

for a low number preference score. A high score can indicate fraud

or bias. Either is a key indication of risk

DH

Risk Profiling – AE Reporting

Both under and over performing sites can be an indications of poor

quality. These are shown by the green and red

circled sites

DH

Activity Efficacy with Visual OPRA

Visual OPRA will allow different assessment periods to allow historic comparison with current state. This

view build evidence of quality improvement or shows that

different actions need to be taken

Clear indication of reduction of site quality

risk

Adverse Event Reporting Rates – Assessment period 2

Adverse Event Reporting Rates – Assessment period 4

DH

DH

Turning Insight Into Activity

RBM: QUALITY RISK DETECTION PROCESS Triumph Research Intelligence

TF

Quality by Design

o RBM is only a piece of the puzzle

o Protocol design is critical to the success of any study

o “quality cannot be monitored into a study”

o Use risk profiles and root causes to inform:

− Protocol design process

− Site, patient and study team training

− Country specific (potentially site specific) monitoring plans

TF

Business Process

TF

RBM with Visual OPRA

Monitoring Plan

• Risk indicators (RIs) are used as a means of measuring site risk

• SDR is directed based on site risk profile

Operational Activities

• Number of monitoring visits is reduced as less planned visits

• As RIs identify risk, likely root cause is predicted and suggested course of action

• Central monitors review subject data based on defined rules, to direct CRA to the patients that will reveal the problems

• CRA confirms root cause and implements suggested course of action

Operational Impact

• SDV is reduced significantly, and SDR is performed on targeted patients

• Site risk is reduced or decision is quickly taken to exclude site

• Data quality is improved as decisions can be taken quickly and are less dependent on CRA performance i.e. finding the issues and taking the correct course of action

• Tracking root cause and actions linked to changes in site risk profile provides evidence of effectiveness and oversight

TF

RBM: THE FUTURE OF VISUAL OPRA Triumph Research Intelligence

TF

Current state

Pilot

o We are currently piloting VisualOPRA with a mid-sized CRO on 2 studies

o Study A

− CV device study

− Risk analyses on 4 cuts of historic data

− Comparing the results from the risk profiles with the issues discovered during the statistical

analysis at the end of the study

− Determining if the issues discovered at the end of the study during the analysis phase could

have been identified earlier, and even prevented with early intervention

o Study B

− ‘Live’ CV study starting in Q2

− Using VisualOPRA to inform the monitoring visits

− Starting to relate risk profiles to root causes and corrective monitoring actions

o Case studies will be published in Q2 (Study A) and Q4 (Study B)

TF

Quality Risk Identification – Ongoing Development

Further development of 4-6 QRIs planned Focus will be in treatment emergent QRIs

Con meds Physical exams Visit window deviations

QRI library by TA and geographic location will allow appropriate application on per study basis

Risk profiles linked to likely root causes and effective actions

QUALITY RISK INDICATORS DEVELOPMENT

Turning Quality metrics into 2 dimensional metrics as QRIs Missing data

Query resolution

Data entry timeliness

Screen failures

Early termination rates

QUALITY METRICS

0

0.1

0.2

0.3

0.4

0.5

0.6

0 5 10 15 20

Failu

re /

10

su

bje

cts

# subjects

0

2

4

6

8

Site 1 Site 2 Site 3 Site 4 Site 5 Site x

Screen failures

Threshold or %

Traditional BI tool view TF

Next steps

Demo

o If you would like to see VisualOPRA, contact us for a live demo, approx. 1hr

Data Analysis

o You provide the data for 1 study, we will use VisualOPRA to identify high risk sites and

compare to your study analyses

2 day workshop

o You provide the venue and we will facilitate a 2 day workshop on getting up to speed

with RBM

o The workshop will include:

− Comparison of the EMA and FDA guidance

− Review of current RBM strategy (if available)

− Examination of quality risk vs. performance measures

− Identifying what is important to your organization

− Process change required to turn RBM into reality

TF

Summary

What IS RBM About? o Proactive detection of behaviour likely to lead to poor quality

o Measurable level of risk to prioritise activity

Getting It Right o Clear signal detected through the noise

o Probabilistic analysis if signal to rank significance

o Documented, repeatable evidence for regulatory authority

Getting It Wrong o Lots of noise, pretty noise!

TRI Approach o Funnel plots to normalise data and allow earlier detection

o Clear operational direction, managed through platform

Visual OPRA technology o For purpose RBM platform, designed for study teams

o Proprietary Risk Engine and algorithms

o Dynamic reporting, review and analysis

Any Questions?

Presenters biographies and contact details

Tammy Finnigan, COO, Triumph Consultancy Services

tammy.finnigan@triumphconsultancy.co.uk

Tammy’s entire career has been focused on clinical research, having worked in project management and clinical operations for 10 years, with both large Pharma and CRO businesses prior to joining Triumph. Her experience both in monitoring, and managing clinical trials made her a significant hire for Triumph in 2007. Tammy’s experience, passion and eye for quality saw her promoted to Head of EU Operations within her first year, and in 2011 she was appointed COO to take over global operations responsibility.

Duncan Hall, Founder and CEO

Duncan.hall@triumphconsultancy.co.uk

Duncan has over fifteen years of consultancy experience, thirteen of which have been within clinical R&D in both CRO and Pharma businesses. Duncan started Triumph in 2002 with the aim of building a global business which would be focused in improving the delivery, quality and value of clinical systems. Duncan now takes a primarily strategic role in Triumph, but still performs client delivery roles where possible

TRIUMPH RESEARCH INTELLIGENCE IN THE US 7100 Six Forks Road Suite 135 Raleigh NC, 27615 USA

Tel: +1 919 239 4989 Fax: +1 919 239 4989

Email: info@tritrials.com Web: www.triumphconsultancy.com

TRIUMPH RESEARCH INTELLIGENCE IN THE UK Unit 6 Riverside Business Park Stoney Common Road Stansted Mountfitchet Essex, CM24 8PL UK

Tel: +44 1279 812 550 Fax: +44 1279 816 884

Email: info@tritrials.com Web: www.triumphconsultancy.com