The Secrets to Implementing a Successful Metrics Program...• Ensure all trial deliverables are met...

The Secrets to Implementing a

Successful Metrics Program

Ravin Warnakulasuriya

Sr. Director Vertex Pharmaceuticals

MCC Clinical Trial Risk and Performance

Management Summit

The Loews Philadelphia Sept 4-5th 2019

©2018 Vertex Pharmaceuticals Incorporated

1 ©2018 Vertex Pharmaceuticals Incorporated

Agenda

• Finding your Data

• Data Bad Habits / Data Ailments• Data Indigestion

• Data Hoarding

• Data Dilution

• Data Control• Third Party Data

• Vendor Management

• Subscriptions and Data Dilution

• Data Culture (The Human Element)• The Technology Illusion

• The Human Technology Interdependency

• Data Control (who do you trust)

• The 4 Pillars to Establishing a Data Culture

©2018 Vertex Pharmaceuticals Incorporated2

The Data Wood for the Technology Trees


• Emerging Concepts

• Risk (Hype Curve)

• Early Adopter

• Increased responsibility

(adopter vs user)

• Closed Systems

• Confined Data

• Multiple vendors

• eClinical

• Best in Breed

• Business / Financial Driver

• Coalition pros and cons

• Tactical vs Partnership

• Product Roadmap

• Bottled Water (onboarding vs new service)

• Its YOUR data, you only need the data and

you’re entitled to it

• Be System Agnostic

• Be Vendor Agnostic

• Accessibility Right

• Accountability

Back to Basics: Reclaiming your Data

The Hype Cycle


Hype Cycle


Bad Habits


1. Consume your Data like you would consume your food!


• CDW = communal

kitchen refridgerator

• Who’s is it?

• How was it prepared?

• Is it still good?

• How old is it?

• What needs to be

thrown out?

• Does it smell funny?

2. Data Hoarding is Unhealthy


Data Indigestion


• Data Noise

• Data Purity / Data Relevence

• Muddying the waters (Averages, medians)

• Inconsistency in Governance over time

• Meshing and Mapping efforts

• Orphan data

• Public data and its assumptions

• Benchmarking and Big Data Dilution

• Swimming against the legacy current

• Data evolves with the company

• Be Picky with your Data

• Standardization efforts keep us honest

• Out of the box vs configuration

• Canonicals as standard onboarding

• Circle of Life (take what you need)

• Qualify it before ingesting it

• Data Integrity Score• System

• Process

• People

• Validation

• Cleanliness is next to Godliness.

• Data ‘Value Proposition’

Data Selection

Third party Data – a Hostage Situation


• Third Party data• CROs

• Technology vendors

• Subscriptions / Data brokers

• Bells and Whistles• Canned Reports are still canned

• Report Builders

• Dashboards

• APIs and ETLs• APIs are no longer shiny

• Expect it from your Partners

• The Onboarding Advantage• RFIs and RFPs

• Using Canonicals

• It’s YOUR Data!!!

Third Party Data


• Buying Data ( Data subscriptions)

• Public data• Broad inferences (Informed decision)

• Industry data• What’s your Commitment

• What’s your return

• Definitions

• Beneath the surface• The devil is in the details

• Volume for Validity

• Study designs getting more complex

• Are we truly all operating the same?

• The deeper you go, the narrower it

gets

• Normalization and Vendor Agnosticism• Consider your vendor strategy / portfolio

• Aggregation vs Configuration

• Knowing when to Normalize your Data

• Using Canonicals across vendors

Question . . .

1. Is your Third Party Data managed / accessed differently from the

rest of your data?

2. Do you have APIs and data provision requirements as part of

your vendor selection criteria or RFI / RFP Process?

3. For those of you who have APIs set up, do you pay for this?

4. Do you mix your third party data with the rest of your data /

dashboards?


The man Element – Data Culture


Hu

The Technology Illusion



Purpose / Need

Data

Provider /

Access

Data

Interpreter

Data

Analyzer

Data

Keeper

Data

Owner

Data

Inferences

Decision

Maker

Options Action

“Even a driverless car

Still needs a passenger”

Calculations Context

Data Driven Emotional

The 4 Pillars of Establishing a Data-Driven Culture

DATA

ACCOUNTABILITY

DATA

TRANSPARENCY.

DATA

INTERPRETATION

DATA

ANALYTICS

Data

Culture

1. Accountability, Responsibility and Support

ROBUST GOVERNANCE MODEL

• Champions (Executive Support)

• Data Owners (Functional Accountability)

• Sys .Owners (Tech Perf. & Enhancements)

• Stewards (User Input, Diagnostics, Champ)

• Process Owners (Formal Adherence)

• Trainers (Awareness)

INVESTMENTS

• Talent

• Bandwidth

• Financial

EMBEDDED BUSINESS MODEL

• Integrated IT/Business Operations

• Integrated IT/Business Strategy

• Cadence and Committees (Operational /

Steering)

DATA ACCOUNTAIBILITY

2. Transparency, Trust and Co-Dependency

• Trust

• Data Sharing

• Xfunctional Dependency

• Single Source of Truth culture

• Data Integrity

• Trusting the Source / Owner

• Data Origins / Definitions / Logic

Accessibility

• Appropriateness

• Ease of Access

• Integration / Formalization

• Formal Process

• Marketing

• Adoption and Expansion

DATA TRANSPARENCY

3. Analysis, Access and Autonomy

• Robust visualizations

• Interactive user autonomy

• Interoperability (meshed reports)

• Accessibility Appropriateness

• User Friendly (internet, exports)

• Up to date (nightly feeds)

• Diagnostics (alerts and reports)

• Intended Uses• What questions are being

asked?

• Problem vs Solution oriented

• Exploratory vs Purposeful• Data Developers

• Data Scientists

DATA ANALYTICS

4. Application, Understanding and Action

• Meaningful Metrics

• Actionable outcomes

• Measurable

• Data Driven Decisions

• Reaction and Context• Risk reporting (R/A/G signals)

• Metadata vs Text

• Emotional Quotient

• (Adoption / Application)• Exposure (reluctance)

• Paranoia (delays)

• Defensive (downplay)

• Chore vs Tool (deprioritize)

• Adopter (dependent)

DATA INTERPRETATION

Question . . .

1. Does your company have a robust Data Culture?

2. Do you have a formal Data Governance Model?

3. Do you trust the data that you don’t own?

4. Are you willing to surface your Data to Executive Management?


Summary


• Find your Data

• Be Data Healthy and Hygienic

• Free your Data

• Find your sweet spot• Depth vs Breadth

• Volume vs Validity

• Configuration vs Standardization

• Data Agnosticism• Use Canonicals

• Data Accountability

• Establish a Data driven Culture

Questions?


Secrets to Implementing Successful Metrics Programs

Mike Fitzpatrick04 Sep 2019

Agenda

Poll Question

My Analytics Journey – Challenges and Recommendations

Case Study - Treehouse Brewing

Key Takeaways

Clinical trial teams excel at

leveraging data in my company.

“The ability to take data – to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it’s going to be a hugely important skill in the next decades”

Google’s Chief Economist Hal Varian – 2009 Interview

Poll Question:

Strongly Agree

5

Agree 4

Neither 3

Disagree 2

Strongly Disagree

1

Traditional Clinical Trial Manager Skills &

Qualifications

do not optimize

Clinical Trial Execution

My Hypothesis….

Responsibilities (Trial Execution)

• Day to day management of assigned clinical trial to ensure deliverables are met in all phases of trial

• Communication to internal team members on deliverables and ensuring timelines are met

• Manage trial feasibility process

• Work closely with Rare Disease or Infectious Disease Clinical Operations Strategy Leader to implement Operations plan for Clinical Study Teams.

• Drafting site materials including training, manuals and support documentation

• Monitoring of clinical data to ensure timely entry and readiness for data review meetings

• Periodic review of clinical data

• Review of clinical monitoring reports to ensure timely completion and identification of issues

• Ensure trial samples are ready to be sent to vendors in order to meet trial deliverables

• Maintain trial metrics

• Ensure trial is conducted in accordance to company and ICH/GCP standards including multiple regions in global trials

• Support the CRO in testing clinical trial systems/databases (i.e. UAT)

• Develop and maintain strong, collaborative relationships with key stakeholders within company

• Lead strategic planning and management of all study and team activities, leveraging internal and development partner resources, expertise and knowledge, along with optimizing CRO resources, expertise and knowledge.

• Lead development of overall clinical strategic operations plan for assigned projects including timelines, internal and external costs, CRO/Vendor Oversight Plans, key deliverables and risk/mitigation strategy.

• Provide oversight of study timelines and trial execution.

• Ensure all trial deliverables are met according to timeline, budget, and resource requirements

• Partner closely with Clinical Research Organization (CRO) interactions, including sponsor oversight of relevant functional activities (e.g., study management, monitoring, site management, and project master files).

• Monitor oversight of study progress including: CRO and third party vendors, ensuring compliance with relevant regulations and guidelines by maintaining regular contact with external vendors, including CRAs, as applicable.

• Work with CRO to develop and/or revise scope of service agreements, budgets, plans and detailed timelines and ensures that performance expectations are met.

• Collaborate with internal medical director in development and authorship of study protocol

• Participate in Case Report Form design, user acceptance testing in partnership with Data Management.

• Execute CRO/Third Party Oversight Activities including but not limited to: Monitoring visit report review, periodic review of data quality, review of CRO/Vendor KPIs, performing Sponsor Oversight Visits, etc.

• Contribute to and/or author clinical sections of regulatory documents including Investigators’ Brochures, safety updates, and responses to Health Authority / Ethics Committee questions

• Identify potential study issues, proactively recommending and implementing solutions or corrective actions, as needed.

• Develop and actively manage Site/Investigator relationships; serve as a liaison and resource.

• Organize, schedule and attend internal & external team trial-specific meetings.

• Perform occasional travel to CRO/vendor facilities and/or investigative sites to train and assess study team and CRAs.

• Assist Senior Clinical Trial Manager in management of clinical programs, preparation and execution of sound clinical development strategies, and Clinical Operations Department Initiatives.

• Participate in the development, review and implementation of departmental SOPs and processes.

• Other Job-related duties as assigned.

Job Description #1 (Biotech ~700 employees): Job Description #2 (Pharma ~2500 employees):

Two randomly selected CTM job postings from Linkedin, Aug 2019

• Day to day management of assigned clinical trial to ensure deliverables are met in all phases of trial

• Communication to internal team members on deliverables and ensuring timelines are met

• Manage trial feasibility process

• Work closely with Rare Disease or Infectious Disease Clinical Operations Strategy Leader to implement Operations plan for Clinical Study Teams.

• Drafting site materials including training, manuals and support documentation

• Monitoring of clinical data to ensure timely entry and readiness for data review meetings

• Periodic review of clinical data

• Review of clinical monitoring reports to ensure timely completion and identification of issues

• Ensure trial samples are ready to be sent to vendors in order to meet trial deliverables

• Maintain trial metrics

• Ensure trial is conducted in accordance to company and ICH/GCP standards including multiple regions in global trials

• Support the CRO in testing clinical trial systems/databases (i.e. UAT)

• Develop and maintain strong, collaborative relationships with key stakeholders within company

• Lead strategic planning and management of all study and team activities, leveraging internal and development partner resources, expertise and knowledge, along with optimizing CRO resources, expertise and knowledge.

• Lead development of overall clinical strategic operations plan for assigned projects including timelines, internal and external costs, CRO/Vendor Oversight Plans, key deliverables and risk/mitigation strategy.

• Provide oversight of study timelines and trial execution.

• Ensure all trial deliverables are met according to timeline, budget, and resource requirements

• Partner closely with Clinical Research Organization (CRO) interactions, including sponsor oversight of relevant functional activities (e.g., study management, monitoring, site management, and project master files).

• Monitor oversight of study progress including: CRO and third party vendors, ensuring compliance with relevant regulations and guidelines by maintaining regular contact with external vendors, including CRAs, as applicable.

• Work with CRO to develop and/or revise scope of service agreements, budgets, plans and detailed timelines and ensures that performance expectations are met.

• Collaborate with internal medical director in development and authorship of study protocol

• Participate in Case Report Form design, user acceptance testing in partnership with Data Management.

• Execute CRO/Third Party Oversight Activities including but not limited to: Monitoring visit report review, periodic review of data quality, review of CRO/Vendor KPIs, performing Sponsor Oversight Visits, etc.

• Contribute to and/or author clinical sections of regulatory documents including Investigators’ Brochures, safety updates, and responses to Health Authority / Ethics Committee questions

• Identify potential study issues, proactively recommending and implementing solutions or corrective actions, as needed.

• Develop and actively manage Site/Investigator relationships; serve as a liaison and resource.

• Organize, schedule and attend internal & external team trial-specific meetings.

• Perform occasional travel to CRO/vendor facilities and/or investigative sites to train and assess study team and CRAs.

• Assist Senior Clinical Trial Manager in management of clinical programs, preparation and execution of sound clinical development strategies, and Clinical Operations Department Initiatives.

• Participate in the development, review and implementation of departmental SOPs and processes.

• Other Job-related duties as assigned.


Legend:Tasks improved with data skill set

Skills and Qualifications

• At least 5 five years of trial coordination experience in a clinical research environment, some should be in industry

• Advanced degree preferred or Bachelor’s in a science-based subject

• Relevant therapeutic experience in global setting

• Solid understanding of drug development

• Good project management skills

• Good understanding of FDA and ICH GCP guidelines to ensure the appropriate conduct of clinical studies

• Excellent verbal and written communication skills

• Good organizational skills and attention to detail

• BS/BA degree in science or similar area.

• 2-5 years’ experience as integral Team support for clinical trials for or within the biotech or pharmaceutical industry, including CRO/vendor management, monitoring of budget scope, clinical site monitoring, protocol, ICF, CRF, CSR development and review and knowledge of basic data management/biostatistics processes including Electronic Data Collection (EDC) and Pharmacovigilance.

• Knowledge of global clinical trial management in fast paced CRO outsourced environment, including interpretation and implementation of FDA regulations/ICH guidelines, is required.

• Excellent oral and written communications.

• Ability to establish study operational plans and strategy. Requires a balance of scientific and operational/project management and team leadership expertise.

• Strong computer skills including knowledge of Microsoft Excel, Word, PowerPoint and Outlook.

• Ability to work within established timelines, in a fast paced environment.

• Excellent organizational and priority management skills.

• Good interpersonal skills and ability to deal with people at all levels with sensitivity and tact.

• Prior CNS or gene therapy experience preferred.


Legend:Data or technology related skills

Two randomly selected CTM job postings from Linkedin, Aug 2019

• Embedded within the clinical trial team

• Change management is built in – because the change is coming from embedded team

Thinking Different about Resourcing for Clinical Operations

https://www.linkedin.com/jobs/view/digital-innovator-urovant-at-roivant-sciences-866413569/

https://www.linkedin.com/jobs/view/digital-innovator-urovant-at-roivant-sciences-866413569/

My Analytics Journey

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

Studied Industrial Engineering

Joined Biogen as Clinical Operations Supply Manager

Cross trained with Clinical Trial Managers and Co-Monitors/CRAs

Clinical Supply SME for FDA, MHRA, PMDA Inspections

Transitioned to Clinical Operations Capabilities Performance team

Metrics and Analytics lead for:• CRO Oversight and

Partnership/Functional Metrics• Program and Study Reporting• Study Planning and Baseline Analytics• Site satisfaction (NPS)• Ad-hoc decision support

Joined Sarepta to build out Clinical Systems and Analytics capabilities

CLINICAL OPERATIONS DATA & ANALYTICS BUILDING

Challenges to Data and Analytics Success

DEPENDENT BLACK BOX UNCERTAIN CONFUSING SLOW

Relied on IT or vendors to produce dashboards/report Clin Ops as business

owner

Development was a “black box”

Did not understand what data are

available and what kind of reports were possible

“I’ll know it when I see it”

No one knew what they wanted –people want to

explore

Technical people did not understand clinical operations

or data visualization/

storytelling best practices

Deliverables took months to create and often needed to be re-worked

First 6-12 months

• Full of distractions

• Too much information

• Too many colors make it difficult to concentrate

• Difficult to take information in at a glance

• Visually appealing, but not distracting

• Nice color scheme – red colors to highlight important call-outs

• No unnecessary charts/data

• Filters make it easy to access necessary data

https://www.bigbookofdashboards.com/https://adniasolutions.com/dashboard-design-principles/introduction-to-dashboards/

What I was receiving… What I wanted….

My Recommendations

DATA

Understand and prepare your data assets

TOOLS & PEOPLE

Embed self-service analytics capabilities

within clinical operations organization

SO WHAT

Focus on key questions and related actions

before metric definitions

Recommendations:

• Request that your CRO or vendor provide access raw data programmatically or data transfer

• Ask what can they provide in weeks not months

• Do not be picky on the format or structure - whatever is easier for them

• Negotiate to include any missing data as second phase (or if they charge for initial transfer be more comprehensive in data ask)

Why it’s Important:

• Answer questions independently

• Drill into data for root cause investigation

• Don’t duplicate data tracking efforts with CRO

• Build new reports and dashboards

• Leverage vendors more strategically

• Rather than data cleaning, spend more time on meaningful analysis and innovation: machine learning, NLP, etc.

• Hire or develop experts in self-service data preparation and data visualization

• Be able to understand business problems and build reproducible data/technology solutions in days not weeks or months.

What is self-service?

What are the capabilities/ use

cases?

What are some tools/software?

• Teams will want lots of metrics if you just ask them what they want to use for metrics

• Much harder for teams to answer: • What is the question to answer?

• Why is it important?

• What action to take?

Metrics Methodology

Strategy Map

Example of SharePoint

Analytics Portal

‹#›

Treehouse Brewing Case Study

‹#›

Source Data: 2+ years of daily can releases: tracked diligently by Facebook group member

‹#›

‹#›

What are the key questions?

• What is the best day to go if I want the best beer variety?

• What beer is most likely to be available to buy? Which should I buy?

• How much is it likely to cost me?

‹#›

Key Takeaways

DATA

Understand and prepare your data assets

TOOLS & PEOPLE

Embed self-service analytics capabilities

within clinical operations organization

SO WHAT

Focus on key questions and related actions

before metric definitions

• Set up data transfers with CROs• Define data model + quality checks• Create a data dictionary

• Select a tool(s) (software)• Embed analysts to support teams• Promote data visualization and

storytelling best practices

• Brainstorm and strategy map• Rapid development and iteration • Create website for sharing content

Contact Information

Mike Fitzpatrick @fitzviz [email protected]

https://www.linkedin.com/in/mjf88/

Measuring Quality of Clinical

Development ProcessesNancy Dynes, Quality Metrics Consultant

Medicines Quality Organization

Intro

Disclaimer

• These slides and the content of this presentation represent the work and opinions of the

author and do not constitute official positions of Eli Lilly & Company or any other

organization.

• This document provides an outline of a presentation and is incomplete without the

accompanying oral commentary and discussion.

• Graphics, figures and data contained herein are for illustrative purposes and are not

representative of actual data.

About Me

• Experience

• Current Role

9/5/2019 ©2019 Eli Lilly and Company 54

Metrics Project Kickoff


Process & Quality Metric Evolution

• Clinical development operational metrics are

often focused on efficiency and productivity

• Processes must also be evaluated for quality

and effectiveness


CD System Structure

• The quality system is integrated into a cross-functional

“operating system” that differentiates regulatory

requirements from business requirements.

• The document management system is structured to

enable linkages to the SOPs.

• A search engine overlays process maps and

documents to help end users find content/guidance

across multiple elements of the system.


Document Repository

• Procedures and required tools are created only

for requirements as defined in the operating

system.

• All documents are linked to their step in the

corresponding process flow using metadata.

• Metadata is output to the deviation

management system.


Document Metadata


The storage system

provides descriptive fields

for each SOP. The

metadata then follows the

document into the

deviation management

system.

Systems Running

• When departures from the requirements are

found, a deviation is created.

– Variations in business processes that do not represent a

departure from quality system requirements do not drive

deviations.

• Deviations are robustly linked to the processes

and quality system documents via metadata.

• Method is in place to assess conformance to the

process/requirements.12/6/2018 ©2018 Eli Lilly and Company 60

Measuring Conformance

Problem: Deviation data are difficult to interpret

in their native form.

Solution: Use a visualization tool to build a

dashboard that summarizes deviations according

to the processes in the operating system


Occurrences Over Time

• Measure the absolute

number of deviations

per topic per quarter

• Identify which are

above a trigger value

or increasing over

time


Doc Process

Process

Area

Threshold

Met

2017

Q4

2018

Q1

2018

Q2

2018

Q3

Process 1 Medical Yes 19 6 15 14




Process 12 Medical No 1 1 4 1




• Review subtopics. If necessary, separate further by

country or disease state.

Dig Deeper


This one needs a

further look as the

deviations are

increasing

This one is

ok to monitor

over time

Process 12

Outputs of Review

• Trends identified in the deviation analysis are

reviewed by quality and process owners to

discern if there are any underlying problems.

• Process owners identify areas in need of

further investigation or change.

• All areas monitored over time by quality and

review boards.


Outcomes

• Process owners and management value the

more robust outputs

• Process improvements are identified and

tracked

• SOP metadata and deviation quality has

improved

• Focused improvements are based on

systematic reviews12/6/2018 ©2018 Eli Lilly and Company 65

Caveats

• Analyses are limited to quantities only.

• True statistical significance not possible (no

underlying population, no normal distributions,

only errors recorded).

• Deviation volume may vary depending on

business activities.


Ongoing

• Quality and process owner partnership is

needed to correctly identify where the problems

have occurred.

• Users learning how to use the data outputs

appropriately as an initial investigative tool.

• Data stewards monitoring metadata to ensure

proper linkages as processes are updated.


Summary

• Use documentation tools to link deviation findings

directly to processes.

• Create a way to monitor the deviations by process

over time to identify areas in need of improvement.

• Implement changes based on data (regulations and

deviations) instead of anecdotes.



Bonus Learnings

Design

• Start simple

• Definitions are crucial– First Patient In or First Subject Visit?

– What’s an “Active Trial”?

• Definitions need to be clear and available to end users

• No matter how carefully you design, someone will question your

data and the resulting measures

• Keep in mind: What is the question that you’re trying to answer?


Data

• Identify data sources

– Reliable

– Current

– Accessible

• Find your data experts and assign a data wrangler

• Degrees of separation between those who input data

and those who use it downstream may introduce

challenges

• Static data rarely is9/5/2019 ©2019 Eli Lilly and Company 71

Technology

• Implementation is rarely as easy as the vendor says

• Flexibility = complexity

• IT probably doesn’t know your business or your data

• Learn to speak “IT”

• No matter how sophisticated the software, you still

have to feed it good data


Thank you!

Questions?


CLINICAL TRIAL RISK AND PERFORMANCE

MANAGEMENT SUMMIT

SEPTEMBER 4-5, 2019

PHILADELPHIA, PA USA

Secrets to

Implementing

Successful Metric

Programs

Keith Dorricott

MCC Ambassador

Director, DMPI & DIGR-ACT

[email protected]

CLINICAL TRIAL RISK AND PERFORMANCE MANAGEMENT SUMMIT

SEPTEMBER 4-5, 2019 PHILADELPHIA, PA USA

Secrets to Implementing Successful Metric Programs

• Focus on the questions you are trying to answer

• Use existing standards where possible – much of the hard work

has already been done!

• Metric definitions – more than a metric title

• How MCC can help in metric selection

75



MCC Metric Development Framework

76

Define Critical

Success

Factors (CSFs)

Define Key

Performance

Question (KPQ)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Format display to highlight answer(s)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Define Key

Performance

Question (KPQ)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Develop

Process Map

©2019 Metrics Champion Consortium, All rights reserved



Define Critical

Success

Factors (CSFs)

Define Key

Performance

Question (KPQ)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Format display to highlight answer(s)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Define Key

Performance

Question (KPQ)

Define &

Measure Metric

to Answer KPQ

Generate Metric

Display that

Answers KPQ

Develop

Process Map


MCC Metric Development Framework

77

Metric Description: Percentage of initial supplies shipped by the target date

for the study

Critical Success Factor: Ready to take lab samples during the study

Key Performance Question: Are initial supplies shipped by the target date?



Critical Success Factor:

Metric #Metric

Description

Key

Performance

Question to be

answered with

metric

Why is the Key Performance Question important?

What actions might be taken based upon results?

What the metric

does not tell you

Metric TypeReporting

Level

Basic/

AdvancedSub-Process Who is being measured?

Formula / Example

Performance

Target

(suggested)

Additional Analysis for Missed

Target

Companion

Metrics (Portfolio,

Study Country,

Site)

Reporting

Frequency

Glossary Terms (See Glossary Tab) Data Elements

78




Critical Success Factor:

Metric #Metric

Description

Key

Performance

Question to be

answered with

metric

Why is the Key Performance Question important?

What actions might be taken based upon results?

What the metric

does not tell you

Metric TypeReporting

Level

Basic/

AdvancedSub-Process Who is being measured?

Formula / Example

Performance

Target

(suggested)

Additional Analysis for Missed

Target

Companion

Metrics (Portfolio,

Study Country,

Site)

Reporting

Frequency

Glossary Terms (See Glossary Tab) Data Elements

79

New to MCC Metrics

Definitions




See MCC Metric Workbook for full

metric definition

Tool A: Key Performance Questions to Metrics Mapping Tool

80





metric definition

Data elements

Tool B: Issues to Metrics Mapping Tool

81





metric definition

Tool C: Basic/Advanced Metrics to Data Elements Mapping Tool

82




Tool D: Data Elements to Metrics Mapping Tool

See MCC Metric Workbook for full metric

definition

83



Secrets to Implementing Successful Metric Programs

• Focus on the questions you are trying to answer

• Use existing standards where possible – much of the hard work

has already been done!

• Metric definitions – more than a metric title

• How MCC can help in metric selection

84


MANAGEMENT SUMMIT

SEPTEMBER 4-5, 2019

PHILADELPHIA, PA USA

Secrets to

Implementing

Successful Metric

Programs

Keith Dorricott

MCC Ambassador

Director, DMPI & DIGR-ACT

[email protected]


MANAGEMENT SUMMIT

SEPTEMBER 4-5, 2019

PHILADELPHIA, PA USA86


Secrets to Implementing Successful

Metric Programs

• Deciding what to measure

• Establishing data requirements

• Aggregating reliable data — make or buy?

• Designing metric reports

• Using metrics to understand opportunities

to improve performance

• Future opportunities?

The Secrets to Implementing a Successful Metrics Program...• Ensure all trial deliverables are met...

Documents

Transcript of The Secrets to Implementing a Successful Metrics Program...• Ensure all trial deliverables are met...