Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD...

1

Presenter: Paule Poulin, PhDAuthors: P. Poulin, L. Austen et al.,

Health Technology & InnovationDepartment of Surgery

Calgary Research Portfolio

Multiple CriteriaHTA Into Practice

Local HTA Program

CADTH Symposium April 3-5, 2011Vancouver, BC

InnovationCREATE

OR DIE!

2

Objectives:

1. Local HTA Program Overview

2. Multiple Criteria for Technology Evaluation

3. Lessons Learned

3

Objectives:



3. Lessons Learned

4

Local decision-makers need to consider

• Local priority-setting• Local operations• Local population health needs• Local alternatives• Local presence of trained personnel• Local infrastructure impact• Local budget impact……..• As well as HTAs by HTA agencies

HTA into Practice“Local HTA Decision-Support Program”

5

SIGNIFICANT CHANGEUncertainties about clinical safety, effectiveness

and/or cost or resources impact

Local HTA Committeeevaluation

MINOR CHANGENo significant clinical, contractual,

cost or resource issues

Referredfor further assessment

Forms to gather information

+ +

Clinical, Financial, or Economic Impact

Forms E-G

TechnologyRequest

Form A

Support of Technology Request

Form B

Contract-Costing Check

Form C

Local HTACheck

Form D

Approvedfor purchase/implementation

6




Tools for Decision Making

RecommendationMade by Local HTA Committee

Evaluation Criteria

Tool 2

Evaluation Worksheet

Tool 3Decision

Made by Executive Committee

MINOR CHANGENo significant clinical, contractual, cost or

resource issues

ApprovedFor purchase/implementation

Screening Guide

Tool 1

Recommendation and Decision Guide

Tool 4

Local HTA Committee evaluation

Prioritization Tool

Tool 5

7

Objectives:



3. Lessons Learned

8

Criteria Development (supported by CADTH grant)

Criteria list from literature

Criteria revision by selected group

Group retreat (criteria revision)

Member survey (criteria ranking)

Final List of Criteria

Revised

Forms

Revised

Tools

Criteria cross-checked withinformation-gathering Forms

Criteria used to formalizeTechnology Evaluation and Decision

Tools

Mod

ified

Del

phi P

roce

ss

9

Tool 2: Evaluation Criteria

Domain (5)

Health Gain

Service Delivery

Strategic Fit

Innovation

Financial

10

Domain (5)

Health Gain

Service Delivery

Strategic Fit

Innovation

Financial


11

Domain (5)

Health Gain

Service Delivery

Strategic Fit

Innovation

Financial


12

Domain (5)

Health Gain

Service Delivery

Strategic Fit

Innovation

Financial


“CREATE” OR

“DIE”

13

Domain (5)

Health Gain

Service Delivery

Strategic Fit

Innovation

Financial


14

Domain (5) Criteria (12) Example Clarifying Questions

Health Gain Efficacy (patient) • evidence for health gain?

Population Health • addresses burden of disease?

Standard of Care • new Standard of Care?

Service Delivery Safety • as safe as current practice?

Training • requires training or credentialing?

Access • improves access to the under-served?

Service Coordination • impact on other health services?

Strategic Fit Strategic Fit • aligned with strategic goals?

Innovation Knowledge & Research • knowledge generation or transfer?

Financial Cost • start-up and on-going costs?

Sustainability • can we afford it ?

Economic Analysis • cost-effective?


15




Tools 2 & 3: Evaluation Criteria and Worksheet


Evaluation Criteria

Tool 2


Tool 3Decision



resource issues


Screening Guide

Tool 1


16

For each criterion:

A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA

B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5

C. Make a RECOMMENDATION

Tool 3: Evaluation Worksheet

17

Tool 3A: “ Device X” – Quality and Completeness of Information

Domain Criteria Example Clarifying Questions SCORE

Health Gain Efficacy (patient) • Evidence for efficacy? Indications for use? Inadequate

Population Health • Adequate projections of prevalence? Adequate

Standard of Care • Potential for new standard of care? Inadequate

Service Delivery Safety • Complications and risks addressed? Inadequate

Training • Training implications described? Adequate

Access • Improves access? NA

Service Coordination • Impact on other health services described? Adequate

Strategic Fit Strategic Fit • Aligned with strategic goals? Adequate

Innovation Knowledge & Research

• Innovative characteristics described?• Outcome measures described?

Inadequate

Financial Cost • Information on resources and impact complete? Adequate

Sustainability

Economic Analysis • Evidence to support cost-benefits? Adequate

18

For each criterion:

A. Score the QUALITY and COMPLETENESS of the information provided by the applicant: Adequate, Inadequate, NA

B. Score the SIGNIFICANCE and IMPACT of the technology on a scale of 0-5

C. Make a RECOMMENDATION

Tool 3: Evaluation Worksheet

19

Domain (5) Criteria (12) Example Clarifying Questions

Health Gain Efficacy (patient) • evidence for health gain?

Population Health • addresses burden of disease?

Standard of Care • new Standard of Care?

Service Delivery Safety • as safe as current practice?

Training • requires training or credentialing?

Access • improves access to the under-served?

Service Coordination • impact on other health services?

Strategic Fit Strategic Fit • aligned with strategic goals?

Innovation Knowledge & Research • knowledge generation or transfer?

Financial Cost • start-up and on-going costs?

Sustainability • can we afford it ?

Economic Analysis • cost-effective?


20

Tool 2: Criteria - Scoring Significance and Impact

CRITERIA 0 POINTS 1 POINT 3 POINTS 5 POINTS

Efficacy No improvement in patient health gain compared with current practices

Minimal improvement

Moderate improvement

Vast improvement

Population Health

Addresses a condition with very low prevalence (rate/100,000 < 1)

Low prevalence (rate/100,000 is 1-10)

Moderate prevalence (rate/100,000 is 10-1000)

High prevalence (rate/100,000 is 1,000-10,000)

Standard of Care

Does not represent Standard of Care in Alberta

Represents Standard of Care in some health regions in Alberta

Represents Standard of Care in most regions in Alberta

Represents a new Standard of Care in our region or Alberta

21

Tool 3B: “ Device X” – Scoring Significance and Impact

POTENTIAL IMPROVEMENT

Domain Criteria None: 0 Minimal: 1 Moderate: 3 Vast: 5

Health Gain Efficacy (patient) X

Population Health X

Standard of Care X

Service Delivery Safety X

Training (vast to minimal) X

Access X

Service Coordination X

Strategic Fit Strategic Fit X

Innovation Knowledge & Research X

Outcome Measures (quality) X

Financial Cost (viability) X

Sustainability

Economic Analysis X

22




Tool 4: Recommendation and Decision Guide


Evaluation Criteria

Tool 2


Tool 3Decision



resource issues


Screening Guide

Tool 1



Tool 4

23


DecisionMade by Executive Committee

1. Rejected (No)

2. Approved (Yes)

3. Conditional (Yes but…a. Single caseb. Clinical auditc. Other (e.g. training)

4. Research Usea. Clinical trialb. Support of project

5. Referred


Tool 4


C r

i t e

r i

a

24


RECOMMENDATIONOR DECISION

CRITERIA & RATIONALE(e.g., may include)

1. Rejected• Negative, poor, or no data on efficacy• Insufficient evidence of safety• Decreases or worsens service delivery

2. Approved • Efficacy and Safety well established• Enhanced population health is likely• Strategic fit is strong• Will likely improve service delivery• Financial impact is the same or better than current

practice (Cost Neutral)• Cost-effectiveness is likely the same or better than

current practice

25



CRITERIA & RATIONALE

3a. Conditional –Single Case

• Urgent/emergent approval of a single case where options are limited

3b. Conditional –Clinical Audit

• Evidence promising for efficacy, but limited for effectiveness

• Good evidence for safety• Cost-effectiveness may be uncertain• Advantage over current practice needs to be established• Cost needs to be established

3c. Conditional –Other

• May require new training protocols• Patient population may need to be defined

26




4a. Research Use –Clinical Trial

• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed

4b. Research Use –Support of Project

• Innovative project proposed by applicant• Funding and research protocols in place• Acceptable use of resources

5. Referred • Technology has potential impact beyond the scope of surgery

• Working group required

27

Tool 4: “Device X” - Decision



4a. Research Use –Clinical Trial

• Efficacy uncertain or controversial• Safety uncertain• Population health is uncertain• Effect on service delivery is uncertain• May be innovative• Clinical trials protocols must be observed

28

Prioritize Technologies for funding or purchaseRecommendationsRecommendationsRecommendationsRecommendations

Prioritize Technologies for funding or purchase

DecisionsDecisionsDecisionsDecisions

Prioritization Tool

Tool 5

Proposal forTechnology OneTechnology OneTechnology OneTechnology One

Proposal forTechnology TwoTechnology TwoTechnology TwoTechnology Two

Proposal forTechnology ThreeTechnology ThreeTechnology ThreeTechnology Three

Tool 5: Prioritization Tool

29

Step 1. Screening Guide

Step 2. Criteria Review

Step 3. Criteria Weighting

Step 4. Criteria Rating Scales

Step 5. Technology Scoring

Step 6. Technology Ranking

Step 7. Additional Checks

Adapted from Drs. Craig Mitton and Cam Donaldson

Tool 5: Prioritization Tool

30

Objectives:



3. Lessons Learned

31

Lessons Learned:

WEAKNESSES

•Criteria have not been weighted

•Prioritization Tool has not been tested

•Patient interests not well represented

•Administrators and practitioners sometimes resistant to change

32

Lessons Learned:

STRENGTHS

•Bottom-Up approach – User develop ensures buy-in

•Reviewers satisfied with criteria and technology evaluation tools

•Provides a systematic, consistent and transparent approach to technology evaluation and decision-making

33

Acknowledgements

Supported by:• Canadian Agency for Drugs and Technology in Health• The Department of Surgery & Surgical Services – Calgary• The Calgary Health Research Portfolio – Calgary• Surgery Clinical Network HTAI - AHS

Participating Departments from the former Calgary H ealth Region:• Surgery & Surgical Services• Anesthesia • Cardiac Sciences• Critical Care Medicine• Internal Medicine• Professional Practice & Development• Medical Device Safety & Risk Management• Pharmacy• Physician Leadership Portfolio, and the • Calgary Health Research Portfolio

34

Multiple CriteriaHTA Into Practice

Local HTA Program

InnovationCREATE

OR DIE!

Contact information:

Dr. Paule Poulin

Alberta Health Services - Calgary

(403) 944-1652

[email protected]

Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD...

Documents

Transcript of Multiple Criteria HTA Into Practice Local HTA Program - Multi... · 1 Presenter: Paule Poulin, PhD...