Can We Reliably Benchmark HTA Organizations?

Michael Drummond Centre for Health Economics

University of York

Outline of Presentation

• Some background • Methods

• Results

• Discussion

Some Background

• In recent years, several reimbursement agencies have begun using HTA.

• Essentially, HTA has become ‘hard-wired’ into the reimbursement or coverage decision.

• Economic evaluation has an increasing prominence in these HTAs (for resource allocation decisions).

• It is important to establish some principles of ‘good practice’.

Methods (1)

• Developed 15 Key Principles for the Improved Conduct of HTA for Resource Allocation Decisions, based on the existing literature and our own views.

• Solicited comments from key opinion leaders in HTA.

• Published in IJTAHC 2008;24:244-58, along with editorial comments.

Methods (2) • Selected 14 HTA entities from 9 countries, to

reflect: - worldwide coverage; - traditional HTA agencies and reimbursement agencies; - established and newly-formed entities; - public and privately funded entities.

• Assessed whether the entities ‘supported’ and/or ‘implemented’ the various principles.

• Used our own expertise, supplemented by local experts for Australia, Brazil, Canada, Korea and Taiwan.

Methods(3)

• Based on the preliminary effort, several issues were raised

• A series of audit questions were developed which, it is argued, would enable us to benchmark HTA organizations in a more reliable manner

Results

• The Key Principles • The Assessments

• Additional considerations in

benchmarking HTA organizations • Audit questions for the Key principles

Key Principles for HTA

• Structure of HTA • Methods of HTA • Processes for Conduct of HTA • Use of HTA in Decision-making

Structure of HTA

• The goal and scope of the HTA should be explicit and relevant to its use.

• HTA should be an unbiased and transparent exercise.

• HTA should include all relevant technologies.

• A clear system for setting priorities for HTA should exist.

Methods of HTA • HTA should incorporate appropriate methods for

assessing costs and benefits. • A full societal perspective should be considered

when undertaking HTAs. • HTAs should explicitly characterize uncertainty

surrounding estimates. • HTAs should consider and address issues of

generalizability and transferability. • HTAs should consider a wide range of evidence and

outcomes.

Processes for Conduct of HTA

• Those conducting HTAs should actively engage all key stakeholder groups.

• Those undertaking HTAs should actively seek all available data.

• The implementation of HTA findings needs to be monitored.

Use of HTA in Decision-Making

• HTA should be timely. • HTA findings need to be communicated

appropriately to different decision-makers. • The link between HTA findings and

decision-making processes needs to be transparent and clearly defined.

Key Principles: How Widely Are They Adopted?

Assessment Criteria

‘+’ Means that the organization ‘supported’ the Principle in written guidelines or other forms, regardless of whether it was followed.

‘++’ Means that the organization

‘implemented’ the Principle in published reports and that the decisions based on these reports demonstrate adoption of the Principle.

Key Principle

CMS (US)

Washington Medicaid/ DERP (US)

WellPoint

(US)

BCBS TEC

(US)

NICE (UK)

IQWiG

(Germany)

DAHTA@ DIMDI

(Germany)

Structure of HTA program

1 The goal and scope of the HTA

should be explicit and relevant to its use

++ ++ ++ ++ ++ ++ +

2 HTA should be an unbiased and transparent exercise

++ ++ ++ ++ ++ ++

3 HTA should include all relevant technologies

++ ++ ++ ++ ++

4 A clear system for setting priorities for HTA should exist

+ + ++ + ++

Methods of HTA

5 HTA should incorporate appropriate methods for assessing costs and benefits

+ ++ + ++

6 HTAs should consider a wide range of evidence and outcomes

++ + + ++ ++ ++ ++

7 A full societal perspective should be considered when undertaking HTAs

+

8 HTAs should explicitly characterize uncertainty surrounding estimates

++ ++ + +

9 HTAs should consider and address issues of generalizability and transferability

+ ++ +

Key Principle

CMS (US)

Washington Medicaid/ DERP (US)

WellPoint (US)

BCBS TEC (US)

NICE (UK)

IQWiG (Germany)

DAHTA@ DIMDI

(Germany)

Processes for conducting HTA

10 Those conducting HTAs should actively engage all key stakeholder groups

++ + ++ ++

11 Those undertaking HTAs should actively seek all available data

++ + ++ ++ ++ +

12 The implementation of HTA findings needs to be monitored

+ +

Use of HTA in decision making

13 HTA should be timely ++ ++ + + ++ +

14 HTA findings need to be communicated appropriately to different decision makers

++ ++ ++

15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined

+ + +

Notes: “+” signifies that the organization “supported” the principle in question in written guidelines or other form, regardless of whether they actually follow it. “++” means that the organization “implemented” the principle in published reports, and decisions based on these reports demonstrate adoption of the specific principle.

Key Principle

TLV (Sweden)

SBU (Sweden)

CADTH (Canada)

HIRA (Korea)

PBAC (Australia)

Anvisa (Brazil)

DHTA (Taiwan)

Structure of HTA program

1 The goal and scope of the HTA should be explicit and relevant to its use

+ + ++ ++ ++ + ++

2 HTA should be an unbiased and transparent exercise

++ ++ ++ + +

3 HTA should include all relevant technologies

++ + +

4 A clear system for setting priorities for HTA should exist

+ ++ ++ ++

Methods of HTA

5 HTA should incorporate appropriate methods for assessing costs and benefits

++ + ++ + ++ +

6 HTAs should consider a wide range of evidence and outcomes

++ ++ ++ + + ++

7 A full societal perspective should be considered when undertaking HTAs

++ ++ +

8 HTAs should explicitly characterize uncertainty surrounding estimates

++ + ++ +

9 HTAs should consider and address issues of generalizability and transferability

+ ++ + +

Key Principle

TLV (Sweden)

SBU (Sweden)

CADTH (Canada)

HIRA (Korea)

PBAC (Australia)

Anvisa (Brazil)

DHTA (Taiwan)

Processes for conducting HTA

10 Those conducting HTAs should actively engage all key stakeholder groups

++ + ++ +

11 Those undertaking HTAs should actively seek all available data

++ ++ + ++ + ++

12 The implementation of HTA findings needs to be monitored

++

Use of HTA in decision making

13 HTA should be timely ++ + + + + + ++

14 HTA findings need to be communicated appropriately to different decision makers

+ ++ ++ + +

15 Link between HTA findings and decision-making processes needs to be transparent and clearly defined

++ + ++ ++

Summary of Findings

• Support of principles High: HTA should be unbiased and transparent Low : HTA should adopt a societal perspective

• Implementation of principles High: HTA should consider a wide range of evidence Low: Link between HTA and decision-making should be

clearly defined

Discussion Points • Only a convenience sample of HTA entities. • Different entities have different remits, which

affects the adherence to some principles. • The Key Principles merit more discussion: -some are controversial -some may be too idealistic -some are partially contradictory.

• Judgements of ‘implementation’ are impressionistic and we need a tighter, audit approach.

Additional Considerations in Benchmarking

• Understanding the varying statutory roles of HTA organizations

• Weighting the relative importance of the Key Principles

• Recognizing the varying stages of development of HTA organizations

• Developing unambiguous audit criteria

Understanding the Varying Statutory Role of HTA organizations

• The remit of an organization can limit its ability to

adhere to a principle • Some principles (eg. KP 3 HTAs Should Consider

All Relevant Technologies) relate to the whole jurisdiction, not an individual organization

• Therefore, need to distinguish between ‘maximum score’ and ‘maximum attainable score’

• Nevertheless, some elements of the remit need to be questioned within the jurisdiction (eg KP 7 HTAs Should Adopt a Societal Perspective)

Weighting the Relative Importance of the Key Principles

• Clearly, all key principles may not be of equal importance

• The relative weightings could vary by jurisdiction

• There are trade-offs between principles (eg between KP 10 on Stakeholder Involvement and KP 13 on Timeliness)

• Trade-offs should be discussed at the level of the individual jurisdiction

Recognizing the Varying Stages of Development of HTA organizations

• It takes time to formulate a view about the

various key principles • As HTA organizations undertake

studies/make decisions, case-law is developed

• Initially, the organization may not have the human resources to undertake all the tasks implied by the principles, even if it wanted to

Developing Unambiguous Audit Criteria

• Need audit criteria that can be defended

• The objective should be to develop a series of questions that can be answered ‘yes’ or ‘no’, or graded in an unambiguous manner (eg a Likert Scale)

• Paper published in the International Journal of Technology Assessment in Health Care 2012;28(2): 159-65

Audit Criteria for the 15 Key Principles

• The objective was to create, from the 15 Key Principles, a series of questions that can unambiguously be answered

• This preliminary list is based on the points we raise in the text of the original Key Principles paper

• It may be possible to supplement this list with other questions

Principle1.The goal and scope of the HTA should be explicit and relevant

to its use.

• Is the remit for the HTA organization

clearly defined? • Is a scoping document drawn up prior to

an HTA, containing information on the specific decision problem to be addressed, the alternatives to be considered, the relevant patient populations and the time horizon?

Principle 2. HTA should be an unbiased and transparent exercise

• Is the HTA organization independent of the body making the reimbursement or coverage decision?

• Are the recommendations of the HTA organization made by an independent expert advisory committee?

• Are the meetings of the committee held in public? • Are all supporting information and the basis of the

recommendations made publicly available? • Does the organization normally commission outside groups to

undertake the HTA? • Are the reports produced subjected to independent peer-

review prior to final determination? • Are the draft conclusions subject to review by stakeholders

and the public, with rationale underlying final determinations of contentious issues?

Principle 3. HTA should include all relevant technologies

• Are all types of technologies (e.g. drugs, devices, diagnostics, procedures, behavioural modification) considered?

• Within each category, are both new and existing technologies considered?

• In assessments of new technologies, are all relevant alternatives considered?

Principle 4. A clear system for setting priorities for HTA should exist

• Does a formal system for prioritizing and selecting topics exist?

• Is the priority-setting approach clear and transparent?

Principle 5. HTA should incorporate appropriate methods for assessing

costs and benefits • Does the HTA organization consider costs as well

as benefits and harms? • Does the HTA organization have published

methods guidelines for assessing the benefits, harms and costs of health technologies?

• Is a full systematic review of clinical evidence required as a basis for economic modeling?

• Does the team undertaking HTAs on behalf of the organization include individuals with skills in epidemiology/biostatistics, health services research and economics?

Principle 6. HTAs should consider a wide range of evidence and outcomes.

• Does the relevant clinical evidence include

observational and non-randomized studies, as well as RCTs?

• Does the HTA consider impacts on quality of life and other patient-reported outcomes, as well as clinical events?

• Does the HTA consider relevant sub-groups of the patient population (e.g. by baseline risk)?

Principle 7. A full societal perspective should be considered when

undertaking HTAs • Does the HTA consider only consider the impact on a

specific budget, for example for drugs? • Does the HTA consider all health care costs? • Can other costs be included as extra information? • Are productivity gains and losses (ie indirect costs and

benefits) considered when relevant? • Are costs for informal care included when relevant? • Are costs in added years of life included in the cost-

effectiveness ratio?

Principle 8. HTAs should explicitly characterize uncertainty surrounding

estimates • Does the HTA include a sensitivity analysis? • Are confidence intervals presented for key

estimates? • Have the key deficiencies in available data

been identified and discussed? • Is an agenda for key future research

proposed?

Principle 9. HTAs should consider and address issues of generalizability

and transferability

• Does the HTA organization have methods guidance for dealing with transferability issues when using data or analyses from other jurisdictions?

• Does the HTA organization consider the generalizability of the results of its studies to other patient populations, health care delivery systems or practice settings that are relevant for its jurisdiction?

Principle 10. Those conducting HTAs should actively engage all key stakeholder groups (eg professional

bodies, patient organizations, manufacturers). • Is the HTA organization formally required to engage stakeholders in

its activities? • Does the HTA organization involve stakeholders in the scoping of

HTAs? • Does the HTA organization have a mechanism for identifying the

relevant stakeholders • Does the organization encourage or require submissions of evidence

from stakeholders? • Does the organization allow stakeholders to comment on reports at

the draft stage? • Does the organization allow stakeholders to appeal against

recommendations/decisions? • Do the organization’s committees include stakeholder representation

(eg patient groups, technology manufacturers, clinical specialists)?

Principle 11. Those undertaking HTAs should actively seek all

available data • Does the systematic review of clinical

evidence include the grey literature and unpublished data?

• Does the HTA organization have processes for handling confidential data from manufacturers?

Principle 12. The implementation of HTA findings needs to be monitored

• Does the HTA organization develop an implementation plan for its HTAs?

• Does the HTA organization monitor the impact of its recommendations?

Principle 13. HTA should be timely.

• Does the HTA organization have a defined time period for conducting HTAs/producing recommendations?

• Does the HTA organization adhere to the agreed timelines?

• Does the organization have a mechanism to update its HTAs/recommendations within a given time period?

Principle14. HTA findings need to be communicated appropriately to

different decision-makers. • Does the HTA organization develop a

communications plan for its recommendations and decisions?

• Are separate versions of reports produced for different audiences (e.g. health professionals, decision-makers, general public)?

• Is the effectiveness of communication monitored and evaluated?

Principle 15. The link between HTA findings and decision-making

processes needs to be transparent and clearly defined

• Does the organization distinguish between the scientific assessment of the evidence and the appraisal decision?

• Does the organization have an explicit decision rule for acceptance/non-acceptance of health technologies?

• Does the organization have a transparent approach for weighing various considerations (e.g. cost-effectiveness, equity)?

• Does the agency recommend, or operate, conditional reimbursement/coverage-with-evidence-development schemes?

• Does the organization distinguish between identifiable sub-groups of the patient population when making decisions?

Conclusions • Benchmarking of HTA entities is currently

controversial, but it is feasible • New approaches need to be developed that

yield unambiguous assessments • Benchmarking could either be undertaken by

an independent organization, or by the HTA agencies themselves

• The main focus should be on self-improvement by the agencies themselves, rather than on external criticism