EUnetHTA WP7: Deliverable 7.2 - Final Report

EUnetHTA WP7 Deliverable 7.2 – Final report

1

EUnetHTA WP7: Deliverable 7.2 - Final Report

Disclaimer: EUnetHTA is supported by a grant from the European Commission. The sole responsibility for the content of this document lies with the authors and neither the European Commission nor EUnetHTA are responsible for any use that may be made of the information contained therein.


2

Contents

Abbreviations ........................................................................................................... 4 1. Executive Summary .......................................................................................... 5 4. Introduction and Objectives ........................................................................... 11

Introduction ........................................................................................................... 11 Objectives ............................................................................................................. 11

5. Methods ........................................................................................................... 12 Data collection ...................................................................................................... 12 Methods of Analysis .............................................................................................. 13

6. Report structure .............................................................................................. 14 7. Results: Overview of the JA3 use of JA/CA.................................................. 16

Response Rate ..................................................................................................... 16 Topic relevance .................................................................................................... 16

Use of JA/CA ........................................................................................................ 18 Countries reporting use of JA/CA ......................................................................... 20 Type of use of JA/CA ............................................................................................ 20 Benefits of use ...................................................................................................... 22

Non-use of JA/CA ................................................................................................. 23 Summary of limiting and preventing factors .......................................................... 25

8. Results: Comparison of use in JA2 and JA3 ................................................ 28

Pharmaceutical Assessments ............................................................................... 28 Other Technology Assessments ........................................................................... 29

Conclusions .......................................................................................................... 32 9. Relationship between participation and implementation ............................ 34

Other technologies ................................................................................................ 34

Pharmaceuticals ................................................................................................... 36 Conclusions .......................................................................................................... 38

10. Results: Awareness .................................................................................... 39 Fitness for purpose of procedures ........................................................................ 39

Agency perceptions of changes introduced in JA3 ............................................... 39 Agency recommendations for the future ............................................................... 40

Conclusions .......................................................................................................... 41 11. Results: Timing ............................................................................................ 43

Fitness for purpose of procedures ........................................................................ 43


Conclusions .......................................................................................................... 45 12. Results: Relevance and transferability ...................................................... 46

Fitness for purpose of procedures ........................................................................ 46


Conclusions .......................................................................................................... 49 13. Results: Reliability and scientific rigour ................................................... 50

Fitness for purpose of procedures ........................................................................ 50 Agency perceptions of changes introduced in JA3 ............................................... 51 Agency recommendations for the future ............................................................... 52 Conclusions .......................................................................................................... 52

14. Results: Evidence and Methodology ......................................................... 53 Fitness for purpose of procedures ........................................................................ 53


3

Agency perceptions of changes introduced in JA3 ............................................... 54

Agency recommendations for the future ............................................................... 55 Conclusions .......................................................................................................... 56

15. Results: Usability and Reporting Structure .............................................. 57 Fitness for purpose of procedures ........................................................................ 57 Agency perceptions of changes introduced in JA3 ............................................... 57 Agency recommendations for the future ............................................................... 58 Conclusions .......................................................................................................... 59

16. Results: Language ...................................................................................... 60 Fitness for purpose of procedures ........................................................................ 60 Agency perceptions of changes introduced in JA3 ............................................... 61 Agency recommendations for the future ............................................................... 61 Conclusions .......................................................................................................... 61

17. Results: Transparency and Independence................................................ 62 Fitness for purpose of procedures ........................................................................ 62

Agency perceptions of changes introduced in JA3 ............................................... 63

Agency recommendations for the future ............................................................... 63 Conclusions .......................................................................................................... 64

18. Recommendations for implementation support in a future model of HTA cooperation ............................................................................................................. 65

Conclusions .......................................................................................................... 67 Appendix 1: Implementation feedback survey .................................................... 68

Appendix 2: Response rate ................................................................................... 75


4

Abbreviations

CA Collaborative Assessment

CHMP Committee for Medicinal Products for Human Use

COI Conflict of Interest

CRP C-Reactive Protein

CUR HTA Core model: Health Problem and Current Use of Technology Domain

DNA Deoxyribonucleic acid

DOI Declaration of interest

EC European Commission

EUnetHTA European Network for Health Technology Assessment

EVAR Endovascular aneurysm repair

EPL EUnetHTA prioritisation list

FLACS Femto-Laser Assisted Cataract Surgery

HIFU High intensity focused ultrasound

JA Joint Assessment

JA2 Joint Action 2

JA3 Joint Action 3

JA/CA Joint or Collaborative Assessment

MD Medical Devices

NIPT Non-invasive prenatal testing

PICO/PICOS/PICOTS/PICOD Population, intervention, comparator, outcome, timeframe, study design

POCT Point of care testing

PT Pharmaceutical

OT Other technology

REA Relative Effectiveness Assessment

SOP Standard operating procedures

TAVI Transcatheter aortic valve implantation

TEC HTA Core model: Description and Technical Characteristics of the Technology

TEVAR Thoracic Endovascular Aortic Repair

WP Work Package


5

1. Executive Summary

In this report we describe:

• Use in Joint Action 3 (JA3) of joint assessments (JA) and collaborative

assessments (CA) and compare this with use in Joint Action 2 (JA2).

• How changes in JA and CA processes have affected implementation.

• The extent to which structures, methods and processes being implemented are

“fit for purpose” in terms of awareness, timeliness, relevance, scientific rigour,

evidence and methodology, usability, transparency and independence.

• Recommendations for structures to support implementation in a future model of

Health Technology Assessment (HTA) cooperation based on the JA3 experience.

We do this using survey data about JA2 use collected in JA2 and followed up in JA3,

and survey data about JA3 use collected in JA3. The survey data are complemented

with interview data, responses to email questions and records about agency

involvement in JA2 and in JA3.

Implementation data are available for 27 JA/CA published under JA3. 298 examples

of use have been reported. In total there have been 89 uses of the 7 pharmaceutical

(PT) assessments and 209 uses of the 20 other technology (OT) assessments. Most

uses of the PT assessments (n= 60) are in assessment procedures, with only 29

examples of sharing via dissemination. In contrast, for OT there have been 209

examples of use, with 85 of those in assessment procedures and 124 examples of

sharing via dissemination. Use of PT assessments is usually as part of a

reimbursement and/or pricing procedure, while use of OT assessments is varied

including assessment procedures but also decisions to assess, decisions to review

and targeted and non-targeted dissemination to support local decision making.

For PT there has been increased production and use of JA/CA in JA3 compared to

JA2. For OT there has been increased production of JA/CA, but there is currently

less use of JA/CA in JA3 compared to JA2. For both PT and OT there is an

increased number of countries that have used JA/CA in JA3 compared to JA2.

Changes in JA3 processes are seen to have improved implementation. Changes

made to awareness, timeliness and relevance have allowed greater use. Changes in

scientific rigour, evidence and methodology, usability, transparency and

independence have improved the confidence with which agencies use JA/CA.

Most agencies still report factors that can prevent or limit the extent to which JA/CA

are used. The most frequent factors identified are timeliness and relevance. In

addition, for PT assessments requirements to use a specified report structure and

national language can limit use.


6

The tables overleaf make recommendations for key scientific and procedural

features that if put in place would help maximise implementation of JA/CA in a future

model of HTA cooperation. It makes recommendations for 3 groups:

1. the entity responsible for JA/CA in a future model of HTA cooperation (table 1),

2. agencies and countries participating in future HTA cooperation (table 2), and

3. the PT industry (table 3).

This latter group is included in the recommendations because of the role that

industry plays in initiating PT assessments in many countries and providing the

evidence submission for these assessments. Different groups have the primary

responsibility for resolving different issues. Agencies and countries participating have

primary responsibility for addressing language barriers to uptake, while the entity

producing JA/CA in a future model of HTA cooperation has primary responsibility for

creating timely, rigorous, usable and transparent reports. For areas such as

awareness, relevance and evidence and methodology groups have a shared

responsibility for overcoming barriers.

Support for implementation is needed in a future model of HTA cooperation. The

level of implementation support required will differ by agency and country and will

depend on how developed HTA systems are in individual countries and the levels of

HTA experience and expertise within these countries. Once the details of a future

model of HTA cooperation are known, agencies still establishing HTA systems

should be prioritised and offered early, individualised, facilitative support at a country

level (that is, encompassing HTA agencies, and also commissioners and decision

makers) to put in place the capacity, expertise and processes to implement future

ways of working. Modalities of support proposed included online resources and

implementation advisers. The use of audit and monitoring provoked mixed opinions

as to whether these tools were appropriate.


7

Table 1: Recommendations for the entity responsible for JA/CA in a future model of HTA cooperation

Awareness • Identify the target user group and ensure target users are engaged in the network

• Develop a formal outreach strategy to individuals and groups who are not part of the network as well as to networks and organisations working in related areas

• Offer advice and support for developing or scoping HTA capacity for countries who are still establishing HTA programmes.

• Implement an alert system within and outside of the network using multiple media channels to proposed JA/CA, JA/CA starting, project plans publishing and JA/CA publishing.

• Provide regular activity updates through newsletters

• Facilitate breadth of involvement in activities to develop familiarity and trust among agencies

• Regularly repeat training and knowledge sharing activities to respond to high staff turnover in agencies

Timeliness • Support planning by agencies to use JA/CA: o Early information sharing about the JA/CA scope and contents o Early notice of publication dates and alerts to delays o Sharing of draft documents

• For PT, publication close to marketing authorisation

• For OT, collaborate with Medical Device (MD) regulators to develop an anchor for OT HTA timing in the future

Relevance • Link to horizon scanning initiatives to identify innovative topics early in their lifecycle

• When selecting topics for OT JA/CA consider the number of local assessments already available

• Develop a clear definition of what agencies can expect in a ‘European’ scope (that is, population, intervention, comparator, outcomes, timeframe and study design (PICOTS)) for JA/CA

• Require authors to author with a European perspective in mind

• Provide an opportunity for agencies to feedback and influence scope of the JA/CA

• Target the PICO survey to individuals and groups who define the scope for the JA/CA in a country

Scientific rigour • Clearly define the methodological expertise required by JA/CA authoring teams

• Put in place a programme to ensure that methodological expertise required is available e.g. through internal capacity building and/or external expert support

• Allocate sufficient time in the JA/CA process for authoring

• Define a mechanism acceptable to agencies that ensures published reports do not contain factual errors

Evidence and Methodology

• Include consideration during OT scoping of the study designs available to respond to the question and evidence availability and consequent suitability of the topic for JA/CA

• Develop mutual agreement of methodology to be applied and the extent of the interpretation of the evidence

• Ensure consistent application of methodology across JA/CA to support planning by agencies


8

• Put in place a regular programme of maintenance, updating and extending of methodological guidance

• Put in place a programme of capacity building to support development of methodological expertise

• Further development of guidance on how to incorporate patient and clinical expert evidence into JA/CA

Usability • Provide guidance to authors about using templates and expectations for information to be included in JA/CA

• Broker an agreement about JA/CA length and depth to meet the needs of a majority of agencies

• Agree common phrases for use in JA/CA

• Avoid use of terminology and language peculiar to the network

• Include an editing procedure for JA/CA to support clarity of language

• Ensure any new templates are given enough time for agencies to get used to them before they are changed

• Develop a monitoring and evaluation system to capture the impact of HTA cooperation over time and support ongoing changes to improve usability

Language -

Transparency and Independence

• Publish company evidence submissions used to develop JA/CA

• Monitor conflict of interest (COI) procedures to ensure that rules of involvement do not detrimentally affect the engagement of the most knowledgeable experts


9

Table 2: Recommendation for agencies and countries participating in HTA cooperation

Awareness • Where feasible create an opportunity in standard operating procedures (SOPs) to allow priority topics to be proposed for JA/CA and for JA/CA topics to be circulated to decision makers

• Implement a communication procedure to raise awareness of JA/CA to relevant external groups and organisations using and commissioning HTA in the country

• Include an alert in agency procedures for assessors to check for ongoing or published JA/CA

• Include a question in company applications for reimbursement or evidence submissions about whether JA/CA is being prepared or is already available

• Develop a standardised agency process for adapting JA/CA

Timeliness -

Relevance • For HTA systems that define the topics for assessment, consider how priorities for assessment can be collected in advance through horizon scanning or notification systems

• Allocate dedicated agency resource to input into PICOTS surveys for JA/CA

• Develop an agency procedure for gathering PICOTS information to inform JA/CA

• Embed sufficient flexibility in processes to extract relevant content material and ignore other content material in JA/CA

Scientific rigour • Ensure staff have the skills to participate in JA/CA assessment teams by allowing participation in training courses and webinars

Evidence and Methodology

• Identify and resolve any areas of methodology where agency methodology actively hinders use of JA/CA

• Promote awareness of JA/CA methodology and processes within the agency through staff training and knowledge transfer

Usability -

Language • Adjust procedures and legal regulations to allow technical documents to be prepared in English

Transparency -


10

Table 3: Recommendations for PT Industry

Awareness • Disseminate from a global level to a local level the availability of JA/CA

• Highlight in local evidence submissions the availability of JA/CA

Timeliness • Coordinate local launch strategies with publication of JA/CA

Relevance • In scenarios where industry define the scope used in local reimbursement applications, ensure this scope is incorporated into the JA/CA evidence submission

• Use local submission requirements to justify the scope of the JA/CA evidence submission

Scientific rigour • When preparing JA/CA evidence submission follow submissions guidelines and methodological guidelines

Evidence and methodology

• When preparing JA/CA evidence submission follow submissions guidelines and methodological guidelines

Usability -

Language -

Transparency and independence

• Allow publication of company submission files

• Allow publication of all relevant data for the assessment


11

4. Introduction and Objectives

Introduction

There are two main objectives of the European Network for Health Technology

Assessment (EUnetHTA) JA3:

1.1 To increase the use, quality and efficiency of joint HTA work at

European level to support evidence-based, sustainable and equitable

choices in healthcare and health technologies and ensure re-use in

regional and national HTA reports and activities.

1.2 To support voluntary cooperation at scientific and technical level

between HTA agencies by providing a sustainable model for the scientific

and technical mechanism of a permanent European cooperation on HTA

This report supports these objectives. The report describes use of JA3 JA/CA and

compare this with use in JA2 and makes recommendations for structures to support

implementation in a future model of HTA cooperation based on the JA3 experience.

Objectives

This report answers the following research questions:

1. Compared with JA2, has there been an increase in JA3 in the number and

uptake of JA/CA?

2. What is the relationship between changes in uptake and changes in JA/CA

processes, including:

a. changes that were implemented between JA2 and JA3, and

b. changes that were implemented in JA3?

3. To what extent are the structures, methods and processes being used fit for

purpose meeting the individual needs of agencies?

4. What are the key scientific and procedural features that need to be put in

place by the individuals, groups, and agencies coordinating and participating

in future HTA cooperation to maximise implementation of JA/CA?

5. What support for implementation (e.g. implementation advisers, mechanisms

for shared learning, measurement and evaluation of implementation) should

be built into a future model of HTA cooperation?


12

5. Methods

Data collection

At the start of JA3, EUnetHTA developed a feedback tool to capture use of JA/CA

(see appendix 1). The tool was developed collaboratively by Work Package (WP) 1,

WP3 and WP7 to support monitoring and evaluation by WP3 and WP7. The draft

tool was subject to consultation with EUnetHTA WP7 partners.

The final tool is available on the EUnetHTA intranet and EUnetHTA partners are

asked to complete the feedback survey after publication of each JA/CA and update

their survey entry if their work status changes. A password protected version is

available on the EUnetHTA internet so that non-EUnetHTA agencies can also

provide feedback on their use of JA/CA.

The data used in this analysis were downloaded June 29th, 2020.

The data from the WP7 implementation survey is supplemented in this report by the

findings from an e-mail survey sent to selected agencies (N=38) to identify reasons for

patterns of use and changes in use between JA2 and JA3. We received 22 responses

to the email survey.

Additionally, semi-structured qualitative interviews were undertaken with 12 selected

agencies to collect more detailed information, specifically on:

• their perceptions of the changes made to the production processes and the

impact of these on implementation

• their ability to participate in process stages requiring user engagement (for

example topic identification, selection and prioritisation pilots, scoping and

authoring)

• the extent to which the structures, methods and processes being implemented

are considered "fit for purpose" in meeting their needs. Areas of discussion

focussed on awareness, timeliness, relevance, scientific rigour, evidence and

methodology, usability, transparency and independence.

WP7 also updated the JA2 evaluation results to capture examples of use of JA2

assessments that had occurred after the end of JA2. The JA2 implementation data in

this report includes follow up to April 2018. Further details of the JA2 data used in

this report are available in the implementation report published in May 20181.

1 https://eunethta.eu/wp-content/uploads/2018/06/May-2018-Implementation-report_website_FINAL.pdf


13

Methods of Analysis

Stage 1: Identification of procedural changes

In stage 1 of the work WP7, WP4 and WP6 lead and co-lead partners developed a

list of changes in JA3 assessment processes to be evaluated. Changes evaluated

were those perceived to have an effect on implementation either in terms of the

absolute level of implementation (e.g. number of agencies using JA/CA) or on the

implementation experience (e.g. trust in the JA/CA or ease of JA/CA adaptation).

The changes were grouped into key themes for analysis, for example: changes

aimed to improve timeliness, relevance, scientific rigour and awareness.

Differences in production processes for PT and OT assessments means that this

report evaluates a different set of changes for PT compared with OT assessments.

Stage 2: Relationship between existing procedures, procedural changes and uptake

In stage 2 of the work the relationship between changes in JA3 procedures and

uptake of JA/CA was explored. In this stage of the work the quantitative and

qualitative data from the implementation feedback survey, records of partner

involvement, email survey and interviews were thematically analysed. The themes

for analysis were the effect of changes on (1) awareness of reports, (2) report

timeliness, (3) scope and content relevance, (4) reliability and scientific rigour, (5)

appropriateness of evidence and methodology applied, (6) usability in terms of report

structure and language, (7) transparency and independence.

Stage 3: Definition of uptake processes

The final stage of the work sought to make recommendations for the scientific and

procedural features that need to be in place to maximise uptake of JA/CA in a future

model of HTA cooperation. These recommendations were developed from the

outcomes of the stage 2 work and additionally areas where agencies identified that

further improvements were needed.


14

6. Report structure

The results and recommendations are presented in the following order:

Section 5 presents a summary of the implementation data reported in JA3

Section 6 addresses the first research question:

1. Compared with JA2, has there been an increase in JA3 in the number and

uptake of JA/CA?

Section 7 explores the relationship between reported use in JA2, participation in

production processes in JA3 and reported use in JA3.

Sections 8-14 address the second, third and fourth research questions:

2. What is the relationship between changes in uptake and changes in

JA/CA processes, including:

a. changes that were implemented between JA2 and JA3 and

b. changes that were implemented in JA3?

3. To what extent are the structures, methods and processes being

implemented “fit for purpose” meeting the individual needs of

agencies?

4. What are the key scientific and procedural features that need to be

put in place by the individuals, groups, and agencies coordinating

and participating in future HTA cooperation to maximise

implementation of JA/CA?

These sections are structured around the following key themes

• Awareness

• Timing

• Relevance and transferability

• Reliability and scientific rigour

• Evidence and methodology

• Usability and reporting structure

• Language


15

• Transparency and independence

Section 16 addresses the final research question:

What support for implementation should be built into a future model of HTA

cooperation?


16

7. Results: Overview of the JA3 use of JA/CA

This report includes 27 JA/CA that have been published under JA3 up to the end of

April 2020 for which implementation data are available, 20 OT assessments and 7

PT assessments. An 8th PT assessment sotagliflozin (PTJA04), has been published

but as yet no implementation data are reported because it is not launched in Europe.

To accurately capture the implementation of JA/CA a follow up period of

approximately 18 months to 2 years is needed for PT and of approximately 3 years

for OT. At the time of writing this report, 4 OT assessments (OTCA17, OTCA18,

OTCA20 and OTCA22) and 3 PT assessments (PTJA06, PTJA08 and PTJA09)

have been published for under 6 months. An additional 3 OT assessments

(OTCA10, OTCA14 and OTCA19) and 1 PT assessment (PTJA07) have been

published for less than 1 year. Data reported for these JA/CA will be affected by an

inadequate follow-up period.

Response Rate

The agency response rate to the survey is high with similar levels for OT and PT. For

JA/CA published over 6 months ago the response rate ranges from 43-86% (see

appendix 2 for a breakdown). As expected, response rates increase over time as

companies submit applications for reimbursement and/or agencies make decisions

to complete or not to complete work on a topic. The highest response rates are for

assessments with the longest follow-up period.

Shortly after publication of JA/CA responses to the implementation feedback survey

are usually from agencies who know they will not or are unlikely to work on a topic

e.g. it is out of remit or work has already been carried out. In addition, earlier

responders to the survey tend to be (1) agencies in countries where companies

launch their PT products earliest, (2) from authors of OT JA/CA where the topic is

already in their work programme and from (3) agencies who are able to choose their

own work areas and have identified the topic as relevant to their context. Agencies

responding early on to the implementation feedback survey are not a representative

sample of all JA/CA users and this should be considered when interpreting the

implementation feedback survey data from recently published assessments.

Topic relevance

As shown in table 4, EUnetHTA is usually selecting topics that are within an

agency’s remit. There are slightly higher levels of PT topics being out of remit than

OT topics. In their responses agencies indicated that they may (1) only do certain PT

products e.g. inpatient or outpatient products, (2) have additional criteria to select PT

topics for HTA such as likely budget impact or (3) only do specific types of PT HTA

e.g. multiple technology assessments.


17

In contrast OT topics are more often within remit but not in the work programme.

From interviews there was mixed feedback about JA3 OT topic relevance, for some

agencies, topics were more relevant than for others:

“The topics, for the most part, have also been relevant to our work”

“Agency also sees a potential for using (parts of) OT assessments in our

own assessments, but so far we have not been commissioned to do an

assessment for a technology that already has been evaluated by

EUnetHTA”.

For all topics there is relatively little planned activity reflecting that agencies often

react to requests for assessments and work with little notice of an assessment being

needed.

Table 4: Work status in the topic area subject to JA/CA

JA/CA

Work on this topic is……

Not in our remit

In our remit but

not currently planned

Planned but not started

Ongoing or

complete

OTCA01 (Wearable cardioverter-defibrillator) 2 (6%) 23 (64%) 1 (3%) 10 (28%)

OTCA02 (Antibacterial-coated Sutures) 5 (16%) 20 (65%) 0 (0%) 6 (19%)

OTCA03 (NIPT) 2 (6%) 17 (50%) 1 (3%) 14 (41%)

OTCA04 (MammaPrint) 1 (3%) 14 (45%) 2 (6%) 14 (45%)

OTCA05 (Magnetic stimulation) 3 (9%) 21 (64%) 0 (0%) 9 (27%)

OTCA06 (TAVI) 2 (8%) 9 (36%) 3 (12%) 11 (44%)

OTCA07 (FLACS) 1 (5%) 17 (77%) 0 (0%) 4 (18%)

OTJA08 (Glucose Monitoring) 0 (0%) 10 (34%) 1 (3%) 18 (62%)

OTCA09 (HIFU) 3 (10%) 15 (52%) 1 (3%) 10 (34%)

OTCA10 (Stool DNA testing) 2 (11%) 13 (70%) 0 (0%) 4 (21%)

OTCA11 (3D Implants) 2 (10%) 14 (70%) 0 (0%) 4 (20%)

OTCA12 (CRP POCT) 2 (7%) 14 (52%) 1 (4%) 10 (37%)

OTCA14 (Robot assisted surgery) 3 (13%) 12 (52%) 1 (4%) 7 (30%)

OTCA15 (Irreversible electroporation) 2 (9%) 14 (64%) 0 (0%) 6 (27%)

OTCA16 (Stents) 1 (4%) 17 (68%) 0 (0%) 7 (28%)

OTCA17 (Lithium triborate) 2 (15%) 6 (46%) 0 (0%) 5 (38%)

OTCA18 (Regional hyperthermia) 3 (20%) 7 (47%) 1 (7%) 4 (27%)

OTCA19 (Screening for osteoporosis) 4 (21%) 12 (63%) 0 (0%) 3 (16%)

OTCA20 (EVAR / TEVAR) 2 (17%) 8 (67%) 0 (0%) 2 (17%)

OTCA22 (POCT: D Dimer and Troponin) 3 (21%) 7 (50%) 0 (0%) 4 (29%)

PTJA01 (Midostaurin) 6 (14%) 13 (31%) 3 (7%) 20 (48%)

PTJA02 (Regorafenib) 7 (17%) 16 (39%) 1 (2%) 17 (41%)

PTJA03 (Alectinib) 6 (15%) 8 (20%) 2 (5%) 24 (60%)

PTJA06 (Polotuzumab) 4 (27%) 4 (27%) 2 (13%) 5 (33%)

PTJA07 (Ustekinumab) 6 (29%) 6 (29%) 2 (10%) 7 (33%)

PTJA08 (Siponimod) 5 (33%) 3 (20%) 4 (27%) 3 (20%)

PTJA09 (Brocilizumab) 3 (30%) 3 (30%) 2 (20%) 2 (20%)


18

Gray shaded cells are the most recently published JA/CA <6 months

Use of JA/CA

Under JA3 implementation data are collected on two principal types of use:

1. Support for or as an alternative to the agency’s existing HTA procedures

2. Dissemination of JA/CA to support awareness of the availability of JA/CA and/or

evidence informed decision making

As shown in figure 12, 298 examples of use of the 27 published JA3 JA/CA have

been reported. In total there have been 89 uses of the 7 PT assessments and 209

uses of the 20 OT assessments. The majority of uses of the PT assessments (n= 60)

are in assessment procedures, with just 29 examples of sharing via dissemination. In

contrast, for OT there has been 209 examples of use, with 85 of those in

assessment procedures and 124 examples of sharing via dissemination.

For PT assessments the total number of uses ranges from 22 (PTJA03) to 6

(PTJA08). For OT assessments the total number of uses ranges from 15 (OTCA06)

to 5 (OTCA17). The examples of lowest use are in recently published JA/CA where

use is expected to increase further over time.

For PT use of JA/CA is usually part of a reimbursement and pricing process that

occurs after regulatory authorisation. For OT, a much more varied range of uses are

reported. One OT agency commented that in JA3 they had thought much more

about the different ways in which JA/CA could be used:

“we have been thinking more laterally about usage, and realising that we

don’t have to either take or leave a report; rather we can use parts of it,

and at the very least, we can disseminate it to appropriate people”

Examples of OT use include:

• use in agency HTA assessment procedures

• use to monitor the need to review an existing assessment

• use to decide if an assessment should be carried out

• use to inform data collection protocols for evidence generation activities.

• targeted dissemination to decision makers or clinical networks

2 In the count of use each agency is only counted once for each JA/CA. Agencies reporting both use and dissemination of the same JA/CA are counted once under use in assessment only.


19

• non-targeted dissemination in newsletters and on websites

Targeted dissemination to decision makers and clinical networks sometimes

included an explicit decision about whether a topic should proceed for agency

assessment, or if an existing agency assessment should be reviewed. In other

cases, the JA/CA was shared for information only.

Figure 1: Use of the JA3 JA/CA


20

Countries reporting use of JA/CA

For PT assessments 19 countries report using a JA3 JA/CA. All 19 countries report

use in assessment activities. Countries using PT assessments in their assessment

activities report using between 1-7 JA/CA. Six countries report using 1-2 JA/CA, 11

countries report using 3-4 JA/CA and 2 countries report using 5 or more.

For OT assessments 20 countries report using a JA3 JA/CA. Of those, 17 report use

in assessment activities and 3 countries report dissemination of reports only

(Germany, Romania and Sweden). Six countries report using 1-2 JA/CA in their

assessment activities, 5 countries report using 3-4 JA/CA, 3 countries report using 5-

6 JA/CA and 3 countries report using 7 or more JA/CA. The highest use is reported

by Austria.

Figure 2 below provides colour coded maps illustrating the number of uses of JA/CA

in assessment activities. The darker the shading indicates the more reported uses.

Figure 2: Number of uses of JA/CA – analysis by country

Type of use of JA/CA

Figure 3 below details the type of use of JA/CA published under JA3. Most frequently

the JA/CA is read for background information or cited in the agency assessment as

background or additional information. Agencies also frequently reported using the

JA/CA and adding local information and budget impact or cost-effectiveness

analysis. PT assessments are also frequently used to inform the evaluation of a


21

company submission. The data show that more agencies use OT assessments with

no changes or carry out translation only compared to PT assessments.

Figure 3: Type of use of JA/CA

Throughout JA3, agencies assessing OT have more frequently reported using JA/CA

to replace agency work. This observation was explored in a question added to the

implementation feedback survey that asked agencies if they used the JA/CA to

support or replace their work. The responses to this survey question are only

available for the most recently published JA/CA. Data show that JA/CA are more

likely to support agency work for PT, but that they replace agency work in a majority

of OT cases (figure 4).


22

Figure 4: Reported use to support or replace agency work (%)

Data for this question are available for the 12 most recently published OT assessments and 3 most

recently published PT assessments.

Benefits of use

The implementation feedback survey collects data about the benefits of using a

JA/CA. Initially these data were collected with free text questions. The question was

revised part way through JA3 to be a question with categorical response options

based on the free text answers given previously.

Figure 5: Reported benefits

Data are only available for the 8 most recently published JA/CA

For the 8 most recently published JA/CA for which use is reported (PTJA06 and 07,

OTCA10, OTCA22, OTCA15, OTCA17-18) the benefits identified are shown in figure


23

5. The data show that the main benefit identified was efficiency, namely that the

JA/CA saved agency staff resources or time. Agencies using OT assessments also

reported that the JA/CA improved the quality of their work, while agencies using PT

assessments reported a range of other benefits. Timeliness was reported as a

benefit when an agency was also an author of a JA/CA.

For the 3 PT assessments published at the start of JA3, the free text answers

suggested variable efficiency gains for these PT assessments. Responses ranged

from no efficiency gains because the JA/CA became an additional part of the

evidence package, to considerable savings because the JA/CA replaced the agency

assessment and only a national language summary was needed. For these 3 PT

assessments 12 agencies reported no savings or limited savings, while 6 reported

greater savings. Greater savings came from being able to use the report without any

changes, adding only a local summary and from being part of the authoring team

and therefore being able to use the knowledge gained from the JA/CA and transfer

that to the agency assessment.

For the 14 OT assessments published before the revised question was implemented

in the feedback survey free text answers again suggest variable efficiency gains.

Responses ranged from no savings and additional workload to a saving of 6 person

months. Of the 32 responses to this question 8 reported none or limited savings and

24 suggested greater savings. OT assessment users are more likely to report

savings in efficiency from using JA/CA than PT assessment users.

Non-use of JA/CA

As shown in table 5 for the 20 OT assessments published to date under JA3 there

have been 62 cases reported of an agency working on the topic but not using the

JA/CA. The highest recorded count was 11 examples for OTCA03 (NIPT). For the 7

PT assessments published under JA3 there have been 15 cases of an agency

working on a topic but not using the JA/CA. The highest recorded count was 4

examples each for PTJA01 and PTJA03.

Table 5: Non-use of JA/CA by agencies working on topic area

Worked on the topic but did not use

the JA/CA

OTCA01 (Wearable cardioverter-defibrillator) 4

OTCA02 (Antibacterial-coated Sutures) 2

OTCA03 (NIPT) 11

OTCA04 (MammaPrint) 7

OTCA05 (Magnetic stimulation) 7

OTCA06 (TAVI) 2

OTCA07 (FLACS) 1

OTJA08 (Glucose Monitoring) 7

OTCA09 (HIFU Ablation) 7


24

OTCA10 (Stool DNA Testing) 1

OTCA11 (3D Implants 1

OTCA12 (CRP POCT) 2

OTCA14 (Robotic Surgery) 2

OTCA15 (Irreversible electroporation) 0

OTCA16 (Stents) 2

OTCA17 (Lithium triborate) 3

OTCA18 (Regional hyperthermia) 0

OTCA19 (Screening for Osteoporosis) 0

OTCA20 (EVAR/TEVAR) 0

OTCA22 (Point of Care Tests D-Dimer and Troponin) 3

OT Total 62

PTJA01 (Midostaurin) 4

PTJA02 (Regorafenib) 2

PTJA03 (Alectinib) 4

PTJA06 (Polotuzumab) 2

PTJA07 (Ustekinumab) 1

PTJA08 (Siponimod) 2

PTJA09 (Brocilizumab) 0

PT Total 15

Total (PT and OT) 76

Figure 6 overleaf provides colour coded maps detailing which countries report doing

HTA on the topic area but not using the JA/CA. The data are presented by country

and the darker shading indicates higher levels of non-use. For Spain and Italy

higher levels of non-use are observed for OT because there are multiple HTA

agencies in these countries; a JA/CA may be used by one or two agencies but not

used by others depending on context and work programme.

It is underlined that non-use rarely arises because an agency has made a conscious

decision not to use a JA/CA. Reasons for non-use are explored in more detail in

sections 8-15 of this report.


25

Figure 6: Number of non-uses of assessments – analysis by country

Summary of limiting and preventing factors

Most countries report at least some factors that limit or prevent the use of JA/CA.

Two countries (Austria and Croatia) report use of PT assessments and no factors

that limit or prevent their use. Four countries (Lithuania, Denmark, Ireland and

Wales) report use of OT assessments and no factors that limit or prevent their use.

Table 6 shows that users of PT assessments report more limiting factors than users

of OT assessments. Users of OT assessments report more preventing factors than

users of PT assessments.

Table 6: Number of preventing and limiting factors reported

Limiting Preventing Total

PT 95 33 128

OT 51 77 128

Total 146 110 256

Figure 7 shows that no single factor limits use, a variety of factors are reported. In

contrast figure 8 shows that for OT challenges with timing is an overriding factor that

prevents use. Very small numbers of other preventing factors are reported.


26

Figure 7: Number of limiting factors reported by category

Figure 8: Preventing factors reported by category

The maps (figure 9) below show the number of factors reported to limit or prevent

use of JA/CA by country. In each map the darker the shading indicates that a country

reports more limiting and preventing factors. When interpreting these data, the data

will in part be related to the number of assessments that were used or not used e.g.

a country reporting more use or non-use of JA/CA also has more opportunities to

report factors that either prevent or limit the use of the JA/CA. However, the maps

show that not all countries have the same level of challenges.


27

Figure 9: Total number of limiting or preventing factors by country

Note: For OT Spain and Italy report high levels of limiting and preventing factors. This because there

are multiple HTA agencies, each reporting a small number of challenges.


28

8. Results: Comparison of use in JA2 and JA3

The results of the analysis presented in this section of the report draw primarily on

findings from the JA3 implementation feedback survey and data on use of JA2

relative effectiveness assessments3. In JA2, both full Core HTA and rapid relative

effectiveness assessments were produced, but for this analysis only JA2 rapid

relative effectiveness assessments are used as this is similar to the JA3 output.

Pharmaceutical Assessments

In figure 10 below the use of PT assessments published in JA2 and JA3 is

compared, JA2 assessments are denoted in blue and JA3 assessments are denoted

in red. Comparison is only made on use in assessment procedures (rather than total

JA3 use) because JA2 data was predominately focussed on use in assessment

procedures. For the 4 assessments (PTJA01-3 and PTJA07) published for 6 months

or longer the median number of uses of the 4 PT assessments is 14.5. This

compares with a median number of uses of 6.5 for the 6 PT assessments published

in JA2.

Figure 10: Use of PT assessments in JA2 and JA3

3 A list of agencies taking part of the JA3 is here: https://eunethta.eu/about-eunethta/eunethtanetwork/ A smaller different set of agencies took part in JA2

https://eunethta.eu/about-eunethta/eunethtanetwork/


29

Country change in use from JA2 to JA3

Data were analysed to explore changes in the number of countries reporting use of

relative effectiveness JA/CA in JA2 and JA3. The results for PT assessments are

presented in figure 11 below. In JA3 more countries are using PT assessments than

in JA2 as shown by the additional countries shaded in blue. A small number of

countries reported using JA2 assessments but not JA3 assessments. This is

primarily because of topic relevance (Netherlands) and limited and/or delayed

company submission for reimbursement (Latvia). Some countries do not report using

either JA2 or JA3 assessments, reasons for this include topic relevance (Wales,

Slovenia), lack of alignment with the timing of decision making (Switzerland) and

HTA systems still becoming established over the JA3 period (Romania, Greece).

Figure 11: Changes in use JA2 and JA3 – PT

Other Technology Assessments

Figure 12 compares the use of OT relative effectiveness JA/CA published in JA2 and

JA3. JA2 assessments are denoted in blue and JA3 assessments are denoted in

red. Again, comparison is only made on use in assessment procedures (rather than

total use) because JA2 data was predominately focussed on use in assessment


30

procedures. Excluding the 4 recently published OT assessments, the median

number of uses for the 16 JA3 OT assessments is 4. This compares with a median

number of uses of 6.5 for the 6 JA2 OT assessments. The data show more variation

in use than in JA2 (5-13 reported uses in JA2 and 1-11 reported uses in JA3).

Figure 12: Use of OT assessments in JA2 and JA3

Country change in use from JA2 to JA3

The data were analysed to explore whether the number of countries using OT

relative effectiveness JA/CA has changed between JA2 and JA3. The results are

presented in figure 13 below. As can be seen in JA3 there are an additional 5

countries using OT assessments for the first time (denoted in blue). There are still a

number of countries yet to report using OT assessments in either JA2 or JA3

(denoted in red), in some of these instances this is because the remit of OT HTA is

still very limited (Latvia, Slovenia), developed during the JA3 period (Greece),

responds specifically to company applications (Hungary) or must follow a required

procedure (Germany). One country used JA2 assessments but because of HTA

reorganisation has not reported use of JA3 assessments (Finland). It is noted that in

JA2 both REA and full Core HTA were produced. Five countries: Sweden, Estonia,


31

Romania, Switzerland and Slovenia reported using JA2 full Core HTA but not the

JA2 relative effectiveness JA/CA.

Figure 13: Changes in use JA2 and JA3 – OT

In the agency e-mail survey, we explored the reasons why agencies felt that use of

OT assessments was currently lower in JA3 than in JA2. Some agencies identified

issues with assessment timing, relevance and evidence availability that meant that

OT assessments were not prioritised for implementation.

“All EUnetHTA OT assessments have been carefully

assessed…Generally, the assessment either did not have a clear result

(certainty of evidence high or moderate), was not timely or were not

prioritized due to other available assessments. Another reason was that it

was not relevant for our context”.

It was also highlighted that although the much higher number of OT assessments

produced in JA3 was positive, limited agency capacity means that agencies can only

adapt a certain number of assessments and in JA3 have had to prioritise to a greater

degree which JA/CA were used.


32

“We do not have the resources to make use of all the produced EUnetHTA

assessments, which means that we have to prioritise in relation to the

relevance to our context”

Other agencies reflected that their low use of OT assessments arose from the fact

that their own OT HTA processes were still developing:

“The OT assessment process is under improvement in our Agency”.

In addition, it was noted that for some countries the recipient of an OT assessment

was not always very clear which affected the ability to use:

“The processes in the pharmaceutical field are more established. For

example, there has been a clear recipient of PT assessments, as opposed

to OT assessments”.

Other agencies noted environmental factors and the challenges that arise because

there is no regulatory timetable for other technologies:

“PT assessments are linked to the regulatory timetable and are on

products that are expected to be evaluated by all or most agencies….

Other technologies have their own unpredictable way and timing of

coming to the attention of decision makers”

And other agencies noted there may simply have not yet been sufficient follow-up:

“this could be due to several reasons: 1) Assessments from JA2 are done

a long time ago, so HTA agencies have an opportunity to use them longer;

2). It takes a while for a technology to come into some small countries and

into their reimbursement systems, so maybe older technologies currently

became more relevant for some parts of Europe”.

Conclusions

• For PT there has been increased production and use of JA/CA in JA3

compared to JA2

• For OT there has been increased production of JA/CA, but there is currently

less use of JA/CA in JA3 compared to JA2, reasons for low uptake of OT

assessments include:

o Limited agency capacity and increased output in JA3 means agencies

have had to prioritise which JA/CA they use

o The characteristics of access to OT across Europe means that OT can

be introduced and come to the attention of decision makers in different

countries at very different time points


33

o OT HTA processes are not fully established in some countries and the

process of linking OT HTA to decision making often less clear than for

pharmaceuticals

o Short follow up time for some JA/CA

• For both PT and OT there is an increased number of countries that have used

JA/CA in JA3 compared to JA2


34

9. Relationship between participation and implementation

In this section we explore the influence that participation in previous joint actions and

in JA3 assessment procedures has had on implementation.

A much larger number of agencies have been involved in EUnetHTA JA3 than were

involved in JA2. Summarising the possible reasons for increased uptake of PT

assessments in JA3, one agency commented that they felt that the increase in the

number of PT agencies participating in JA3 positively contributes to PT

implementation.

“Timeliness, increased awareness of the product and EUnetHTA

collaboration, relevance of the topics, higher number of published JAs,

easy access, publicity, involvement of many HTA [agencies] as partners in

EUnetHTA”.

In addition, several agencies highlighted the positive role that participation in JA/CA

processes and activities have on uptake:

“[the agency] became an active member in the Assessments (as a DR,

co-author and author) and more involved in the different EUnetHTA

working parties”

Other technologies

Figure 14: Reported use of OT REA in JA2 and JA3


35

Figure 14 above shows the reported use of JA2 OT REA and JA3 OT REA. Darker

shades indicate more reported use. The figures are in broad alignment. Countries

who reported using more JA2 REA assessments also report using most JA3 REA

assessments.

For OT, project management can be decentralised to activity centres (shown in

figure 15) who commit to producing JA/CA. For the 20 published OT assessments

most topic proposals (15 topics (75%)) have come from agencies who are activity

centres or the WP4 OT co-lead partner. Further, just under half (39 of 85 (46%)) of

reported OT uses in assessment procedures come from the agencies who are the

WP4 OT co-lead partner or activity centres.

Figure 15: Countries with OT Activity Centres in JA3

Figure 16: Countries who have proposed OT topics in JA3

The 20 published assessments were proposed by a total of 14 agencies in 10

countries (figure 16). 53 out of 85 (62%) reported uses of OT assessments come

from agencies who have proposed topics for assessment.

Figure 17 shows the countries that acted as OT JA/CA authors and reviewers. 24

authors or co-authors of OT assessments report using the assessment they


36

authored (out of 44 (55%)). It is noted that for recently published assessments some

authors may not yet have used or reported using the report, so this figure is

expected to go up in the future. Nine reviewers report using the assessment they

reviewed (out of 56 (16%)). Six agencies report being involved in an assessment (all

as reviewers) and working on the topic area and not using the JA/CA. This was

usually because the agency previously worked on the topic area and were using

their involvement in the JA/CA to understand whether new evidence that would affect

their original HTA or decision had been published. Twelve out of 85 (14%) reported

uses of OT assessments in assessment procedures were from agencies who have

not been involved in any OT assessment production (either as authors or reviewers).

Figure 17: Countries who were OT JA3 authors or reviewers

Pharmaceuticals

Figure 18 shows the reported use by countries of JA2 and JA3 PT assessments.

Darker shades indicate more reported use. For PT in JA3 there is less variation in

use of assessments – in JA2 a small number of countries made most use of

assessments, whereas in JA3 use is more evenly spread. Agencies who reported

using more JA2 assessments are not reporting most use of JA3 assessments.


37

Figure 18: Reported use of PT REA in JA2 and JA3

Figure 19: Countries who were PT JA3 authors or reviewers

Figure 19 shows the countries who acted as PT authors and reviewers for the PT

assessments for which implementation data are available. For PT seven authors or

co-authors of assessments report using the assessment they authored (out of 18

(39%)). Eleven reviewers report using the assessment they reviewed (out of 30


38

(37%)). It is noted that for recently published assessments some authors and

reviewers may not yet have received company applications for reimbursement, so

this figure is expected to go up in the future. Only 1 agency reports being involved in

an assessment (as a reviewer) and working on the topic area and not using the

JA/CA (reason given timing – JA/CA was not available at the time of agency

assessment). Where authors or reviewers have not used the JA/CA reasons for this

include:

• the topic was not subsequently prioritised for HTA

• the topic was found to be in the remit of an alternative agency in the country

• the company has not (yet) asked the agency to assess the product at the time

of the cut-off date for this analysis.

Eleven out of 60 (18%) reported uses of PT assessments in assessment procedures

come from agencies (n=4) who have not been involved in any PT assessment

production (either as authors or reviewers).

Conclusions

• Engagement of a larger number of agencies in JA3 compared with JA2 is seen to

have positively influenced awareness and uptake

• Participation in production processes is seen to support subsequent uptake.

• OT agencies who reported more use of JA2 REA assessments are also more

likely to report higher use of JA3 REA assessments. For PT reported use in JA2

appears to be less related to reported use in JA3.

• The model of topic identification and selection used in OT supports use of

assessments by activity centres, authors and co-authors, but appears less

successful at consistently identifying topics that also gather significant use from

agencies acting as reviewers and agencies outside of the assessment team.

• Involvement in PT assessments is less related to uptake than OT involvement,

this is because the PT assessment team is identified before an agency knows

whether a company will apply for reimbursement in their country and therefore

whether they will need to carry out HTA.


39

10. Results: Awareness

Fitness for purpose of procedures

Figure 20: Countries reporting awareness as a limiting or preventing factor

Table 7: Limiting and Preventing factors reported about awareness

Prevented use Limited use Total

PT OT PT OT

Not aware of the assessment being produced

1 0 2 0 3

Number of agencies reporting

The data in figure 20 and table 7 do not show issues reported about JA/CA

awareness in JA3. The challenges reported are for PT assessments and in two

cases are reported by an agency who is not a EUnetHTA partner and so has access

to fewer communication channels and alert systems than other agencies.

Agency perceptions of changes introduced in JA3

During JA3 EUnetHTA has made the following changes to its procedures to improve

partner awareness of JA/CA:

• Publication of the project plan at CHMP opinion (PT)

• Publication of the project plan as soon as it is final (OT)


40

• Alerts for JA/CA starting and publication of project plans and assessments (PT

and OT)

Most agencies interviewed were aware of these procedural changes to improve

awareness of JA/CA.

The earlier publication of the project plan was felt to be helpful for PT and OT

assessments and expected to improve use in the future. The notification systems

and publicity of assessments in JA3 was considered to have improved compared

with JA2 and generally felt to have improved awareness of JA/CA and uptake in JA3.

The recent introduction of a monthly newsletter by WP4 for PT assessments was

identified as being particularly helpful.

“Better information about the timelines for planned and ongoing

assessments”.

Agency recommendations for the future

A need for further improvements to internal communication with partners was

highlighted. An extension of the PT newsletter to OT would be welcomed. In

addition, the following areas were mentioned:

• Regular and routine updates of distribution lists (as staff turnover is high in many

agencies)

• Enhanced use of e-mails and social media (felt to be particularly important given

varying levels of use of the intranet in different agencies)

• The list of assessments on the intranet and internet should list the phase

and timescales of each assessment stage e.g. scoping, assessment.

Agencies reporting higher use of JA/CA were able to add JA/CA topics into agency

topic selection processes and were able to take topics from their agency topic

selection processes and put them into JA/CA topic selection processes.

“The basic reason [for our increased use] is that EUnetHTA topics have

been included as part of the list of topics to be prioritized by the Ministry of

Health. If they are selected, they are included as part of the Work Plan for

translation/adaption”.

“Now we are better in coordinating proposals of topics relevant to our

Commissioning Forum with selection/suggestions of topics in JA3”

This level of awareness of JA/CA with commissioners and decision makers outside

of partner organisations is still being put in place in some countries.


41

“in the future, [agency] aims to notify health insurance companies about

the EUnetHTA reports if we do not have the topic on our internal agenda”

“OT assessments could also be relevant for the regions. We are in contact

with the regions in order to increase awareness of the EUnetHTA reports”.

While some awareness raising activities must be undertaken at an agency or country

level, it was also suggested that EUnetHTA could do more to raise awareness in

other organisations and networks, for example support for JA/CA could be improved

by engaging more with Cochrane Groups and the Guidelines International Network

to develop strategic relations and build relationships:

“what I am missing in this discussion is the possibility to strengthen the

support for EUnetHTA products and processes by reaching out to two

particular communities: Cochrane and Guidelines International Network,

both doing assessments often on the very same topic”.

There was agreement that at an agency level inclusion of the reuse of JA/CA in

agency SOPs and procedures is imperative if joint work is to be properly embedded.

“Key first area is to have procedures in place to support

inclusion…important to embed this infrastructure”.

“It is important that this is noted down in national procedures, so that it is

not only dependent on EUnetHTA “ambassadors”

“We have developed a standardised way of using the assessments as

well, which has also made it easier to use the assessments”.

Conclusions

• A lack of awareness of JA/CA among HTA users is not a key factor preventing or

limiting use of JA/CA.

• The following changes to notification systems have been positively received:

o More alerts for planned, ongoing and published assessments

o Earlier publication of OT and PT project plans

• Agencies recommend:

o The implementation of a robust information and communication system

o Internal communication mechanisms that use a wide variety of channels

without relying on any single channel

o Contacts lists require regular updating because of staff turnover


42

o Extension of the PT newsletter to OT

o Greater dissemination to and collaboration with people, groups and

organisations who are not part of the network or not directly involved in the

production work

o Formal inclusion in agency procedures of an alert to check for ongoing or

published relevant joint work to support improved awareness of JA/CA

across agency staff not directly connected to EUnetHTA

o Agency communication strategies to ensure relevant HTA users and

commissioners in a country are aware of JA/CA


43

11. Results: Timing


Figure 21: Countries reporting timing as a limiting or preventing factor

Table 8: Limiting and Preventing factors reported about timing


PT OT PT OT

Assessment not available 7 54 12 7 80

Notice too short 0 3 2 2 7

Not up to date 0 1 1 2 4

Total 7 58 15 11 91 Number of agencies reporting

The implementation data (figure 21 and table 8) suggest that for some countries PT

assessment timing remains an issue. This is usually because agency assessment

processes can start at or before CHMP opinion and so an agency may have already

started work when the PT assessment is published. In the interviews one of the

agencies highlighted that delayed launch in some countries means that if a PT

assessment is carried out too early then it may no longer be up to date when a

company submits its application for reimbursement. However, in only 1 instance was

this given as a factor limiting use of a PT assessment.

For OT assessments the implementation data show that timing is, by some margin,

the biggest factor reported to prevent the use of JA/CA.


44

It was noted that in countries with later adoption of an OT having a JA/CA is of value

and can be used even if it needs updating, although opposite views were expressed

as to how resource intensive updating is:

“when the topic actually crops up at the national level, a OT assessment is

extremely useful, even though it has become out of date, because update

is generally quite quick and a lot of background work has already been

done, saving time and resources”

“if use EUnetHTA assessment and it is 6 months old you then have to

update on searches etc.so whilst you cut down work you won’t eliminate it

or replace work.”

However, the key barrier for OT timing was when agency work had already taken

place before the JA/CA became available, in some cases some years before

EUnetHTA produced their assessment.

“Some reports were in principle useable but too late”.


The main procedural change made by EUnetHTA in respect of timeliness for PT

assessments were changes made at the start of JA3 allowing the publication of PT

assessments closer to marketing authorisation.

All agencies interviewed viewed changes to PT timing as either positive or neutral in

supporting them to use PT assessments. Among respondents the main reason for

increased use of JA3 assessments compared to JA2 assessments was identified to

be improved timing, assessments often came too late in JA2.

“Timing – all reports are published close to EPAR which is in line with our

national procedure”.

“The key reason for the change [in use of assessments] is timing. PT

EUnetHTA assessments in JA3 have been available at the time [agency]

receive a submission”.

The main procedural change made by EUnetHTA in respect of timelines for OT

assessments was the sharing of draft assessments. In September 2019 it was

approved that draft assessments would be shared with partners outside of the

assessment teams, with this to become operational from OTCA23 onwards.

A number of agencies interviewed were not aware of this change and because the

change has yet to be implemented agencies were unable to comment definitively on

the impact of the change, but many felt that it will be positive and will help agencies

to plan to use OT assessments.


45

“It will be really helpful to support early adaptation work and plan at an

earlier point to plan and determine whether they will use the assessment

nationally”.


To further improve the timeliness of PT assessments it was suggested that agencies

could benefit by having the PICO made available to them earlier, rather than an at

CHMP opinion as currently. It was suggested that it may be possible to achieve this

through making the PICO available internally to partners on the intranet and not

publicly on the internet.

Whilst the findings from the interviews confirmed OT timing as a key barrier to use,

partners identified no obvious solutions to solve the challenge of timing. It is,

however, hoped and anticipated that going forward the new medical devices

regulations in Europe will help to provide an anchor to ensure less spread out and

variable uptake across Europe for OT.

Conclusions

• Earlier publication of PT assessments close to marketing authorisation has

supported increased use of these assessments. However, for agencies who can

start work at CHMP opinion, timing of the joint PT assessment remains a limiting

factor in the ability to use EUnetHTA assessments.

• There are ongoing challenges to identify the best possible timing to carry out OT

assessments so they are of value to the largest number of agencies. It is hoped

that EU MD regulations will help create an anchor that will allow for greater

alignment of timing of OT HTA across Europe.

• Early sharing of information about PICO and draft assessments supports better

planning and use of JA/CA. The improvements in JA3 are welcomed, but

partners suggest it would be helpful if more PICO information can be shared

earlier.


46

12. Results: Relevance and transferability


Figure 22: Countries reporting relevance as a limiting or preventing factor

Table 9: Limiting and Preventing factors reported about relevance


PT OT PT OT

Scope Relevance 4 7 7 11 29

Content Relevance 0 1 14 7 22

Transferability 0 0 1 2 3


Respondents to the implementation feedback survey are asked if the following factors

either prevented or limited use of JA/CA.

• Differences in the scope of the JA/CA and agency assessment (“scope

relevance”)

• Differences in the content of the JA/CA and agency assessment (“content

relevance”)

Implementation data (figure 22 and table 9) show that relevance remains a key issue

limiting and preventing use of both PT and OT assessments.

The data show that a lack of alignment of the scope more often prevents use, but a

lack of alignment of content does not. For PT assessments fewer agencies report


47

issues with alignment of scope and more report issues with requiring different

content information. For OT assessments more users report issues with scope

relevance and fewer report issues with content alignment.

Issues of content alignment in PT arise in part from the use of economic information

in agency assessments and need to analyse clinical data to inform and interpret

economic analyses. In contrast, OT assessments have more heterogenous

objectives and PICOs which create more challenges with creating a European

scope. Giving reasons for non-use of OT assessments one agency commented:

“In some other cases the PICO was not in line according to ours (different

place of OT in the therapy, which we scope at the professional societies)”

The challenge of adaptation when there was a lack of alignment of scope was

illustrated by one agency:

“for those reports that were chosen to be adapted (which means not only

translated in their summaries and published on our websites) we usually

found that the PICOD did not fit our context (relevance) and in some

cases we needed to basically re-run literature searches, re-make

systematic reviews and rewrite CUR and TEC chapters”.


Several procedural changes have been implemented by EUnetHTA to improve the

relevance of PT assessments, including:

• Formal request to partners for interest in topic

• EUnetHTA prioritisation list (EPL) for topic selection

• PICO Survey

• Early access to confidential information in the draft regulatory assessment

report allowing citation of the CHMP assessment report in PT assessments

Most agencies were aware of these changes. Agencies identified PICO surveys as

being important in improving relevance and supporting engagement of agencies

early in the assessment process.

“PICO survey is very useful, and we always participate in this”.

“The PICO surveys were able to involve our Agency from the beginning of

the process”

However, not all agencies found taking part in the PICO survey easy:


48

“PICO surveys are difficult to answer as often the populations are more

limited / focussed to limit budget impact”.

“But difficult to input to the PICO at the stage it is undertaken – but have

tried to do so to ensure that the report is of relevance when it comes

through”.

The EUnetHTA prioritisation list (EPL) was also identified by PT users as being

helpful in improving relevance for some agencies. However, several agencies

commented that the prioritisation had little impact nationally either because they

assess all new medicines in that country or because the identification and selection

of topics is industry led.

For OT, the key procedural change implemented to improve the relevance of JA/CA

was the EUnetHTA Prioritisation List (EPL) for topic selection. This was published on

2nd July 2019 for OT.

Reactions to the OT EPL were mixed, some agencies considered that it had been

helpful for their own topic selection processes, but for the most part it was not

considered to have been very useful for OT assessments.

“Would be good to have a shorter more focussed prioritisation list that

gives more detail such as the PICO”.

“An administrative burden but of limited use for OT. As in most instances -

authors will only act as authors if they have been asked to review the topic

at a national level”.

“Helpful document and we shared it very widely but feedback now is that a

topic is prioritised but then it does not start, and we get questions as to

why”

Some agencies considered that a greater use of horizon scanning would help

better alignment of topics for JA/CA with agency priorities, while other agencies

proposed sharing lists of agency priorities before work had been started.


For PT it was identified that the PICO survey has been particularly helpful in

improving relevance.

“Keep emphasis on PICO development (PICO survey and adherence to

PICO in report) to support relevance”

It was suggested that further work on the PICO would be helpful to establish who

defines the PICO in different partner countries and that the PICO survey should be


49

specifically targeted at these people to ensure the best and most informed feedback

is obtained from the PICO survey.

It was also suggested that a teleconference would be helpful after the PICO survey,

in order to allow the authors of the assessment to respond to any questions on how

survey comments have been addressed in the final PICO.

Reflecting on the challenges of creating a JA/CA to suit multiple agency

requirements, it was noted that it was unlikely that JA/CA could have a scope that

would suit everyone, therefore agencies will need to have sufficient procedural

flexibility to extract information relevant to their context. However, EUnetHTA should

ensure that authors adopt a European perspective.

Do not limit content to have a report that suits every country 100%.

Rather, allow for a report with more PICO’s or outcomes if necessary, so

that every country can pick what they need. (NB: authoring teams should

author with this European perspective in mind.)

Further suggestions to improve relevance for OT assessments focussed on the need

for better scoping, particularly at the pre-scoping stage to ensure that topics coming

through are well defined. It was suggested that the PICO survey should be extended

to cover OT assessments. This is currently being considered by WP4 and a pilot

PICO survey for OT assessments is being undertaken.

Conclusions

• Relevance is a factor that prevents or limits the use of JA/CA. However, the

challenge for OT and PT differs, for OT alignment of scope is a bigger challenge,

whereas for PT alignment of content is the bigger challenge.

• The PICO survey introduced for PT assessments is perceived to have had a

positive impact on assessment relevance and implementation.

• The EPL lists are perceived to have been less useful for implementation. For OT

there remains a need to work further on topic identification and alignment of

topics for JA/CA with agency priorities.

• Agencies need to ensure there is sufficient procedural flexibility to allow them to

extract relevant information from JA/CA and ignore irrelevant elements.

• Authoring teams should adopt a European approach to authoring JA/CA


50

13. Results: Reliability and scientific rigour


Figure 23: Countries reporting reliability as a limiting or preventing factor

Table 10: Limiting and Preventing factors reported about reliability


PT OT PT OT

Not sufficient quality 3 2 0 0 5

Disagreed with findings

2 0 2 0 4

Disagreed with quality assessment

1 0 0 0 1

Needed more methodological information

1 0 2 0 3


Respondents to the implementation feedback survey are asked if the following factors

either prevented or limited use of the JA/CA.

• The JA/CA was not of sufficient quality

• The agency disagreed with the findings of the JA/CA

• The agency disagreed with the assessment of quality in the JA/CA


51

• The agency needed more methodological information to make a judgement

about quality

The data (figure 23 and table 10) show that factors relating to reliability are more

apparent for PT assessments than for OT assessments, and most often prevent use.

No single reliability factor is identified as a bigger issue than the others.

In general, the data show that the scientific rigour of JA/CA was felt to be fit for

purpose, though one agency commented that quality of OT assessments was

variable. In addition, quality remains an important factor limiting use of assessments

in a small number of locations:

“agency has not used the EUnetHTA PT assessment reports….as they

did not meet the requirements of transparency, standardization,

completeness, independence and methodological quality which are

prerequisites for the use in our country”.


The procedural changes made to improve scientific rigour were common across PT

and OT assessments and were:

• Implementation of the Companion Guide

• Submission requirements documents

• Optional factual accuracy checks

In JA3 EUnetHTA has implemented a quality management system through the

Companion Guide to support production of JA/CA. The Companion Guide was

largely viewed positively for both PT and OT, but agencies were equivocal about

whether it has or will improve implementation of assessments. One agency viewed

the benefits mainly to be in providing more confidence in assessments while another

commented that the decision makers did not have that level of knowledge and so

would not really consider it in detail when choosing to use an assessment

“Absence of quality management system was not stopping the agency

using EUnetHTA assessments previously…although the presence may

give users more confidence”.

“Not sure if changes will improve uptake as our commissioner is not at that

level of knowledge of procedures when undertaking HTA and considering

use of EUnetHTA assessments”.

The introduction of a submission requirements document was again viewed

positively by the agencies that were aware of it, although not all were aware that it

has been introduced. For OT it was recognised that some companies, particularly


52

smaller companies, do not always fill in the submission file. There was, therefore,

uncertainty about how beneficial it would be in this context.

The optional factual accuracy check by manufacturers on draft documents was

welcomed by some, with this being an important additional step to allow companies

to check and if necessary, get corrected assessment errors. Some agencies were,

however, not in favour of company involvement in this stage.

“Factual accuracy check by manufacturers is of great interest and should

be mandatory, to increase scientific rigour and transparency”.

“Addition of factual accuracy checks is particularly helpful for complex

devices or where there are issues around the CE mark”.


Several agencies underlined that there was a need to ensure that authoring teams

had sufficient methodological expertise to undertake JA/CA through internal training

and internal or external expert support.

In addition, it was noted that authoring teams need to have sufficient time allocated

to undertake JA/CA using the procedures that have been defined.

Finally, it was noted that while the implementation of the quality management system

was useful, it has focussed a lot on procedures and there was still a need to focus on

methods.

Conclusions

• Reliability of JA/CA is not reported as a major factor preventing or limiting use

• The implementation of the Companion Guide and submissions requirements

documents give greater confidence in the JA/CA, but mostly have not

impacted on whether agencies use them


o There is a need to ensure that there is sufficient methodological

expertise within authoring teams through internal capacity building

and/or external methodological expertise

o There needs to be alignment of the procedure followed and time

allowed for authoring. Robustness of JA/CA is reduced if authors are

not given sufficient time to follow the defined procedure

o A factual accuracy process to correct errors in JA/CA before

publication can help to give confidence in the findings.


53

14. Results: Evidence and Methodology


Figure 24: Countries reporting evidence and methodology as a limiting or preventing factor

Table 11: Limiting and Preventing factors reported about evidence and methodology


PT OT PT OT

Evidence too restrictive

1 1 2 5 9

Evidence too wide 1 0 0 2 3

Out with agency approach

3 1 2 1 7


Agencies are asked in the implementation feedback survey if the following factors

either prevented or limited use of the JA/CA.

• Whether the type of evidence included in the JA/CA were too restrictive, for

example inclusion of randomised data only

• Whether the type of evidence included in the JA/CA was too wide, for

example inclusion of all available evidence


54

• Extent to which the evidence and methodology used aligned with that used by

the agency.

The data (figure 24 and table 11) show that for OT in the majority of cases issues

arose from the choice of evidence included in the assessment rather than that the

approach to methodology was out with that adopted by the agency. For PT, more

agencies comment on evidence and methodology being out with the agency

approach than the choice of evidence inclusion. However, the numbers in all

categories are small.

In regard to evidence inclusion, among OT users a number of interviewees noted

that they or their decision makers had reviewed the JA/CA and decided that an

agency assessment was not timely based on the evidence availability in the report.

“We shared the report with the relevant managed clinical network. They

scanned it and considered that insufficient evidence to indicate action at

this time”.

It is positive that agencies are using JA/CA to support topic identification and

decisions about agency assessments. However, it may not be the best possible use

of joint HTA resources if outputs of the process primarily inform a decision that an

agency assessment is not timely because of evidence availability. At the stage of

determining topics for joint assessment and scoping these, it may be appropriate to

include consideration of evidence availability and the types of evidence to be

included in the JA/CA. This might help target joint HTA resources on topics most

likely to result in agency assessments.

Survey data shows that the methodology used for JA/CA is not a key limiting factor.

This is consistent with the findings from the WP7 case study4 that identified that the

EUnetHTA guidelines do not hinder uptake.


In JA3 EUnetHTA developed recommendations and guidance for involving European

patient organisations.

Most agencies were aware of the procedural changes and viewed this change as

positive. The involvement of patients in JA/CA was seen most positively by agencies

who engaged patients less in their agency process compared with agencies who

already had methods of patient engagement. Several agencies are currently

developing their own procedures for involving patients and learning from EUnetHTA

on improving patient involvement was felt to be helpful. A number of agencies

reported that whilst the improved patient involvement in JA/CA would not specifically

4 https://eunethta.eu/wp-content/uploads/2019/06/Implementation-Report-May-2019-FINAL.pdf


55

impact on their ability to use the JA/CA, the involvement of patients does improve

their confidence in the findings of the JA/CA.

“It doesn’t impact the ability to use it but the ability to understand and trust

its conclusions”.

One agency also felt that the evidence and methodology focus in JA3 had

become more pragmatic with less focus on specific terminology which helped

them engage better in the process.

“JA2 seemed to very focussed on the core model and all the terminology

around this, and we struggled to get our heads round a lot of what was

being discussed and proposed. We have felt much more engaged in JA3,

and this has made it much easier to get a handle on what is going on and

get involved”


Some agencies indicated that the role of patients in scoping needs to be extended.

In addition, having developed the procedures for involvement, there is a need to

ensure that there is follow through from the procedures to engagement.

“Development of procedures for patient involvement is a good step but

often patient organisations do not engage and hence often it does not

improve the assessment”.

In addition, patients are only one group of experts and there is a need for

engagement of clinical experts and potentially also payers.

“Patient participation has improved a lot through changes. Contribution of

clinical experts has not improved much in PT and needs to be improved”.

“Would be good to have more involvement from a payer perspective –

particularly in terms of their input into the [OT] project plan”.

Considering other aspects of evidence and methodology, it was felt that there should

be an increased focus on methodology, particularly in respect of updating existing

EUnetHTA methodological guidelines and developing new methodological guidelines

in emerging areas such as personalised health and precision medicine.

Partners underlined the ongoing need to develop mutual acceptance of certain

methodologies (use of GRADE was highlighted by several agencies).

“It is important that EUnetHTA supported methods are accepted by the

partner agencies and there is mutual agreement in the use of certain

methodologies. The piece of work done by the Common phrases and


56

GRADE task group, as well as the PICO subgroup, is very important in

this matter”.

Several agencies identified that once these methods and processes have been agreed

by EUnetHTA, these then needed to be fed back to staff involved in agency

assessment through training and knowledge transfer. One agency stated:

“It is very important that all agency staff involved in the national HTA is

well trained in utilising the joint HTA tools and assessments. Also, it is

essential that there is good communication and knowledge transfer

between staff experienced in the European joint work and national HTA”

Conclusions

• The methodological approach adopted by EUnetHTA is not a key factor limiting

or preventing use of JA/CA

• The decisions made about the study types to be included in an OT assessment is

more frequently reported as a limiting factor than methodological approach.

Consideration of evidence availability and evidence inclusion in OT assessments

should be considered as part of topic identification and scoping.

• The recommendations and guidance for involving patients developed during JA3

has been positively welcomed.


o Methodological guidance is regularly maintained and extended to cover

new and complex issues arising in HTA

o The methodological approach adopted by EUnetHTA needs to be

consistently applied across all JA/CA

o Agencies need to identify and resolve any methodological areas where

agency adopted approaches actively hinders the use of JA/CA

o Agencies need to ensure that methodologies and processes agreed

through EUnetHTA are communicated to agency assessors through

training and knowledge transfer

o Principles of patient engagement need to be followed up and turned into

actual engagement

o Principles of engagement need to be developed for other expert groups

e.g. clinical experts and payers


57

15. Results: Usability and Reporting Structure


Figure 25: Countries reporting report structure as a limiting or preventing factor

Table 12: Limiting and Preventing factors reported about report structure


PT OT PT OT

Agency must use a specified structure

1 4 18 2 25

Number of agencies reporting

For PT assessments many countries report having to use a specified report structure

for their agency assessments. This acts as a factor that limits (but not prevents) their

use of JA/CA (figure 25 and table 12). Specification of a report structure appears to

be much less of a limiting factor for OT assessments.

Usability is related to familiarity with the JA/CA report structure. It is expected that at

the beginning of the introduction of a new template, agencies will experience

challenges because of a lack of familiarity with the reporting structure and

experience of using it. These challenges should reduce over time as more reports

are published and familiarity with the template increases.


In JA3 the PT assessment report template has been revised.


58

There was limited feedback available on changes to the PT template given the

relatively small number of PT assessments published in the last 12 months. Those

that have used the new template were largely positive reporting that it is easier to

use, simpler and less repetitive. There were also favourable comments on the

availability and use of medical editors.

“The new template is now more simplified and less repetitive”.

“Input of medical editor was very useful….and we suggest keeping this for

future assessments”.

In JA3 the OT template has not yet been revised (a pilot for the revised

template started after the implementation data cut-off), but improved

standardisation of the process and a small number of changes to the

presentation, summary and discussion have positively influenced the readability

of the JA/CA.

“The assessments have become more standardised and the format more

familiar and therefore it is easier for us to use the assessments in JA3”.

“More standardised structure of the assessments”


Some agencies felt that further instruction is still required for authors on how to use

the new template

“We are currently an author of an assessment and would be helpful to have

more information on the requirements / expectations of authors in each

section”.

An outstanding issue for PT assessments is about the length of the assessments.

From interviews, there is, however, no clear sense of the best way forward. Some

agencies require detailed JA/CA and others want shorter more concise documents.

Some agencies want specific headings included and covered in the JA/CA and

others do not.

“The structure of the reports has improved, but they are still too

excessive”

For OT assessments one agency commented that there is still a need for

further improvements in clarity and readability.

“In order to make OT assessments more useful, it is important that they

have a clear conclusion on the value of the product in relation to the

relevant comparator/s. Several conclusions of EUnetHTA reports do not


59

help the reader to know the answer to the relative effectiveness question

due to unclear language”

Conclusions

• The requirements for agencies to use a specific report structure acts as a limiting

factor in use of JA/CA but rarely prevents use

• The new template for PT assessments has been positively received from those

who have used it

• Changes to the OT process and influence of these on the report structure have

also been positively received

• The heterogenous needs of agencies in their assessments means that defining a

report structure that meets all agencies’ needs completely is unlikely to ever be

fully realised. A balance will be required between different agency needs and

time and resources available for authors.

• To accurately capture usability issues, the introduction of new templates needs to

allow sufficient follow-up time for agencies to become familiar with it, before it is

further revised


o Further guidance is provided for authors about using the template and

information expectations

o Active brokering of an agreement about the length of JA/CA to meet most

needs of most agencies

o Agreement on the conclusions that can be made in JA/CA

o Editing of JA/CA to ensure language is clear


60

16. Results: Language


Figure 26: Countries reporting language as a limiting or preventing factor

Table 13: Limiting and Preventing factors reported about language


PT OT PT OT

Agency assessments must be written in national language

1 3 22 7 33

Documents received must be in national language

0 0 1 1 2


The implementation feedback survey collects data about whether the language that

the agency must use is a factor that prevents, or limits use of JA/CA.

The data show (figure 26 and table 13) that agencies can be required to write their

assessments in a national language and that this can act as a limiting factor

particularly for PT assessment users. However, it does not prevent use.

Very few agencies report that the requirement to receive documents in a national

language limits their use of JA/CA and none report that it prevents use of JA/CA.


61


Not applicable


There was quite a strong view that language and specifically the use of English in

agency assessments could be a tool to improve use of JA/CA. This was suggested

as one of the few areas where potential legal changes could be made.

Agencies with the freedom to prepare reports in English commented on how this

made use easier.

“Agency can publish reports in English or even partially in English which

makes it more convenient to re-use the EUnetHTA report without the need

to translate the information”

Conclusions

• Many agencies must prepare their assessments in their national language. This

acts as a factor that limits use, because text needs to be translated and cannot

be directly used. However, it does not prevent use of JA/CA.

• There is a perception that a change to allow technical documents to be written in

English is required to support implementation, this change is seen as possible by

agencies but to make this change requires legal changes for some countries


62

17. Results: Transparency and Independence


Figure 27: Countries reporting transparency as a limiting or preventing factor

Table 14: Limiting and Preventing factors reported about transparency


PT OT PT OT

Transparency of process

2 0 1 1 4

Access to documents informing assessment

3 0 4 0 7

Total 5 0 5 1 11

Transparency of process and availability of documents are not reported to be a

major factor limiting or preventing use (see figure 27 and table 14).

Where transparency is reported to be an issue it is mainly for PT assessments and

relates to the availability of documents supporting the JA/CA procedure. Company

submission documents are now published, and it is expected that this will be less of

an issue for future PT assessments.


63


The main key change made to improve transparency has been the publication of

submission dossiers for PT assessments. This change has also been implemented

for OT submission dossiers but was not implemented at the time of data cut off.

The key procedural changes made to improve independence were common across

PT and OT assessments and were:

• Publication and communication of DOI form and guidance

• Establishment of the EUnetHTA COI committee

Agencies either felt that publication of PT submission dossiers was a helpful change,

or they viewed this as a neutral change. Some agencies did not need to see the

submission dossier and others did not consider there was enough time in their

agency process to review both the JA/CA and the submission dossier.

“A positive change and good that the REA actually cross-refers to the

submission dossier. Really pleased as lack of submission dossier was a

big issue in JA2”.

Not sure it is a big or particularly helpful change as would probably just

use the EUnetHTA report rather than also check the submission dossier.

All the agencies interviewed were aware of the changes made to declaring COI.

There was a clear consensus from the interviewees that having these procedures in

place give users more confidence and trust in JA/CA, though not necessarily

changing how they will use the JA/CA.

“It increases trust in the assessment”. “They are again positive changes. They create a more transparent process...but there are bigger issues that impact on our ability to use EUnetHTA REAs at a national level, in particular relevance”


Regarding publication of the data behind the assessments one agency commented

that publication of company submission dossiers could be extended to also included

clinical study reports.

In regard to COI, Some felt that the COI policy may need to be more flexible in some

instances to ensure appropriate access to clinical expertise and that the length of the

document put people off from participating in assessments.

“The COI procedure was actually too restrictive. Compared to the

[national] COI procedure, an expert can declare an interest but normally


64

this expert wouldn’t be excluded as reviewer but flagged or skipped on

those matters at which his interest might affect his judgement”.

“we are told that the DOICU form can be quite long and some people drop

out rather than complete it. So, a recommendation would be to simplify

the DOICU”.

Conclusions

• The principles of transparency and independence adopted by EUnetHTA are

not key factors limiting or preventing use of JA/CA

• The publication of company submission files is viewed as having a positive

impact on PT implementation

• Changes made to the COI policy adds confidence to use of JA/CA


o Ongoing monitoring of the COI policy to ensure that the stringent

measures in place are not acting as a barrier to getting the best

possible expert involvement


65

18. Recommendations for implementation support in a future model of HTA cooperation

The fifth and final key question that we answer in this report is:

“What support for implementation should be built into a future model of HTA

cooperation?”

In JA3 implementation support has mainly focused on raising awareness of JA/CA,

encouraging use, sharing implementation experiences, putting in place a reporting

system to monitor use, feedback to EUnetHTA to support improvements in

processes and reaching out to agencies who are not part of the network but are

potential users of reports. Because the details and requirements of a future model of

HTA cooperation are not yet known JA3 implementation has not worked towards a

set of specific implementation requirements for using JA/CA.

Most agencies agreed that support for implementation should be built into a future

model of HTA cooperation. However, there was a range of different views and

opinions on what types of implementation support should be available.

Agencies considered that the level of implementation support required will differ by

agency and country, and will be dependent on:

• how developed HTA systems are in individual countries

• the levels of HTA experience and expertise within these countries

• whether HTA cooperation has a legislative basis in the future.

Agencies with less developed systems or limited HTA experience will require more

implementation support. Likewise, if there is a legislative basis more implementation

support will be required to support the changes needed.

One respondent felt that to really strengthen HTA in Europe capacity building is

required, specifically to support HTA in smaller and less developed countries. This

was echoed by a respondent from a country where HTA is still establishing who felt

that the HTA regulation would serve as a lever for capacity building and that capacity

development should be built into the future model.

“The needs of member states for implementation support will give a broad

spectrum of responses, depending on how developed their HTA systems

are and how recently HTA activities have started….a key priority is

capacity building and organisational development particularly for smaller

countries…the potential that is not yet tapped for smaller agencies could

be bigger if we do some capacity building to take these agencies forward”.


66

Another respondent felt that implementation support could best be provided

centrally:

“A central support office would be helpful, providing a central contract

point for different support…. for example, training, background information

on assessments and procedures and translation support if needed by

some partners”.

In terms of specific types of implementation support that could be offered in the

future a range of options were discussed and proposed.

Monitoring, audit and indicators

Audit tools and indicators can help agencies measure whether they are

implementing outputs appropriately. Some agencies considered the use of audit and

indicators as important as they provide guidance on what is acceptable use.

However, others noted that these can be intrusive and unpopular.

“For PT audit and implementation and workshops indicators most

important”.

“monitoring and audit may be seen as intrusive”.

Training and workshops

It was noted that some training on how to update, adapt and adopt JA/CA might be

needed.

“Some training and update on how to adapt/adopt HTA assessments

might prove useful for implementation purposes”

However, it was also noted that face-to-face approaches do not always work well

because agencies use reports over different time periods. Having online resources

enables agencies to use them when they need them and for more staff to take part.

Online resources and repositories

Sharing implementation experiences through webinars and case studies was viewed

as being helpful because agencies can see the challenges other agencies face and

how they overcame them.

“The sharing of implementation experiences and webinars and case studies are

helpful as enables you to see the difficulties agencies went through and how they

worked through them”

It was also noted that assessment specific webinars where authors shared the key

results and issues in assessments could be helpful for agencies to understand an

assessment and its importance.


67

“Webinars are useful. Important that we are aware what is out there and

available for use – so good mechanisms for sharing information on

assessments”

Outreach and field teams

In certain situations (such as where HTA systems were still establishing) dedicated

advisers to support implementation, to engage agencies to discuss and to resolve

issues and to promote outputs might be needed.

“For OT as still in early stages of development nationally workshops and

implementation advisers would be most helpful”

Conclusions

• Implementation support is required in the future model of HTA cooperation

• Support should be centralised and provided in a range of different formats to

fully meet the needs of a range of different HTA agencies across Europe with

different levels of HTA infrastructure, expertise and experience.

• A clear framework needs to be developed and agreed upon, so users of

JA/CA understand the expectations of use.

• Acting alongside the framework there needs to be appropriate organisational

development for agencies to ensure a sustainable model of HTA cooperation.

• In the first instance those agencies still establishing HTA systems require

early facilitative support to put in place the capacity, expertise and processes

to implement the framework.

• Suggested modalities for support:

o Audit and indicators to support agencies to monitor acceptable

implementation

o Online webinars, case studies and training about JA/CA and

implementation experiences

o Outreach and field teams to support developing HTA systems


68

Appendix 1: Implementation feedback survey

The same set of questions are used for all JA/CA.

The Intranet survey function automatically records the person who completed the

entry, the date of completion and if the survey was fully completed


69

Number Question Response options

Responses Sharepoint branching logic5

1 Please provide your agency name Free text -

26 Have you used the EUnetHTA assessment? (tick the most applicable, use the free text box at the end of the survey to provide further explanation if needed)

Radio Buttons Choose 1

Yes, to support our usual assessment procedures Directed to question 6

Yes, as an alternative to our usual assessment procedures

Directed to question 6

Joint HTA

Yes, other Directed to question 3

No Directed to question 4

3 Please describe Free text - Directed to question 6

4 Please mark the response that best describes your situation


This topic area is out of agency remit Directed to question 17

Work on this topic is not currently planned, but the topic area is in agency remit


Work on this topic is planned but not yet started Directed to question 5

We carried out work on this topic but did not use the EUnetHTA Assessment


5 Might the topic be considered in the future? If so, what is the planned completion date?

Free text - Directed to question 17

6 When you used the EUnetHTA assessment, which parts did you use?


Used methods and/or evidence from specific sections of the assessment in your report


Used information from all sections of the assessment in your report


5 If there is no branching logic given in this column, then the questionnaire proceeds to the next question in the numbered sequence 6 Question amended part way through joint action, answers not available for all assessments


70

7 Which sections of the EUnetHTA assessment did you use?

Multiple selection

Health condition and use of the technology

Description of the technology

Clinical effectiveness

Safety

Checklist for ethical, social, organisational and legal implications

8 When you used the EUnetHTA assessment, what did you do?

Multiple selection

01. Read the assessment for background information

02. Cited in your report as background or additional information

03. Used the EUnetHTA literature searches

04. Used the assessment to inform the evaluation or consideration of a company submission of evidence

05. Made significant changes to information used

06. Made minor changes to information used

07. Carried out translation only (please specify if translation was to the main text or the summary)

08. Made no changes to the information used

09. Updated evidence

10. Added local information

11. Added budget impact or cost effectiveness analysis

12. Added information about organisational, ethical and/or legal aspects

Free text 13. Other

9 Was the main text of your report produced in your national language?


Yes

No


71

10 At which level was your report aimed?


Local

Regional

National

117 What were the benefits of using the EUnetHTA assessment?

Multiple selection

01. Efficiency – Using the EUnetHTA assessment reduced the amount of time and / or resources required to carry out agency assessment work.

02. Quality – Using the EUnetHTA assessment improved the quality of the agency work

03. Knowledge – Using the EUnetHTA assessment provided access to expertise and/or data that the agency would not otherwise have had

04. Credibility - Using the EUnetHTA assessment gave additional credibility / validity to agency work.

05. Timeliness – Use of the EUnetHTA assessment improved the timeliness of decision making (e.g. led to an earlier decision about use of a health technology)

Free text 06. Other (please specify)

12 Please provide additional detail on any of the benefits you have identified of using the EUnetHTA assessment (e.g. the amount of resource / cost saved in days, WTE or money)?

Free text -

13 Did your report directly inform a decision?

Choose 1 Yes, a reimbursement decision

7 Question amended part way through joint action, answers not available for all assessments


72

Yes, another type of decision, please describe the type of decision

Yes, policy strategy or clinical guideline

No, please describe the aims and audience for the report

14 What was the outcome of the reimbursement procedure?

Radio buttons Choose 1

Positive decision

Negative decision

Positive with restrictions

15 Please provide a web link to any publicly available documents

Free text -

16 Which (if any) of the following factors prevented or affected your ability to use the EUnetHTA assessment?

Multiple selection

01. No limiting factors identified

02. Awareness: We did not know the EUnetHTA assessment was available

03. Timing: The EUnetHTA assessment was not available

04. Timing: Notice of an assessment was too short to adjust work planning to allow (best possible) use

05. Timing: The EUnetHTA assessment was not up to date

Free text 06. Timing: Other

07. Evidence and methodology: The range of evidence included was too restrictive

08. Evidence and methodology: The range of evidence included was too wide

09. Evidence and methodology: The methodology used was out with the approach adopted by the agency

Free text 10. Evidence and methodology: Other


73

11. Reliability: The EUnetHTA assessment was not of sufficient quality

12. Reliability: We disagreed with the findings

13. Reliability: We disagreed with the quality assessment.

14. Reliability: We needed more methodological information to assess the reliability

Free text 15. Reliability: Other

16. Relevance: The scope of the national assessment was different (e.g. in terms of comparator, population etc)

17. Relevance: We needed different content information (e.g. alternative presentation or other information such as economic information)

Free text 18. Relevance: Other

19. Transferability: We had difficulty transferring the information to our local context

20. Language: Reports we write must be in our national language

21. Language: Documents we use must be in our national language

22. Reporting structure: Reports we write must use a specified structure

23. Accountability: We need greater transparency of the process of producing the EUnetHTA assessment

24. Accountability: We need greater availability of documents used in the EUnetHTA assessment

Free text 25. Accountability: Other

Free text 26. Other limiting factors


74

17 Have you undertaken any dissemination activities to promote awareness of the EUnetHTA assessment?


Yes

No

18 Please describe dissemination activities

Free text -

19 Are you happy for EUnetHTA to get in touch to follow up any of your responses?


Yes

No

20 Thank you for completing the survey. Please provide any other comments on this EUnetHTA assessment

Free text -


75

Appendix 2: Response rate

Table 1: Response rate by agency – JA3 assessments

Assessment Publication Date Responses from expected agencies8

Number (N) Percentage (%)

OTCA01 (Wearable cardioverter-defibrillator)

Nov-16 36 of 42 86%

OTCA02 (Antibacterial-coated Sutures) Apr-17 31 of 42 74%

OTCA03 (NIPT) Feb-18 34 of 42 81%

OTCA04 (MammaPrint) Jan-18 31 of 42 74%

OTCA05 (Magnetic stimulation) Apr-17 33 of 42 79%

OTCA06 (TAVI) Dec-18 25 of 42 60%

OTCA07 (FLACS) Oct-18 22 of 42 52%

OTJA08 (Glucose monitoring) Jul-18 29 of 42 69%

OTCA09 (HIFU ablation) Apr-18 29 of 42 69%

OTCA10 (Stool DNA testing) Jul-19 19 of 42 45%

OTCA11 (3D Implants) Apr-19 20 of 42 48%

OTCA12 (CRP POCT) Jan-19 27 of 42 64%

OTCA14 (Robot assisted surgery) May-19 23 of 42 55%

OTCA15 (Irreversible electroporation) Jul-19 22 of 42 52%

OTCA16 (Bioresorbable stents) Jan-19 25 of 42 60%

OTCA17 (Lithium triborate) Nov-19 13 of 42 31%

OTCA18 (Regional hyperthermia) Oct-19 15 of 42 36%

OTCA19 (Screening for osteoporosis) Sep-19 19 of 42 45%

OTCA20 (EVAR / TEVAR) Nov-19 12 of 42 29%

OTCA22 (POCT: D Dimer and Troponin) Dec-19 14 of 42 33%

PTJA01 (Midostaurin) Nov-17 42 of 49 86%

PTJA02 (Regorafenib) Oct-17 41 of 49 84%

PTJA03 (Alectinib) Jan-18 40 of 49 82%

PTJA06 (Polotuzumab) Mar-20 15 of 49 31%

PTJA07 (Ustekinumab) Oct-19 21 of 49 43%

PTJA08 (Siponimod) Feb-20 15 of 49 31%

PTJA09 (Brocilizumab) Mar-20 10 of 49 20%

8 Calculated based on 42 agencies currently using HTA to assess non-pharmaceutical technologies and 49 using

HTA to assess pharmaceutical technologies. Data on use of HTA was collected by WP7 in their research and

analysis of HTA and reimbursement processes in EUnetHTA partner countries and from partners in the

implementation network.


76

Table 2: Response rate by country – JA3 assessments

Assessment

Responses from expected

countries9

Number Percentage (%)

OTCA01 (Wearable cardioverter-defibrillator) 24 of 25 96%

OTCA02 (Antibacterial-coated Sutures) 23 of 25 92%

OTCA03 (NIPT) 24 of 25 96%

OTCA04 (MammaPrint) 24 of 25 96%

OTCA05 (Magnetic stimulation) 23 of 25 92%

OTCA06 (TAVI) 20 of 25 80%

OTCA07 (FLACS) 18 of 25 72%

OTJA08 (Glucose Monitoring) 22 of 25 88%

OTCA09 (HIFU Ablation) 21 of 25 84%

OTCA10 (Stool DNA testing) 16 of 25 64%

OTCA11 (3D Implants) 18 of 25 72%

OTCA12 (CRP POCT) 22 of 25 88%

OTCA14 (Robot assisted surgery) 18 of 25 72%

OTCA15 (Irreversible electroporation) 17 of 25 68%

OTCA16 (Bioresorbable Stents) 21 of 25 84%

OTCA17 (Lithium triborate) 13 of 25 52%

OTCA18 (Regional hyperthermia) 15 of 25 60%

OTCA19 (Screening for osteoporosis) 17 of 25 68%

OTCA20 (EVAR / TEVAR) 13 of 25 52%

OTCA22 (POCT: D Dimer and Troponin) 14 of 25 56%

PTJA01 (Midostaurin) 28 of 30 93%

PTJA02 (Regorafenib) 28 of 30 93%

PTJA03 (Alectinib) 25 of 30 83%

PTJA06 (Polotuzumab) 12 of 30 40%

PTJA07 (Ustekinumab) 18 of 30 60%

PTJA08 (Siponimod) 13 of 30 43%

PTJA09 (Brocilizumab) 8 of 30 27%

9 Calculated based on 25 countries currently using HTA to assess non-pharmaceutical technologies and 30 using

HTA to assess pharmaceutical technologies. Data on which countries use different types of HTA was collected

by WP7 in their research and analysis of HTA and reimbursement processes in EUnetHTA partner countries and

from partners in the implementation network.

EUnetHTA WP7: Deliverable 7.2 - Final Report

Documents

Transcript of EUnetHTA WP7: Deliverable 7.2 - Final Report