Existing indicators to measure the biomedical innovation ... · IT Information Technology KRL...

Working Paper

Existing indicators to measure the biomedical innovation ecosystem

A targeted landscape review for FasterCures

Advait Deshpande, Courtney Hood, Brandi Leach, Susan Guthrie

RAND Europe

WR-1312-FC

October 2019

Prepared for FasterCures

RAND working papers are intended to share researchers’ latest findings and to solicit informal peer review. They have been approved for circulation by RAND Europe but have not been formally edited or peer reviewed. Unless otherwise indicated, working papers can be quoted and cited without permission of the author, provided the source is clearly referred to as a working paper. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors. RAND® is a registered trademark.

EUROPE

https://www.rand.org/pubs/working_papers/WR1312.html

https://www.rand.org/randeurope.html

ii

For more information on this publication, visit www.rand.org/pubs/working_papers/WR1312.html

Published by the RAND Corporation, Santa Monica, Calif., and Cambridge, UK

© Copyright 2019 The Milken Institute

RAND Europe is a not-for-profit organisation that helps improve policy and decision making through research and analysis.

Limited Print and Electronic Distribution Rights This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited. Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial use. For information on reprint and linking permissions, please visit www.rand.org/pubs/permissions.html.

Support RAND Make a tax-deductible charitable contribution at

www.rand.org/giving/contribute

www.rand.org

www.randeurope.org

R® is a registered trademark

http://www.rand.org/pubs/working_papers/WR1312.html

http://www.rand.org/pubs/permissions.html

http://www.rand.org/giving/contribute

http://www.rand.org

http://www.randeurope.org

iii

Abstract

This working paper describes a targeted landscape review conducted in order to assess evidence in the existing literature regarding the characterisation and measurement of the biomedical innovation ecosystem. For the purposes of this paper, we use the term biomedical innovation ecosystem to encompass a dynamic network of activity which connects and combines different aspects of the therapeutic supply chain, including R&D, the pharmaceutical industry supply chain, medical professionals, regulatory and legislative elements of the drug testing and approval process, and deployment and adoption of the innovation and research outcomes.

The aim of the work is to identify existing indicators for the biomedical innovation ecosystem, as well as exploring wider considerations necessary in the conceptualisation and evaluation of the ecosystem. The work was conducted to support FasterCures, a Center of the Milken Institute, as part of their wider project to develop a scorecard to measure the performance of the biomedical innovation ecosystem. The work is likely to be of interest to research funders and evaluators, particular those looking to develop indicators, or those interested in evaluation at a system level.

This study was conducted by RAND Europe. RAND Europe is a not-for-profit research organisation that helps to improve policy and decisionmaking through research and analysis.1

For further information on this document or on RAND Europe please contact:

Dr Susan Guthrie

RAND Europe

Westbrook Centre, Milton Road

Cambridge CB4 1YG

United Kingdom

Telephone: +44 (1223) 353 329

Email: [email protected]

1 For more information on RAND Europe, please see http://www.rand.org/randeurope.html (as of 6 November 2018)

mailto:[email protected]

http://www.rand.org/randeurope.html

v

Table of contents

Abstract ................................................................................................................................................... iii

Figures .................................................................................................................................................... vi

Tables .................................................................................................................................................... vii

Abbreviations and Acronyms ................................................................................................................... xi

Acknowledgements................................................................................................................................ xiv

1. Introduction and context .............................................................................................................. 1

1.1. Background ................................................................................................................................ 1

1.2. Methods and scope .................................................................................................................... 1

1.3. Structure of this working paper .................................................................................................. 2

2. Measuring the performance of the biomedical innovation ecosystem ............................................ 3

2.1. The biomedical innovation ecosystem ........................................................................................ 3

2.2. Wider considerations in evaluation and indicator development .................................................. 7

2.3. Structuring indicators: evaluation frameworks .......................................................................... 10

3. Indicators .................................................................................................................................... 23

3.1. Innovation ecosystems .............................................................................................................. 23

3.2. Academic research .................................................................................................................... 28

3.3. Translational research............................................................................................................... 33

3.4. Private sector research .............................................................................................................. 37

3.5. Regulation and patient access ................................................................................................... 41

4. Discussion ................................................................................................................................... 45

References .............................................................................................................................................. 51

Annex A. Methods ........................................................................................................................... 69

Scope .............................................................................................................................................. 70

Literature review ............................................................................................................................. 70

Interviews ....................................................................................................................................... 71

Annex B. Overview of a range of indicators used in the evaluation and assessment of academic biomedical and health research ....................................................................................................... 73

Annex C. TRIS indicators ............................................................................................................... 93

vi

Figures

Figure 2.1 Therapeutic development chevron ........................................................................................... 3

Figure 2.2 NETS model ........................................................................................................................... 5

Figure 2.3 4DM ....................................................................................................................................... 6

Figure 2.4 Framework for prioritization of indicators ............................................................................. 10

Figure 2.5 Payback Framework logic model ........................................................................................... 12

Figure 2.6 Comparison of translational gaps in different frameworks ..................................................... 15

Figure 2.7 Adaptive biomedical innovation ............................................................................................ 21

Figure 4.1 Initial conceptualization ........................................................................................................ 46

Figure A.1 Issues identified for research and analysis within the biomedical innovation ecosystem for the purposes of this project ........................................................................................................................... 69

vii

Tables

Table 2.1 Groupings of frameworks and approaches to Research & Innovation (R&I) impact evaluation .............................................................................................................................................................. 13

Table 2.2 Overview of selected papers on innovation ecosystem frameworks .......................................... 18

Table 3.1 Innovation levels and indicative data sources .......................................................................... 23

Table 3.2 Innovation ecosystem indicators from a country/company-level perspective ........................... 24

Table 3.3 Innovation ecosystem indicators from a linear innovation (input-throughput-output) perspective ............................................................................................................................................. 27

Table 3.4 Indicators used to measure the performance of academic research ........................................... 29

Table 3.5 Typology of intervention types and indicators for their translation stage ................................. 31

Table 3.6 Possible indicators to measure level of research waste and integrity in research ........................ 33

Table 3.7 A sample of novel and relevant metrics from the Translation Research Impact Scale ............... 34

Table 3.8 Examples of ‘process markers’ denoting progress in translation ............................................... 35

Table 3.9 Measures of biomedical innovation, focus on the pharmaceutical industry.............................. 38

Table 3.10 Measures of public – private collaboration and interactions .................................................. 40

Table 3.11 Existing indicators for measuring regulation of biomedical innovation ecosystem ................. 42

Table 3.12 Existing and potential indicators for measuring patient access to biomedical innovations...... 43

Table 4.1 Overview of indicators identified and issues they address ........................................................ 47

Table A.1 Search strategy for issue area ................................................................................................... 70

Table A.2 Summary of interviews conducted .......................................................................................... 72

Table B.1 Summary of most relevant indicators ..................................................................................... 73

Table B.2 Payback framework ................................................................................................................ 74

Table B.3 Research impact model .......................................................................................................... 74

Table B.4 Framework and indicators to measure returns on investment in health research .................... 74

Table B.5 Excellence in Research for Australia (ERA) ............................................................................. 75

Table B.6 Research Excellence Framework (REF) ................................................................................. 76

Table B.7 National Science Foundation (NSF) ...................................................................................... 78

Table B.8 Snowball metrics .................................................................................................................... 79

Table B.9 Research Outcomes Systems (ROS) ....................................................................................... 79

viii

Table B.10 Consortia Advancing Standards in Research Administration Information (CASRAI) ........... 80

Table B.11 Framework for tracking diffusion of research outputs and activities to locate indicators that demonstrate evidence of biomedical research impact .............................................................................. 81

Table B.12 Indicators for measuring the impact of health research ......................................................... 85

Table B.13 Metrics to measure the impact of Tommy’s work ................................................................. 87

Table B.14 Metrics to Assess and Communicate the Value of Biomedical Research ................................ 88

Table B.15 A framework for assessing the impact of health services and policy research on decisionmaking .............................................................................................................................................................. 90

Table B.16 An approach to evaluate the impact of an applied prevention research centre ....................... 91

Table C.1 TRIS indicators ..................................................................................................................... 93

xi

Abbreviations and Acronyms

4DM Drug Discovery, Development, and Deployment Map

ABPI Association of the British Pharmaceutical Industry

AHP Allied Health Professional

ANSI American National Standards Institute

BRDIS Business Research and Development and Innovation Survey

CAHS Canadian Academy of Health Sciences

CASRAI Consortia Advancing Standards in Research Administration Information

CDER Center for Drug Evaluation and Research

CIS Community Innovation Survey

CO2 Carbon Dioxide

CPT Current Procedural Terminology

CTSA Clinical and Translational Science Award

DALY Disability Adjusted Life Year

DARE Diversity Approach to Research Evaluation

ERA Excellence in Research for Australia

FDA Food & Drug Administration

GDP Gross Domestic Product

GMP Good Manufacturing Practice

GPA Grade Point Average

HCP Health Care Professional

HCPCS Healthcare Common Procedure Coding System

HQ Head Quarter

HSPR Health Service and Population Research

ICD International Statistical Classification of Diseases and Related Health Problems

RAND Europe

xii

ICT Information and Communications Technology

IDE Investigational Device Exemption

IND Investigational New Drug

IRB Institutional Review Board

ISI Institute for Scientific Information

ISO International Organization for Standardization

IT Information Technology

KRL Knowledge Readiness Level

MNC Multi-National Corporation

MTA Medical Technology Assessment

NETS Navigating the Ecosystem of Translation Sciences

NGO Non-Governmental Organization

NHS National Health Service

NICE National Institute for Health and Care Excellence

NIH National Institutes of Health

NMA New Molecular Approach

NME New Molecular Entity

NSF National Science Foundation

PDP Personal Development Planning

PhD Doctor of Philosophy

PHE Public Health England

PICTF Pharmaceutical Industry Competitiveness Task Force

PMA Pre-Market Approval

PROM Patient Reported Outcome Measure

PPP Public Private Partnership

QALY Quality Adjusted Life Year

R&D Research & Development

R&I Research & Innovation

RAG Red, Amber, and Green

RCT Randomized Control Trial


xiii

REF Research Excellence Framework

ROS Research Outcomes Systems

S&T Science & Technology

SME Small and Medium Enterprise

SSII Service Sector Innovation Index

STEM Science, Technology, Engineering, and Mathematics

TFP Total Factor Productivity

TRIS Translational Research Impact Scale

TRL Technology Readiness Level

UK United Kingdom

VC Venture Capital

WHO World Health Organization

RAND Europe

xiv

Acknowledgements

The authors would like to thank Anna DeGarmo, Carly Gasca, Brenda Huneycutt, and Esther Krofah of FasterCures for their support and helpful advice throughout the study. We would also like to acknowledge Jon Sussex and the interview participants for their valuable time and inputs to this study. In addition, we thank Evangelos Gkousis (RAND Europe) for his research support. Finally, we very much appreciate the incisive and timely feedback provided by Jenny George, our RAND Europe quality assurance reviewer.

1

1. Introduction and context

1.1. Background

FasterCures, a Center of the Milken Institute, is conducting a project on biomedical innovation metrics. The aims of the project, in a broad sense, are to understand the current metrics that are in place to measure innovation in the biomedical ecosystem, and establish the extent to which these existing approaches are appropriate and relevant to measure and drive innovation to benefit patients. The ultimate aim of the work is to develop a performance scorecard for biomedical and health research and innovation that captures, end-to-end, the performance of the research and innovation system, and that may incentivise the system to benefit patients more effectively and efficiently. The project so far has consisted of broad scoping and consultation with key informants to understand the existing landscape, and identify metrics and frameworks of interest. In October 2018, FasterCures held an initial stakeholder workshop to gather participant perspectives on (1) current incentives for system actors, (2) ideal behaviour, and (3) metrics across the ecosystem. Throughout 2019, FasterCures is holding a further series of workshops with key stakeholders to progress this project.

To support this work, FasterCures has identified the need for additional input to help identify existing practice and draw on previous research in this area effectively. In this context, FasterCures has commissioned RAND Europe to conduct a targeted landscape review to assess evidence in the existing literature regarding the characterisation and measurement of the biomedical innovation ecosystem. The primary aim of the review is to identify existing indicators that have been conceptualised or used previously which may be relevant to FasterCures for the development of a scorecard to assess the biomedical innovation ecosystem. This is underpinned by a reflection on the concept of a biomedical innovation ecosystem and its definition. Secondary purposes are to better understand and characterise existing frameworks available to measure biomedical innovation and the extent to which they address the performance of the ecosystem, holistically and from a patient perspective; and to highlight some of the key issues and challenges in system evaluation and their pertinence to this work.

1.2. Methods and scope

To address these aims we have undertaken an accelerated evidence review to capture and summarise existing literature regarding metrics and frameworks that measure the biomedical innovation ecosystem. This has been informed primarily through a rapid literature review which capitalises on existing, comprehensive reviews, publications, and grey literature known to the research team through previous

RAND Europe

2

work. This has been complemented by targeted additional searches of known literature in the area as well as ‘snowballing,’ or identification of references within relevant publications. The analysis and literature search have also been informed by a series of semi-structured interviews with five experts who have provided additional in-depth information, and who have helped identify further significant work and publications to form part of the project’s literature review.

It is important to note that our search approach is not fully systematic or exhaustive; hence there may be examples and literature which we have overlooked. However, the searches conducted should be sufficient to provide a useful and informative summary of key aspects of the literature and the state of the art regarding indicators for the biomedical innovation ecosystem to provide a basis for FasterCures’ ongoing work in developing a scorecard.

1.3. Structure of this working paper

The remainder of this paper sets out the findings of the study as follows:

• Chapter 2 discusses some of the wider issues to be explored in measuring the performance of the biomedical innovation ecosystem, covering possible definitions and conceptualisations (section 2.1); general considerations in evaluation and indicator development (section 2.2); and existing evaluation frameworks which can provide some insight into how an indicator scorecard might be structured (section 2.3).

• Chapter 3 describes existing indicators for the biomedical innovation ecosystem drawing on a range of literature spanning innovation ecosystems (section 3.1); academic research (section 3.2); translational research (section 3.3); private sector research (section 3.4); and regulation and patient access (section 3.5).

• Chapter 4 provides a discussion and overview of key issues and some potential next steps based on the evidence collected. This chapter also includes a collation and summary of the range of indicators identified across all the areas explored in Chapter 3, and provides an initial ‘straw man’ conceptual framework for the biomedical innovation ecosystem to serve as a starting point for FasterCures’ future work.

In addition, more detail regarding the study methods is provided in Annex A, and Annexes B and C include fuller tables of indicators where the material is too extensive for inclusion in the main report.

3

2. Measuring the performance of the biomedical innovation ecosystem

2.1. The biomedical innovation ecosystem

The term ‘innovation ecosystem’, as used in investment, economic development, and research policy discourses, refers to a loosely defined concept combining fundamental (or basic) research and the role of market forces in the development and deployment of the research (Oh et al. 2016). Jackson (2011) identifies an innovation ecosystem as comprising the research economy and the commercial economy where the different actors and entities have a shared goal of enabling technology development and innovation. The term innovation ecosystem itself is often used interchangeably with terms such as innovation systems, innovation support systems, and innovation support platforms in academic (Niosi 2010, Frenkel and Maital 2014, Chen 2014 as cited in Oh et al. 2016, and Seo 2014 as cited in Oh et al. 2016, Talmar, Walrave, Podoynitsyna, Holmström, & Romme, 2018) and grey literature (Bruns 2013, Feld 2012, Hannes 2014, Leach 2014). Building on this concept, the biomedical innovation ecosystem (also referred to in this paper as ‘the ecosystem’) can be considered a dynamic network of activity which connects and combines different aspects of the therapeutic supply chain, including R&D, the pharmaceutical industry supply chain, medical professionals, regulatory and legislative elements of the drug testing and approval process, and deployment and adoption of the innovation and research outcomes (see The National Academies of Science Engineering Medicine, 2017).

Traditional representations of this ecosystem have tended to identify this as a linear process as shown in Figure 2.1.

Figure 2.1 Therapeutic development chevron

Source: Adapted from Wagner et al. (2018a)

The above chevrons depict the process as a linear pipeline, which is useful to understand the journey from drug discovery to deployment at a high level, but may not capture the complexity of the system itself (Wagner et al., 2018a). A fuller depiction of the complexities of the biomedical innovation ecosystem may be displayed as a pathway of complex networks involving multiple actors. The actors in the network including scientists and physicians at one end, in conjunction with patients, payers, regulators, and

RAND Europe

4

legislators at the other end, engage in an often failure-prone iterative process as part of which new drugs and treatments are developed (Wagner et al., 2018a; 2018b).

Pertinent to the objectives of this study are two depictions of the ecosystem that aim to highlight this complexity on the one hand, and on the other hand demonstrate the way in which the actors engage in innovation often disparately.

• The Navigating the Ecosystem of Translational Sciences (NETS) model developed in 2013.

• The Drug Discovery, Development, and Deployment Map (4DM) developed at the National Institutes of Health (NIH) in 2017.

Both these ecosystem depictions aim to indicate the collaborative, coordinated, and competitive nature of biomedical innovation originating in basic and translational research.

The NETS model places emphasis on drug development to depict a networked ecosystem comprising ‘neighbourhoods’ of activities, based on each segment of the original chevron diagram but depicting in more depth and complexity the different activities associated with each area, as well as how they interact within one ‘neighbourhood’ or between different neighbourhoods, including feedback loops between the various actors (for a more detailed description on the concept of neighbourhoods, see Baxter et al., 2013 and Wagner et al., 2018b). With patient access to modern therapeutic strategies as the intended outcomes, it covers target discovery, therapeutic discovery and nonclinical research, regulatory science, and clinical research as the building blocks of the ecosystem. In contrast to the traditional 'pipeline' depiction of the ecosystem, the boundaries between these building blocks, or ‘neighbourhoods’, are not rigid and are intended to depict a connected, networked system (Baxter et al., 2013).


5

Figure 2.2 NETS model

Source: Baxter et al. (2013). Licensed under Creative Commons Attribution 3.0.

The 4DM depiction of the ecosystem expands on the key tenets of the NETS model to incorporate broader ecosystem elements, including pharmacology and biomarker development, the prevailing medical landscape (including patient needs), standard care, financial reimbursement, pricing of drugs and therapies, and post-marketing networks associated with the deployment of biomedical innovations (Wagner at al., 2018b).

RAND Europe

6

Figure 2.3 4DM

Source: Wagner et al. (2018b). Licensed under Creative Commons Attribution-Share Alike 4.0

The NETS model and 4DM aim to represent a nuanced understanding of the complexity of the biomedical innovation ecosystem, taking into consideration the viewpoints of different actors and


7

organisations that are part of the network, and the primarily anecdotal nature of the present understanding of the biomedical innovation ecosystem. Our exploration of the existing indicators for the biomedical innovation ecosystem is primarily informed by the 4DM model, which we found to be one of the most comprehensive representations of the ecosystem, with an emphasis on empirical, measurable indicators.

2.2. Wider considerations in evaluation and indicator development

Based on our experience in evaluating research and innovation systems in the biomedical and health sciences, we are aware of some key overarching considerations and challenges that pervade, regardless of the specific nature and scope of the evaluation (Guthrie et al., 2013). Each of these merits careful consideration and will have implications for the development of a set of metrics to evaluate the performance of the biomedical innovation ecosystem.

2.2.1. Purpose and audience

A key consideration in the development of an evaluation system is careful consideration of the purpose of the evaluation process and the audience(s) for which the output is intended. Although an evaluation may serve multiple audiences and purposes, consideration of the primary purpose of the work and the key audiences it will serve can be important in informing design, and there may be trade-offs where there are multiple aims. The purpose of evaluation processes, particularly on the research side, can be delineated into ‘four As’: allocation, advocacy, analysis and accountability (Morgan Jones and Grant, 2013). Recently, a ‘fifth A’ of adaptation has been proposed (Hill, 2017), with the aim of evaluation being to drive (desirable) changes in behaviour. This additional purpose is likely key to the aims of FasterCures.

These different purposes are not always served well by the same evaluation approaches. For example, a set of case studies developed for advocacy purposes would likely focus on success stories, aim to set out the benefits of supporting research, and be short and accessible. By contrast, case studies with an analytical purpose are more likely to be lengthy, detailed, explore success and failure, and be selected to comprise a mix of work across a portfolio rather than focusing only on the ‘best’ examples. The same applies to more quantitative metrics. Performance measures intending to drive behaviour change may be focused on key areas where that change is needed, and primarily focused on outcomes, whereas measures for accountability purposes are likely to be more holistic and process-focused. Clarity on the priorities across these aims will support more effective selection of a useful set of indicators.

Similarly, though the FasterCures project is seeking to develop a patient-focused framework, this does not necessarily imply that patients are the only (or even main) target audience for this work. The language and focus of metrics and evaluation approaches selected will differ depending on the concerns, technical expertise and level of engagement of each primary audience. For a policy audience, for example, which is likely to be one key audience for this work, communication should be concise, message-oriented and relatively non-technical. A mix of clear data and persuasive ‘stories’ that create an impetus for action (ideally in a very short format – i.e. one page or less) may be an effective communication mix for this group. This is equally true for patients and the public, but the data that speaks to each group might be

RAND Europe

8

quite different. For example, economic arguments are often useful for policy audiences, but may speak less strongly to a patient audience.

2.2.2. Attribution

A common challenge in evaluation processes is attribution (Hughes and Martin, 2012). That is, to what extent can a change (positive or negative) be causally linked to a particular action, process or individual. This is closely linked to discussion of the counterfactual – what would have happened if we hadn’t taken a particular course of action (e.g. supported a particular programme of research)? Often, providing a definitive attribution is difficult or impossible. However, more feasible, and often preferable, is to consider the contribution of different factors to changes in outcomes. Demonstrating a contribution is more feasible since rather than seeking to identify and enumerate all the specific factors contributing to a change, we are seeing whether there is a plausible and justifiable link between an action and an outcome. One useful tool to support this is a logic model or framework mapping out the processes through which outcomes may be achieved, with indicators along that pathway. This ‘theory of change’-based approach allows intermediate outcomes to be captured, providing a plausible connection between ultimate outcomes and changes upstream in the innovation ecosystem (Connell and Kubish, 1998; Weiss, 1995).

2.2.3. Time lags Further exacerbating issues of attribution is the challenge of time lags. The timescales over which changes in practice and, particularly, ultimate health benefits are realised can be very significant, particularly in relation to research and innovation activities (Morris et al., 2011). Mapping those contribution links not just over a complex and multifaceted ecosystem but over many years (or even decades) is challenging. Measures of outcomes now may often link to actions taken many years previously and changes now will not necessarily be evident in terms of patient benefits for some time. Similarly, snapshots of performance at different stages of the ecosystem time might not enable one to make connections between measured performance upstream within the ecosystem and current outcomes. Careful consideration needs to be made of the dynamic and interconnected nature of the ecosystem, the timescale over which changes can and will be made, and benefits identified across the different elements and actors within the innovation ecosystem.

2.2.4. Data completeness and accuracy

Evaluations are inevitably dependent on the quality and availability of data to support the conclusions drawn (Penfield et al., 2014). Where data are incomplete, inappropriate or inaccurate, the validity of observations made based on that data are inevitably compromised. Therefore, this is a key decisionmaking factor in indicator selection and development.

2.2.5. Incentives As noted above, evaluation can be used for purposes of ‘adaptation’, that is to drive desirable changes in behaviour. However this implies that the converse is also true: that inappropriately designed or incomplete evaluations may drive behaviour changes that are not desirable. These so-called ‘perverse incentives’ (Edwards and Roy, 2017) can arise where evaluation and assessment becomes a box-ticking


9

exercise, with behaviour being tailored to meet the requirements measured rather than the overarching changes desired underpinning these metrics (Wilsdon, 2016). Taking a holistic perspective and considering indicators in synthesis rather than individually can help guard against these negative outcomes.

2.2.6. Baselining and counterfactuals A key challenge in evaluation is identifying appropriate comparators to enable meaningful analysis of results. For example, if a metric tells you how long it takes for a new target to be addressed through a phase I clinical trial, on average, how can you assess whether performance is ‘good’? One approach to this is internal baselining, in which you capture change in performance over time, seeking improvement. However, in that case it is difficult to assess what is driving performance. If you are evaluating whether interventions have improved the performance you are measuring, it can be difficult to assess whether the change would have happened over time without that intervention, due to wider factors (i.e. the counterfactual). Baselining against comparators can also be a useful approach – for example, by making comparisons with other countries. However, there are challenges inherent in making comparisons in this way given the differences in context between settings. There is no one ideal solution to this challenge, but for each metric careful consideration needs to be given to how it will be interpreted, and if a relevant baseline or comparator is required.

2.2.7. Indicator prioritisation

Prioritising indicators requires a number of considerations to be taken into account. One core principle in most contexts is likely to be proportionality – that is, the approach should not be unnecessarily burdensome in terms of data collection and analysis relative to the purpose of the analysis. For FasterCures, where there is no financial incentive (at least in the first instance) for stakeholders to contribute and share data, this will be particularly important. Burdensome data collection approaches are likely to strain goodwill, and metrics and indicators that can draw on existing data sources are likely to be significantly more feasible. With this in mind, one approach to indicator prioritisation is to assess relevance and feasibility (using, for example, a Red, Amber, and Green i.e. RAG or traffic lights approach), as illustrated in Figure 2.4.

RAND Europe

10

Figure 2.4 Framework for prioritization of indicators

Source: RAND Europe

However, this is unlikely to produce, in isolation, an ideal mix of metrics. It is important, for various reasons set out above (such as incentives and interpretation), to consider the portfolio of indicators holistically to ensure the set of measures works collectively to support the development of system-level insights, and that they work together from a data collection perspective. For example, one methodology may be suitable to generate a range of measures, so even if they are not as central in some areas it may be worth including them since they can be collected as part of one data collection exercise. These types of synergies will be important to consider from an efficiency and burden point of view.

2.2.8. Definitions and standards Given the challenges in identifying relevant comparators in evaluation, there have been various efforts to better standardise the measurement of research and innovation (Mugabushaka et al., 2012; Lane, 2010). One notable such exercise is the Consortia Advancing Standards in Research Administration Information (CASRAI)2, an international membership group which aims to provide common definitions to reduce the burden in measurement and evaluation of research. Using common standards and definitions where feasible can both make data collection easier, and enable a wider range of comparators to be identified.

2.3. Structuring indicators: evaluation frameworks

Although the primary focus for FasterCures at this stage is to identify and develop relevant indicators to measure the biomedical innovation ecosystem, it is also important to consider the composition and

2 https://www.casrai.org/

https://www.casrai.org/


11

structure of these indicators. As set out above, individual indicators tend to provide limited information, and when used out of context can even be misleading. For example, a system which rapidly translates new treatments into practice might be considered to be extremely positive, if considered in isolation. However, if this was at the expense of adequate testing for patient safety, the assessment would be very different. Evaluation frameworks can provide a way to structure and interpret indicators, ensure appropriate indicator coverage, and can also be important in mapping and communicating the system you are analysing. An overview of pertinent evaluation frameworks that can provide information on the selection of indicators for the purposes of this study is included below.

2.3.1. Health research evaluation frameworks There is a well-established literature on research evaluation looking at research developments contributing to health in the academic setting. Many of these frameworks comprise elements focusing on the research system itself, though most also include elements relating to the practical application, or non-academic impact, of research. As a result, reviewing a number of these frameworks would be particularly useful when devising and understanding existing indicators for measuring a biomedical innovation ecosystem. There have been several previous comprehensive reviews of this space (e.g. Banzi et al., 2011; Bornmann, 2013; Penfield et al., 2014) which merit a brief discussion on their own. Notable among these is the review by Raftery et al. (2016), which confirmed the breadth of potential approaches to evaluate such programmes of research, building on and extending a previous systematic review conducted by the same team (Hanney et al., 2007). The typology of models described in this review consists of 20 frameworks that have been applied in health and cross-disciplinary contexts.

Most commonly used among these are logic model-based approaches, and the most widely used framework in the existing literature is the Payback Framework (Buxton and Hanney, 1996). The Payback Framework consists of both a logic model (see Figure 2.5) and a typology of research outcomes which can facilitate the analysis of both the outcomes of research programmes and the pathways through which these are obtained. It is particularly well suited to detailed case study analysis, though it has been applied across wider methodological contexts. Many other frameworks draw upon and adapt the core of the Payback approach to apply in different settings – for example, the Canadian Academy of Health Sciences (CAHS) framework (CAHS, 2009) and the Banzi framework (Banzi et al., 2011).

RAND Europe

12

Figure 2.5 Payback Framework logic model

Source: Adapted from Buxton and Hanney(1996)

As set out by Raftery et al. (2016), logic model-based approaches have a number of advantages in the evaluation of research programmes: providing a useful conceptualisation of the links between inputs, processes and outcomes (in the context of wider mediating factors); helping communicate the relationships between various elements of a programme; and as an organisational tool for the purposes of evaluation. However, if not used carefully they carry the risk of falsely creating a sense of linearity that might not be a good representation of the true processes taking place.

In a recent study (Guthrie et al., 2018), we developed a classification approach bringing together the evidence from key reviews, as well as primary sources, to produce ten broad groupings of frameworks and approaches to the evaluation of academic research, which are set out in Table 2.1. These groupings are intended to bring together frameworks which i) draw on largely similar theoretical and conceptual underpinnings; or (ii) have key characteristics in common that mean they naturally fall into the same grouping. This therefore provides a useful resource in setting out the main frameworks used in practice, which could be relevant to capture holistic information across portfolios of research. The majority of the approaches noted have previously been applied to health research, and the remainder have been applied either on aggregate across research in any discipline (e.g. at a national level) or a in a diverse range of research contexts.


13

Table 2.1 Groupings of frameworks and approaches to Research & Innovation (R&I) impact evaluation

Framework Examples of studies which apply or draw on that framework

Brief summary of key principles

Payback Framework-based approaches

Canadian Academy of Health Sciences (2009)

Banzi et al. (2011)

Aymerich et al. (2012)

Bennett et al. (2013)

Bunn et al. (2014)

Donovan et al. (2014)

Expert Panel for Health Directorate of

the European Commission’s Research Innovation Directorate General (2013)

Wooding et al. (2014)

Consists of two components: a logic model and taxonomy for research impact. Taxonomy comprises five categories of health research benefit: knowledge production; research targeting and capacity building; informing policy and product development; health and health systems benefits; broader economic benefits. It has been adapted and used in different contexts, including outside health.

Economic analyses, typically focusing on rates of return

Deloitte Access Economics (2011)

Guthrie et al. (2015)

Haskel et al. (2014)

Sussex et al. (2016)

Frontier Economics (2014)

This is a broad category which comprises different monetary approaches to assessing returns from research, including return on investment, cost benefit analysis, and other wider monetisation approaches.

National-level self-assessment-led approaches

Royal Netherlands Academy of Arts and Sciences (2010)

High Council for Evaluation of Research and Higher Education (France)

Program Assessment Rating Tool (Gilmour, 2006)

These are national-level exercises, typically conducted every year or few years, to evaluate the research performance of institutions or programmes within the country, based primarily on self-assessment and reflection, sometimes supplemented by peer or other external review. Typically there is some additional (limited) use of metrics, and the evaluation is usually against goals and objectives of the institution or programme.

Interaction-based models

Meijer (2012)

Spaapen et al. (2011)

Mostert et al. (2010)

Meagher et al. (2008)

This covers models based on the concept of measuring interactions, relationships and flows of knowledge between researchers and stakeholders as a route to measuring impact.

RAND Europe

14

Framework Examples of studies which apply or draw on that framework

Brief summary of key principles

Programme-specific logic model approaches

Williams et al. (2009)

Sainty (2013)

Evans et al. (2014)

Rycroft-Malone et al. (2013)

Weiss (2007)

Linked to theory of change, these are approaches which develop and tailor logic models and accompanying evaluation approaches to the specific programme or institutions in question, recognising that research may have different impacts in different settings, depending on individuals and circumstances. Builds on Weiss’s logic modelling work and/or realist approaches to evaluation.

Dashboard/balanced scorecard approaches

The Research Council, Oman (Krapels et al. 2015)

NIHR (El Turabi et al., 2011)

Wellcome Trust (2014)

(Ontario) University Health Network (2008)

Measures performance, and drives organisational strategy by incorporating organisational aspects together with financial performance.

Regular monitoring by self-report

Drew et al. (2013)

MRC (2013)

Wooding et al. (2009)

MRC (2018)

Ongoing monitoring through platforms such as Researchfish or other self-report-based approaches replacing or complementing typical project/programme reporting.

Peer review-based large-scale approaches

Research Excellence Framework (HEFCE, 2015)

Group of Eight and Australian Technology Network of Universities, 2012 (Group of Eight, 2012)

PBRF (PBRF Working Group, 2002)

National Research Council (1999)

Peer review-based approaches that use panels to review the performance of research on a mix of criteria drawing on a range of materials which may include metrics, publications and case studies.

Emerging ‘big data’ analytical approaches

STAR METRICS (NSF, 2010)

Altmetric (n.d.)

Approaches that use data and/or text mining to draw together and analyse existing datasets.

‘Added value’ approaches

Arnold (2012) Approaches that aim to address the counterfactual by looking at the ‘added value’ or portfolio-level effect of a funding programme over and above other funding mechanisms.

Source: Guthrie et al. (2018)

2.3.2. Frameworks for translational research

It is frequently suggested that it takes 17 years for research evidence to reach or translate into clinical practice (Balas and Boren, 2000; Grant et al., 2003; Green et al., 2009; Trochim, 2010; Wratcshko, 2009; Westfall et al., 2007; Buxton et al., 2008; Morris et al., 2011). However, these different estimates often measure different parts of the translation process are thus are not comparable when aiming to


15

understand the length of time basic research takes to find a place in clinical practice. In any case, such convergence around an ‘average’ time lag hides complexities that are relevant to policy and practice which would benefit from greater understanding (Hanney et al., 2015). A key challenge in understanding the translation process is the diverse range of models for research translation, with various translation gaps which are differently named and conceptualised in different frameworks (see Figure 2.6).

Figure 2.6 Comparison of translational gaps in different frameworks

Source: Trochim et al. (2011)

This diversity was identified by Trochim et al. (2011), who proposed as a solution the ‘process marker model’, in which specific events along the translation pathway are identified as markers. These can be clearly defined and thus the stage of translation understood and compared across studies. This was tested empirically by Hanney et al. (2015) who operationalised the concept through detailed case studies which were mapped against ‘research tracks’. In this analysis, key points along the translation pathway which could be applied consistently across contexts (for example, first testing in humans, or regulatory approval/first non-research use in patients) were identified and used to provide quantitative estimates of translation time between stages in the research pathway. A similar approach was taken by Marjanovic et al. (2015) in the evaluation of the i4i programme, which identified a set of key translation points and mapped projects against them. This overall translational pathway approach is critiqued by some in the literature (Graham et al., 2006; Littman et al., 2007; Marincola, 2011) as falsely non-linear – since the true nature of research may be highly non-linear, and indeed the process of development may also be characterised in terms of relationship building and sharing of cognitive processes and tacit knowledge. Nevertheless, this provides a useful way to understand and characterise the nature of research within a translational research programme, as long as it is acknowledged that this sits within a wider social context and that the outcomes and the users of the findings of any study may well be upstream as well as downstream from that particular piece of research.

RAND Europe

16

Reflecting this critique of conventional linear approaches, new conceptual models have emerged that break this paradigm. For example, Glasgow et al. (2012) have developed an approach in which evidence and stakeholders are placed at the centre of an ‘evidence integration triangle’ with interactions between and within three components: the intervention; the process through which it is implemented; and measures of progress. Others emphasise the continuum concept, moving beyond specific markers or translational gaps, in the acknowledgement that translation is an ongoing process and that challenges can and do arise across all different stages depending on context and actors (Drolet and Lorenzi, 2011).

Molas-Gallart et al. (2015) stress the importance of process considerations in the evaluation of translational research, noting the importance of bridging gaps between individuals – be they cognitive, social, organisational, institutional or geographical. These relationship building and boundary spanning outcomes are not necessarily easily captured through outcome-oriented conceptual models. This approach has been operationalised through the Diversity Approach for Research Evaluation (DARE) framework (University of Sussex, n.d.) which provides a way to measure the extent to which a research initiative spurs interactions between different groups, and it has been piloted on a number of case studies (Bone et al., 2017). Also relationship focused is the Productive Interactions approach (Spaapen et al., 2011), in which outcomes of research are proxied by interactions. These can be interactions between individuals, either through direct contact, indirectly via a medium (e.g. a document or dataset), or financially-based. The underpinning assumption is that communication and interaction between individuals are the processes through which outcomes can be achieved and therefore can be measured to assess the likelihood of wider influence. Kok and Schuit (2012) also emphasise this in their Contribution Mapping approach, which focuses on the process through which outcomes are generated, rather than the specific outcomes themselves. Given the importance of processes and, particularly, relationships in translational research, it is also worth noting the existing evidence on so-called ‘boundary spanners’ in achieving translational research outcomes. These are individuals who facilitate and embody collaboration and communication between communities, both across disciplines and along the translational continuum, supporting knowledge exchange and managing conflict (Swann et al., 2007; Lander and Atkinson-Grosjean, 2011; Richter et al., 2006).

2.3.3. Innovation ecosystems frameworks In this paper, we consider an innovation ecosystem as collaborative arrangements and complex relationships between individual actors or firms which combine to enable technology developments and innovations as coherent customer-facing solutions, drawing on work by Adner (2006) and Jackson (2011). Such a perspective parallels our reference to depictions of biomedical ecosystems as in the NETS model and 4DM (see section 2.1). In the following discussion, innovation ecosystem frameworks are considered as the framing of the ecosystem from different perspectives of the constituent actors and entities, such as individual actors (researchers), companies, universities, financial and investor institutions, public funding bodies, and regulators. The existing frameworks thus often adopt a singular or dominant perspective such as a private firm (Adner 2006), or a public funding body (Wessner 2007) such as a regional or national government (see Wallner and Menrad 2011, and OECD 1997), to enable a discussion on specific aspects of the interplay between the actors and the entities. In particular, this becomes more readily apparent when national innovation ecosystem frameworks are considered (see


17

OECD 1997). Additional perspectives that often inform the innovation ecosystem frameworks include organisational activity/throughput (St-Germain 2010), human resources (Frey and Osterloh 1997), and entrepreneurship or funding to facilitate innovation (Louma-aho and Halonen 2010). Most frameworks, as Adner (2006) suggests, are aimed at informing the decisionmaking of the individual actors and entities to enable the intended technological developments and innovations to be achieved.

In the table below, we consider some of the most commonly cited frameworks for innovation ecosystems. To identify these frameworks, we combined findings from the literature review with additional searches in the ‘web of knowledge’ and Google Scholar for highly cited and relevant articles. Recognising the changing emphasis of the literature on complex, networked, multi-actor innovation ecosystems, we focussed on the articles published in the last 13 years (i.e. 2006-2019). The search term included 'innovation ecosystem' in the title and abstract, and 'framework' in the article. This approach mirrors the strategy adopted by Durst and Poutanen (2013).

18

Table 2.2 Overview of selected papers on innovation ecosystem frameworks

Author(s) Year Research aim/objectives Research Method(s) Main findings Journal

Adner 2006 To highlight the significance of having a systematic approach to analysing the risks in the ecosystem.

Mainly literature review Managers can establish more realistic expectations and arrive at a more robust innovation strategy if they learn to assess the risks in a more holistic and systematic manner.

Harvard Business Review

Adner and Kapoor 2010 To understand how the challenges faced by external innovators affect the focal firm's outcomes.

Data analysis of the global semiconductor lithography equipment industry, from its emergence in 1962 to 2005, across nine distinct technology generations

The effectiveness of vertical integration as a strategy to manage ecosystem interdependence increases over the course of the technology life cycle. The effects of external innovation challenges depend not only on their magnitude, but also their location in the ecosystem relative to the focal firm.

Strategic Management Journal

Adner and Kapoor 2016 To explore why some new technologies emerge and quickly supplant incumbent technologies while others take years or decades to become established.

Literature review and evidence from ten episodes of technology transition in the semiconductor lithography equipment industry from 1972 to 2009.

Understanding the pace of technology substitution requires joint consideration of the evolution of both the new and the old technologies. Understanding this evolution requires an examination of the interdependencies in the broader ecosystem of components and complements in which the focal technologies are embedded.

Strategic Management Journal

Brusoni and Prencipe

2013 To adopt a problem-solving perspective to analyse the competitive dynamics of innovation ecosystems.

Literature review and secondary data analysis

Features such as uncertainty, complexity and ambiguity entail different knowledge requirements, which explain the varying abilities of focal firms to coordinate the ecosystem and benefit from the activities of their suppliers, complementors, and users.

Collaboration and Competition in Business Ecosystems

Carayannis and 2009 To provide a better conceptual Literature review An innovation ecosystem that encourages the co- International


19


Campbell framework for knowledge-driven events and processes in the economy (public and private sector).

evolution of different knowledge and innovation modes, and balances the non-linear innovation modes in the context of multi-level innovation systems. Innovation and knowledge clusters tie together universities, commercial firms, and academic firms.

Journal of Technology Management

Chesbrough, Kim, and Agogino

2014 To analyse the success factors behind ‘Chez Panisse’, a California-based restaurant which adopted an ‘open innovation’ strategy to grow a small, local business into a global ecosystem.

Case study-based analysis and document review

Chez Panisse thrived and became a business success by building a business ecosystem on the principles of sharing knowledge, encouraging individuals’ growth, and embedding trust among participants (e.g. staff, suppliers, and customers).

California Management Review

Fukuda and Watanabe

2008 To analyse the parallel paths of technology policy in Japan and the US since the 1980s.

Literature review and data analysis

The development cycle in both countries is governed by four ecosystem principles: (1) sustainable development through substitution, (2) self-propagation through co-evolution, (3) organisational inertia and inspired learning from competitors, and (4) heterogeneous synergy.

Technology in Society

Iyer and Davenport 2008 To analyse Google’s innovation ecosystem.

Mainly literature review Google’s success is mainly due to its ability to combine strategic patience with building infrastructure to support innovation, a chaotic ideation process vetted by data-driven methods, and building innovation as part of the employee job description.

Harvard Business Review

Mercan and Göktas 2011 To explore the effects of innovation making based on Global Innovation Index dataset.

Data from Global Innovation Index (2009-2010 database), regression analysis

The interaction between university and private sector for-profit industries accelerates innovation. There is some correlation between innovation culture and innovation output.

International Research Journal of Finance and Economics

Mezzourh and 2012 To analyse how new business Case study of a French start- The authors identify a new concept, ‘keystone The Business

RAND Europe

20


Nakara ecosystems develop to support innovation and strategic choice.

up innovations’, and discuss how such innovations influence business ecosystems.

Review

Rohrbeck, Hölze, and Gemünden

2009 To analyse Deutsche Telekom’s approach to and adoption of an open innovation paradigm.

Case study analysis and interviews

The authors suggest that although Deutsche Telekom uses an open innovation approach extensively, it does not solely rely on such an innovation approach for its success.

R&D Management

Samila and Sorenson

2010 To analyse how local availability of venture capital funding contributes to a commercialisation of innovation.

Literature review and data analysis of metropolitan statistical areas in the USA from 1993-2002

There is correlation between the availability of venture capital and innovation outcomes in US metropolitan areas.

Research Policy

Tassey 2010 To explore whether a new innovation model not based on a neoclassical approach is required to guide economic growth policy.

Secondary data analysis The authors propose a new manufacturing policy. Journal of Technology Transfer

Source: RAND Europe, and Durst and Poutanen (2013)

21

As Wallner and Menrad (2011) highlight, despite the emerging consensus on the complexity of the innovation ecosystems, most of the frameworks tend to focus on specific aspects of the innovation cycle or innovation outcomes. This is particularly so when national innovation frameworks are considered, as such frameworks tend to focus on socio-cultural, regional strengths while borrowing ideas from other countries (see Godin, 2009; Mahroum & Al-Saleh, 2013; and Matei & Aldea, 2012). The main gap in such frameworks has often been the linear, deterministic approaches advocated (Adner & Kapoor, 2010), which according to Wallner and Menrad (2011) contradicts the system metaphor being used. In such a context, the Wellcome Trust’s (2014) assessment of the United Kingdom’s (UK’s) innovation ecosystem is relevant given that the Wellcome Trust primarily funds life science and biomedical research in the UK. In its report, the Wellcome Trust (2014) highlights the importance of translational research and how it is as crucial as basic science research for a successful innovation ecosystem (see also Jones & Wilsdon, 2018). Honig and Hirsch (2016) make similar observations regarding the lag between the translation of science into medical product advances. They suggest that the existing frameworks are ‘slow to adapt, largely reactive, and clearly fragmented’ in capturing the complexity of biomedical systems. In order to achieve effective integration between the discovery, development, regulation, reimbursement, and provision of innovation to patients’ phases, Honig and Hirsch (2016) formulate an adaptive biomedical innovation approach with objectives similar to the 4D Map of the biomedical ecosystem identified in section 2.1.

Figure 2.7 Adaptive biomedical innovation

Source: Based on Honig and Hirsch (2016)

RAND Europe

22

Along with the frameworks discussed in Table 2.2, Honig and Hirsch’s (2016) work informs the discussion on the indicators for the innovation ecosystems in the next section. Where relevant, we also consider the uncertainties inherent to such an ecosystem due to the changing nature of integration between patient preferences, risk assessment of new medicines, and newer forms of clinical trial designs (to name a few of the important aspects) (see also Honig & Hirsch, 2016; Hirsch et al., 2016; and Schneeweiss et al., 2016 for further discussion on adaptive biomedical innovation).

23

3. Indicators

In this chapter we explore the range and nature of existing indicators discussed in the literature that may be of relevance to FasterCures. We have split the chapter into five sections: (i) innovation ecosystems; (ii) academic research; (iii) Translational research; (iv) Private sector research; and (v) Regulation and patient access, based on a combination of key actors within the field and the groupings of indicators which we found both in the literature and following expert guidance.

3.1. Innovation ecosystems

Broadly, approaches to measuring innovation have changed as the linear models of innovation from the 1950s and 1960s have evolved to underscore the interactive nature of the marketplace and the multiple feedback loops between the various actors, including commercial and industrial organisations, institutional actors such as regulators and legislators, and consumers (including patients in a biomedical innovation context) (see Rothwell 1994). From an innovation ecosystem perspective, the indicators for measuring the performance of the system can to be broadly classified at three levels (Hao 2017): country level, company level, and industry level. Below we describe these levels, along with indicative data sources for measuring such indicators. As discussed further below, while these indicators may apply globally and across sectors, they are taken as points of reference to inform indicators of innovation within the biomedical innovation ecosystem.

Table 3.1 Innovation levels and indicative data sources

Level Description Indicative data sources for measuring the indicators

Country-level These indicators are aimed at identifying outcomes related to national innovation policies, and resulting economic growth and development.

Global Innovation Index3

European Innovation Scorecard4

Global Creativity Index5

3 https://www.wipo.int/global_innovation_index/en/ (accessed 2 July 2019). Published by the World Intellectual Property Organization (WIPO). 4 https://ec.europa.eu/growth/industry/innovation/facts-figures/scoreboards_en (accessed 2 July 2019). Provided by the European Commission.

https://www.wipo.int/global_innovation_index/en/

https://ec.europa.eu/growth/industry/innovation/facts-figures/scoreboards_en

RAND Europe

24

Level Description Indicative data sources for measuring the indicators

These indicators focus on the economic, market, and institutional environment.

Global Entrepreneurship Index6

Company-level There is no clear consensus on the approach to measuring innovation at the company-level. As a result, these indicators can be delineated on the basis of linear innovation models (such as input-throughput-output) and complex models based on feedback loops (such as signposts).

Business Research and Development and Innovation Survey (BRDIS)7

Community innovation survey (CIS)8

UK Innovation survey9

Global business surveys conducted by consulting firms such as McKinsey, Boston Consulting Group, and KPMG

Industry-level These indicators are often aggregated as either country-level indexes or company-level measures. These are used either to represent heterogeneity across industries, rank companies in the industries, or present industry trends.

Pharmaceutical Innovation Index10

Service Sector Innovation Index (SSII) based on the CIS

Nesta Innovation Index11

Sources: Hao et al. (2017) and RAND Europe

With increasing globalisation and industrialisation of research, the national boundaries between industry and company are not always well-defined. As a result, we demarcate the following indicators at country- and company-level, drawing on the framework of signposts identified by Hao et al. (2017).

Table 3.2 Innovation ecosystem indicators from a country/company-level perspective

Type of metric (signpost) Country-level Company-level

Technology

These metrics highlight innovative capability at company- and economy-wide levels.

R&D expenditure

Science and technology personnel

Patents

S&T publications

R&D spending as a percentage of sales

Number of patents from national patent offices

Number of ideas or concepts in

5 http://martinprosperity.org/content/the-global-creativity-index-2015/ (accessed 2 July 2019). Provided by the Martin Prosperity Institute. 6 https://thegedi.org/global-entrepreneurship-and-development-index/ (accessed 2 July 2019). Economic activity index compiled by the US-based The Global Entrepreneurship and Development Institute. 7 https://www.nsf.gov/statistics/srvyindustry/ (accessed 2 July 2019). Provided by the US-based National Science Foundation. 8 https://ec.europa.eu/eurostat/web/microdata/community-innovation-survey (accessed 2 July 2019). Compiled by Eurostat, the European Commission’s statistics agency. 9 https://www.ons.gov.uk/surveys/informationforbusinesses/businesssurveys/ukinnovationsurvey (accessed 2 July 2019). Collected by the UK’s Office of National Statistics (ONS). 10 http://ideapharma.com/pii (accessed 2 July 2019). Collected by private sector organisation IDEA Pharma to identify how effective the 30 top pharmaceutical companies are in commercialising new products. 11 https://www.nesta.org.uk/project/innovation-index/ (accessed 2 July 2019). Collected by the UK-based innovation foundation, Nesta.

http://martinprosperity.org/content/the-global-creativity-index-2015/

https://thegedi.org/global-entrepreneurship-and-development-index/

https://www.nsf.gov/statistics/srvyindustry/

https://ec.europa.eu/eurostat/web/microdata/community-innovation-survey

https://www.ons.gov.uk/surveys/informationforbusinesses/businesssurveys/ukinnovationsurvey

http://ideapharma.com/pii

https://www.nesta.org.uk/project/innovation-index/


25


High-tech exports

Technology use by industry

Industry/institute co-patents

Industry/institute co-publications

Industry use of research institute patents

Industry/institute information sharing

the pipeline

R&D conversion metric (new products or services developed/launched)

R&D product conversion rates

New-products-to-margin conversion

Digitization

These metrics indicate the extent of digital transformation through the use of Information and Communications Technology (ICT), Internet, and other digital technology/equipment.

Information Technology (IT) expenditure (% of Gross Domestic Product i.e. GDP)

Mobile phone subscriptions

Fixed telephone lines

Number of Internet users

Internet bandwidth

Broadband subscriptions

Availability of public services online through government-backed gateways

Software spending

ICT and business model creation

General top-level domains

Video uploads on YouTube

Percentage of documents digitally archived

IT spending per employee

Ratio between IT staff and all non-IT staff

Percentage IT budget of total revenues

Customer Experience & Branding

These metrics are used to understand the way customers engage with and contribute to the innovation.

Spending on advertising (% of GDP)

Number of trademark applications

Rank of country brands

Degrees of customer orientation and buyer sophistication

Advertising spending provided by company financial statements

Data on brands, such as familiarity of corporate brand, reputation of brand power

Customer satisfaction

Environmental and social sustainability

These metrics are used to describe firm-level and economy-wide environmental and social impacts.

Resource efficiency innovators (% of firms)

Carbon Dioxide (CO2) emissions

Emissions from alternative and nuclear energies

GDP per unit of energy

Number of International Organization for Standardization (ISO) 14001 environmental certificates

Energy and electricity consumption

Water consumption

Waste reduction

Biodiversity policies (adoption rates, change throughputs)

RAND Europe

26


Internal Innovation Networks

These metrics are used to assess the extent to which firms and the broader economy are positioned to be competitive on innovative capabilities

The extent of staff training

Value chain breadth

Control of international distribution

Production process sophistication

Willingness to delegate authority

Capacity for innovation

Internal funding for innovations

Talent mix

Access to information

Incentives for innovation success

Organizational structures (hierarchical vs flat)

Individual vs. collective decisionmaking

Leadership involvement in innovation process

Inter-firm research cooperation

External Innovation ecosystems

These metrics indicate the external factors, including the regulatory regime, market demand, infrastructure, and ancillary factors such as collaboration with other organisations

Number of third-country assets (subsidiaries, factories)

Number of companies with dual-nationality headquarters

Number of jointly funded projects/research/innovation initiatives with another country

Ease of doing business in a country

Investment in economic and physical infrastructure

Number of innovation projects with third parties

Joint funding of innovation expenses with other organisations

Amount spent on basic research

Amount spent on participation in innovation platforms

Movement of technical personnel among industry, universities, and research

Profit and revenues

These metrics are used to assess the financial performance, and thus by extension the capability to innovate, of the system.

Gross Domestic Product

Gross National Product

Economic growth rate (annual/quarter)

Budget deficit

National/regional debt levels

Percent sales from new products/services in given time period

Increase in return on equity due to innovation

Return on investment in new products or services

Potential of entire new product/service portfolio to meet growth targets

Changes in market share due to new products/services

Net present value of entire new product/service portfolio

Return on innovation investment (actual results of historical investment, expected value of current investments in innovation)

Return on equity (profitability,


27


operating efficiency, financial leverage)

Source: Aase et al. (2018), Anthony (2013), Hao et al. (2017), OECD (1997), RAND Europe

In addition to these indicators, which focus on the complex systems perspective, we also acknowledge the role of the oft-used linear innovation perspective to classify the indicators according to where in the input-throughput-output cycle the indicator takes place.

Table 3.3 Innovation ecosystem indicators from a linear innovation (input-throughput-output) perspective

Type of metric (signpost) Input Throughput Output

Technology R&D Patents Licence fees

Digitization ICT spending ICT access index ICT and business model creation

Customer Experience & Branding

Spending on advertising Relationship duration Customer satisfaction

Environmental and social sustainability

Investment in operational sustainability

Number of ISO 14001 environmental certificates

Environmental performance index

Internal Innovation Networks Spending on innovation projects

Number of new ideas created internally

Number of new products developed from new ideas

External Innovation ecosystems

Venture capital access University/industry collaboration

Innovators (% of Small and Medium Enterprises i.e. SMEs)

Profit and revenues Innovation budget Potential of entire new product/service portfolio to meet growth targets

% of sales revenues from new products/services

Source: Hao et al. (2017) and RAND Europe

From the above lists, the indicators for signposts such as technology, internal innovation networks, and external innovation ecosystems are particularly relevant and transferable when considering an adaptive biomedical innovation ecosystem framework (see section 2.3.3). These indicators align with the complex interactive system paradigms that form the basis of such frameworks, and thus can be potentially mapped to the integration of different phases such as discovery, development, and provision to patients.

Frameworks such as that developed by Adner (2006) and Adner and Kapoor (2010; 2016) offer a firm-level perspective, which when combined with company-level indicators such as R&D spending, S&T personnel, high-tech exports, decisionmaking style, incentives for innovation, and customer engagement/satisfaction, facilitate an understanding of the extent to which basic research is effectively translating into patient treatments, and any bottlenecks in the system limiting patient access to new interventions.

RAND Europe

28

Despite the potential relevance of these indicators to FasterCures’ work however, there are a number of possible systemic challenges not only to producing the data needed, but also measuring these indicators. We briefly discuss the challenges most relevant to a biomedical innovation ecosystem and aligned with the scope of this study:

• Company-level indicators are mainly reliant on the data produced by internal company surveys, or surveys produced by consulting firms which do not necessarily produce a sufficiently wide range of data (Hao et al. 2017).

• The company-level indicators are often likely to be assessed in functional siloes (such as R&D or financial performance), and thus do not cover the full life cycle of innovation (Adams, Bessant, & Phelps, 2006; Cordero 1990).

• Output measures such as patents, licence fees and product launches are over-emphasised in the literature which can obscure the limitations of the innovation process and whether it is successfully addressing customer requirements and achieving customer satisfaction (Morris 2008).

• Country-level measures are likely to over-emphasise input metrics such as R&D spending, but do not necessarily consider output metrics such as environmental and social impacts (which can be difficult to ascertain, but are explored in more detail in section 3.2) (Hao et al. 2017).

• Country-level innovation data in the public domain is often 3-5 years out of date and thus neither aligns with prevalent innovation models nor offers the correct baseline for innovation policy-making (Hao et al. 2017).

• Country-level innovation data does not offer effective means to understand innovation collaboration between different countries – crucial in the context of a global research system (Hao et al. 2017).

3.2. Academic research

Measures of academic research in the biomedical and health field are diverse and span many outputs and outcomes of research. There is a significant literature in the space with many metrics and indicators reported. However, the majority of these focus on evaluating specific projects or programmes of research rather than analysing the health and development of research and innovation at a system level. However, many of these existing metrics may well be readily adapted for application at a broader cross-cutting level.

Some of the most commonly used and longstanding measures focus on impacts on the research system – such as numbers of publications, measures of publication quality (from bibliometrics or peer review), and development of the research system capacity, whether human or infrastructure. However, an increasing focus on the societal benefits of investment in research has also led to the development of many metrics which speak to the policy, health and other benefits of research investment. Analysing existing portfolios of metrics, we can identify five broad groupings of indicators in terms of the benefits from research which they assess:

• Research system indicators – including new knowledge and research tools and techniques, capacity building and network development, and targeting of future research into more promising avenues.


29

• Economic and innovation indicators – including new products developed, companies set up, and evidence of innovative outputs such as patents, licensing etc.

• Health and health system indicators – spanning various stages of development for new interventions, and covering preventative and public health measures, and all types of new technologies, treatments, service delivery innovations, diagnostics and other interventions.

• Policy engagement indicators – including informing policy documentation and guidelines, providing consultation and advice through advisory groups or direct discussions, commissioned policy research.

• Engagement and public involvement in research indicators.

Although the primary interest of FasterCures is likely to be those indicators related to benefits to health and the health system, metrics across all these categories are relevant to the assessment of the overall performance of the biomedical innovation ecosystem, including those measuring the intrinsic quality and productivity of the research system. With this in mind, a summary of some of the types of metrics identified in each of these categories is provided in Table 3.4, with a more detailed list of specific indicators, identified from a range of sources, provided in Annex B.

Table 3.4 Indicators used to measure the performance of academic research

Category Scope of indicators included Relevant examples

Research system indicators

Number and quality of outputs including but not limited to publications; new research tools and resources generated; relevance of research conducted; targeting of future research; funding received; Esteem measures, number and quality of researchers trained, collaboration and networking (academic and wider), career outcomes for researchers and students trained

Quality indicators e.g. relative citation impact, highly cited publications, publications in high-quality outlets

Funding invested and its distribution (e.g. by field, geography, target patient population, level of appliedness)

Patient and clinician perceptions of the relevance of research conducted

Overall researcher population and trends in career advancement and research fields

Level of engagement of clinicians in research

Number and range of researchers trained (at different levels)

Economic and innovation indicators

New products and process developed (and patented, licensed, used); new businesses (spinouts); benefits to companies in terms of profitability, new clients, competitive advantage, efficiency etc.; Job creation, workforce development and increased economic competitiveness of region/country

Number and status of different innovative outputs (e.g. Medical Technology Assessments i.e. MTAs, licences, patents)

Number of spinoffs and start-ups formed, and their outcomes (e.g. revenue generated, duration, value at termination, employees, failure rate)

Number of inventions or innovative discoveries (e.g. number of new drug targets identified)

Health and health system indicators

Impact on professional training or development in health; impact on practice including efficiency/cost

Compliance and adherence to research-informed policies and guidelines

Adoption of health technologies

RAND Europe

30

Category Scope of indicators included Relevant examples

savings; regulatory approval and implementation of new interventions; practice reflects evidence-based guidelines; making processes more efficient or resilient etc.; impact on health and well-being (of a group of patients, or more widely)

Measures of health outcomes (e.g. prevalence, incidence, Quality Adjusted Life Years i.e. QALYs, patient-reported outcome measures, patient satisfaction measures, Reduced adverse events/complications)

Narrowing of health/health care disparities

Research-based interventions progressing through translation pathway (e.g. progressing through clinical trials (Phases I/II/III as appropriate), registered/licensed with the Food and Drug Administration (FDA) agency, used by health care providers and/or consumers)

Policy engagement indicators

Interaction with policy- and decisionmakers through invitations, meetings, committees; citation in or other influence on policy documents including guidelines; resulting changes in policy; improvement in public services (including outside of health – e.g. Science, Technology, Engineering, and Mathematics i.e. STEM education); resulting benefits to society and quality of life

Number of evidence-based guidelines developed, or proportion of research contributing to guidelines

Citations in policy documents (of any form and for different levels of government or non-governmental bodies and associations)

Interactions of researchers with policy- and decisionmakers (e.g. committees, advisory boards, testimony)

Decisionmakers’ awareness of research (and its source), knowledge of research (and its source), attitudes towards research (and its source)

Engagement and public involvement in research indicators

Number and range of dissemination and outreach activities; increased public understanding of and engagement with science and research (e.g. participation in clinical trials); more positive attitudes towards research and researchers

Level of involvement of researchers in outreach and engagement activities

Level of public/patient involvement in research

Recruitment and retention in clinical trials

Attitudes of the public and of research participants to research and researchers

Sources: Banzi et al. (2011), Guthrie et al. (2016), Deshpande et al. (2018); Pollitt et al. (2016); Lavis et al. (2003); Bernard Becker Medical Library (n.d.), CHSPRA (2018), Willis et al. (2017).

It is worth highlighting the extensive and comprehensive library of indicators provided in the Bernard Becker Medical Library which, though aimed at the evaluation of individual research projects, serves as a useful typology of potential measures of the performance and translation of research. In particular, we note their categorisation of both intervention types and translation stages for interventions, which is summarised in Table 3.5. This could provide a useful basis for analysis of the range and nature of interventions being developed across the health system and their stage of development.


31

Table 3.5 Typology of intervention types and indicators for their translation stage

Intervention type

Indicators denoting stages of progression

Drug Potential new drug shows benefit during pre-clinical development trials.

Investigational New Drug (IND) application is accepted by FDA

Drug generated by the research study shows benefit during clinical trials.

Drug generated by the research study registered/licensed with FDA.

Drug generated by the research study is listed on a drug formulary list.

Drug generated by the research study listed on the World Health Organization (WHO) Model List of Essential Medicines.

Drug generated by the research study is prescribed by health care providers.

Drug generated by the research study is used by consumers.

Biological material

Biological material application generated by the research study shows benefit during clinical trials.

Biological material application generated by the research study is registered/licensed with FDA.

Biological material application generated by the research study is used by health care providers and/or consumers.

Service organisation and delivery

Research study findings result in a clinically effective approach to the management and treatment of a disease, disorder or condition.

Clinicians in the field report change in delivery of healthcare based on findings of research study.

Research study findings result in increased performance, quality, and consistency in the delivery of health care services.

Diagnostics and screening

Research study findings result in a diagnostic application for identification of a disease, disorder or condition.

Research study findings result in a screening tool for identification of a disease, disorder or condition.

Public health and prevention

Research study findings lead to identification of measures for disease prevention.

Research study findings lead to identification of risk factors of a disease, disorder, condition or behaviour.

Research study findings lead to public awareness of risk factors of a disease, disorder, condition or behaviour.

Research study findings lead to enhancement of health promotion activities among community members.

Research study findings lead to identification of a lifestyle intervention.

Research study findings lead to eradication of a disease.

Medical device

Clinical trial study testing of a medical device generated by the research study.

Clinical data generated in support of marketing a medical device, 510(k); Investigational Device Exemption, IDE; or Premarket Approval, (PMA) generated by the research study.

Medical device generated by the research study shows benefit during clinical trials.

Medical device generated by the research study registered/licensed with FDA.

RAND Europe

32

Intervention type

Indicators denoting stages of progression

Medical device generated by the research study used by health care providers and/or consumers.

Mobile application

Mobile application developed by the research study is used by health care providers and/or consumers.

Source: Bernard Becker Medical Library, Washington University School of Medicine https://becker.wustl.edu/impact-assessment

One important data source to be aware of in this areas is Federal RePORTER12 which provides comprehensive information on federal investments in science across agencies, including NIH, the US’s national medical research agency. This is part of the work of STAR METRICS,13 an initiative which aims to create a national-level repository of data on the impact of investment in R&D using an automated data infrastructure. At the moment, the majority of the information focuses on research system indicators (e.g. research spend by region), but there are some useful metrics around job creation at a local level, and the intention is to expand the coverage and scope of the data over time to capture information on wider research outcomes.

Although not typically measured in standard research evaluation approaches at the portfolio level, another important element to analyse at the system level will be research integrity, reproducibility and efficiency. This is an area of significant concern, with studies highlighting a ‘reproducibility crisis’ in science (Ioannidis, 2005) and concern around problematic behaviours, such as ‘p-hacking’ (Munafo, 2017) and failure to comply with authorship norms (Ioannidis, 2018), and suggestions that publishing pressures within the academic community are compromising research integrity (Grimes et al., 2018) and team science (Anderson et al., 2007). Closely linked to this is the discussion around research waste and the efficiency of the scientific system. Ten years on from their estimation that 85 per cent of effort in medical research is wasted (Chalmers and Glasziou, 2009), Glasziou and Chalmers (2018) note that although progress has been made, still a large proportion of research is either not published (Ross et al., 2012), has avoidable design flaws (Yordanov et al., 2015), or is unusable or incompletely reported (Glasziou et al., 2014). Responding to these concerns about integrity and waste, Heneghan et al. (2017) have set out a manifesto for evidence-based medicine which comprises the following points:

• Expand the role of patients, health professionals, and policy makers in research

• Increase the systematic use of existing evidence

• Make research evidence relevant, replicable, and accessible to end users

• Reduce questionable research practices, biases, and conflicts of interest

• Ensure drug and device regulation is robust, transparent and independent

• Produce better usable clinical guidelines

• Support innovation, quality improvement and safety through the better use of real-world data

12 https://federalreporter.nih.gov/ 13 https://www.starmetrics.nih.gov/

https://becker.wustl.edu/impact-assessment

https://federalreporter.nih.gov/

https://www.starmetrics.nih.gov/


33

• Educate professionals, policy makers and the public in evidence-based healthcare to make an informed choice

• Encourage the next generation of leaders in evidence-based medicine.

Measuring the reproducibility of research and research integrity is not straightforward. There are, however, some indicators and measurement approaches identified and used in the literature, examples of which are set out in Table 3.6.

Table 3.6 Possible indicators to measure level of research waste and integrity in research

Issues addressed Measurement approaches and indicators

Research addresses questions relevant to clinicians and patients

Proportion of primary research studies which are funded based on a systematic review of existing evidence (Lund et al., 2016; Chalmers et al., 2014)

Engagement of research users in research prioritisation processes (Chalmers et al., 2014)

Research has appropriate design and methods

Proportion of studies for which protocols and analysis plans are published at study inception (Chalmers et al., 2014)

Analysis of risk of bias in trial designs (Yordanov et al., 2015)

Proportion of primary research studies which are designed based on a systematic review of existing evidence (Lund et al., 2016)

Research fully published and accessible

Proportion of registered trials published (Ross et al., 2012)

Proportion of studies with raw data and analytical algorithms publicly available within six months after publication of a study report (Ioannidis et al., 2014)

Research report is unbiased and useable

Proportion of publications without conflicts of interest, as attested by declaration statements and then checked by reviewers (Ioannidis et al., 2014)

Proportion of research studies undergoing rigorous independent replication and reproducibility checks, and proportion replicated and reproduced (Ioannidis et al., 2014)

Source: RAND Europe analysis building on work of Chalmers and Glasziou (2009)

3.3. Translational research

Translational research is a crucial element of the innovation ecosystem. Though basic biomedical science is funded with the primary intention of ultimately improving health, the route to achieving this is lengthy and challenging (Drolet and Lorenzi, 2011). For example, one study showed that less than 25 per cent of highly promising biomedical discoveries led to the publication of a Randomized Control Trial (RCT) and under 10 per cent were implemented into routine clinical practice within 20 years (Contopoulos-Ioannidis et al., 2003) – reflecting previous observations that even for those ideas that do complete the journey from bench to bedside, it will take on average 17 years to do so (Morris et al., 2011). It is also costly – translating a research discovery into clinical practice can cost billions of dollars (Herper, 2014).

Efforts have been made to measure the success of translational research processes and programmes, and though many metrics have been developed (e.g. Pozen and Kline, 2011; Dembe et al., 2014; IoM, 2013), these are often programme- or project-specific, and there is limited consensus regarding the best ways to

RAND Europe

34

assess translational research performance (Grether et al., 2014). In a comprehensive review, Dembe et al. (2014) summarise the key indicators for the performance of translational research, developing the ‘Translational Research Impact Scale (TRIS)’ underpinned by a logic model-based conceptual framework. The indicator set developed, based on their review of 26 prior studies looking at translation research programmes, is set out in detail in Annex C and in large part parallels some of the key indicators identified in Table 3.7. Although many of the indicators developed in the TRIS model are indeed relevant and desirable to the biomedical ecosystem, the approach to operationalising and collecting data for some of these indicators is unclear and may be challenging. We highlight in Table 3.7 some of the more relevant indicators from the TRIS model, focusing specifically on translation and on patient-relevant metrics.

Table 3.7 A sample of novel and relevant metrics from the Translation Research Impact Scale

Rese

arch

-rel

ated

impa

cts

Improved methods for recruitment of study participants are developed and/or implemented

Institutional Review Board (IRB) processes are improved (e.g. shortening the time to approval, ease of completion, and appropriateness of decisions)

There is an increased emphasis on conducting practice-based research (in ‘‘real-world’’ clinical settings)

New or improved methods for conducting studies spanning multiple disciplines are established

New methods of synthesising results from varying disciplines are implemented

New ways of measuring outcomes from translational research studies are adopted

The number of publications authored or co-authored by translational researchers increases

The average impact factor of journal articles authored or co-authored by translational researchers increases

The participation of translational researchers in making presentations at national conferences increases

Tran

slat

iona

l im

pact

s

The effectiveness of different treatment and intervention choices is determined in a manner that is useful for clinicians

Improvements result in better quality of patient care

The incidence of medical errors decreases

Increased training in translational research methods occurs among health care providers and support personnel

Health information technologies are enhanced

Medical practice becomes more evidence-based

Patient outcomes improve

Positive health behaviours expand (within particular populations)

The patient–clinical relationship is demonstrably strengthened

The translation of research findings into clinical practice and outcomes accelerates

Soci

etal

impa

cts Policies and procedures for protection of human subjects are strengthened

Clinical and translational science is conducted in a manner consistent with recognised ethical principles

Community-based health programmes are developed

There are improved reimbursement practices and policies for providers

Self-efficacy and empowerment among health care consumers increase


35

Improvements are made in health literacy of consumers and specific populations

The health status of consumers and specific populations is improved

Organisational coordination raises awareness and stimulates enhanced community programming concerning health issues

New knowledge regarding sustainable development helps to promote and protect the health of communities

Consumer understanding of and support for translational research increase

Consumers feel more empowered about health issues and their ability to affect change

There is improved communication and understanding about health risks among consumers

Health education and health literacy expand

Source: Dembe et al. (2014)

Given that the aim of translational research is to move research across the translational continuum from the bench towards the bedside rather than looking at outcomes specifically, several studies use measurement approaches which emphasise the progress along the translation pathway. This requires markers of progress to be determined and an assessment made of the level of progress, or the time taken for progress between them. Markers used in the four relevant studies are summarised in Table 3.8.

Table 3.8 Examples of ‘process markers’ denoting progress in translation

Source Description Process markers

Hanney et al. (2015) based on process marker model (Trochim et al., 2011)

In a series of detailed case studies, translation processes were mapped against ‘research tracks’, with key points analysed as ways of calculating the time lags associated with different stages in the process

Discovery

First in human (phase 1)

Dosage/design research (phase 2)

Efficacy research (phase 3)

Effectiveness/post-launch research

Research review and synthesis

Regulatory approval/first non-research use in patients

National policy announcement/guidelines/advice

Reimbursement/financial support in health system

Considered standard practice

Marjanovic et al.. (2015)

In an evaluation of the i4i programme, a translational research programme funded by the UK government, translational points (primarily for medical devices) were defined and project-mapped against them through a survey of researchers

Proof of concept (i.e. conceptual phases of project only, usually lab-based)

Prototype development

Prototype completion (i.e. model or process outline or gadget developed in lab)

Testing or Pivotal Clinical Trial (i.e. preliminary evaluation and testing of the product on small number of patients or clinical settings)

Large scale Clinical Trial

Commercialisation

RAND Europe

36


Finalised commercial/business plan

Finalised IP position/IP arrangements/ownership arrangements

Started a company based on the outputs from i4i-funded work

Licensed out/ensured the downstream development of the product by another company

Product on the market/product available for use

Product adopted in practice

Uptake in the health system (National Health Service i.e. NHS in this case)

Johnson et al., 2010

In a workshop aiming to develop metrics for the performance of Clinical and Translational Science Award (CTSA) sites collaboratively with industry, a number of metrics used to assess the timeliness of clinical trial performance by pharmaceutical companies were discussed. These provide more specific metrics for the timeliness and effectiveness of the delivery of a specific element of the translational process14.

Time from protocol approval, by the regulatory sponsor, to the date of the first visit of the first subject

Percentage of clinical trials that fail to meet expected completion dates

Proportion of trials yet to recruit first subject

Proportion of trials which under-enrol

Patient enrolment rates

Time from protocol approval by pharmaceutical company to the receipt of a signed contract from recruiting site

Time from first screened patient to last screened patient

Proportion of study participants remaining in study until completion

Engel et al. (n.d.)

Building on the military R&D concept of ‘technology readiness levels’, or TRLs, to develop a parallel, termed knowledge readiness levels (KRLs), for use across different lines of health research. KRLs are defined using a 9-point scale across 3 stages: 1-3 foundational research; 4-6 application to human subjects; and 7-9 real-world application. Example given shows levels for drugs and biologics.

1 Review of scientific knowledge base

2 Development of hypotheses and experimental designs

3 Target/candidate identification and characterisation of preliminary candidate(s)

4 Candidate optimisation and [non–good laboratory practice] in vivo demonstration of activity and efficacy

5 Advanced characterisation of candidate and initiation of Good Manufacturing Practice (GMP) process development

6 GMP pilot lot production, [investigational new drug] submission, and phase 1 clinical trial(s)

7 Scale-up, initiation of GMP process validation, and phase 2 clinical trial(s)

8 Completion of GMP validation and consistency lot manufacturing, pivotal animal efficacy studies or clinical trials,

14 A comparable example looking at the implementation of clinical guidelines in hospitals (Lowson et al., 2015) used two metrics: lapsed time between the issuing of new guidance and disseminating the guidance through contact with the person responsible for implementing the guidance; and lapsed time between initial contact and a locally-judged satisfactory response.


37


and [U.S. Food and Drug Administration] approval or licensure

9 Post-licensure and post-approval activities

Sources: Hanney et al. (2015), (Trochim et al., 2011), Marjanovic et al. (2015), Johnson et al., 2010, and Engel et al. (n.d.)

Another approach drawing on the literature on implementation research is to look at the characteristics of an intervention in terms of its adoption-readiness as metrics of the performance of research. Drawing on examples of existing practice, Proctor et al. (2011) identify seven characteristics that denote the status of an intervention which is undergoing translation into practice:

• Acceptability: perception among implementation stakeholders that a given treatment, service, practice, or innovation is agreeable, palatable, or satisfactory.

• Adoption: the intention, initial decision, or action to try or employ an innovation or evidence-based practice. Also referred to as ‘‘uptake’’ (Rabin et al., 2008; Rye and Kimberly, 2007).

• Appropriateness: perceived fit, relevance, or compatibility of the innovation or evidence-based practice for a given practice setting, provider, or consumer; and/or perceived fit of the innovation to address a particular issue or problem.

• Cost (incremental or implementation cost): cost impact of an implementation effort – which will depend not just on the intervention but also on the implementation strategy used and will be location-specific.

• Feasibility: the extent to which a new treatment, or an innovation, can be successfully used or carried out within a given agency or setting (Karsh, 2004).

• Fidelity: the degree to which an intervention is implemented as it was prescribed in the original protocol or as it was intended by the programme developers (Dusenbury et al., 2003; Rabin et al., 2008).

• Penetration: the integration of a practice within a service setting and its subsystems (Stiles et al., 2002; Rabin et al., 2008). Similar to ‘reach’ as used in the RE-AIM framework (Glasgow, 2007; Glasgow et al., 1999).

• Sustainability: the extent to which a newly implemented treatment is maintained or institutionalised within a service setting’s ongoing, stable operations (Johnson et al., 2004; Turner and Sanders, 2006; Glasgow et al., 1999).

3.4. Private sector research

While both the public and private sectors play significant roles in the biomedical innovation ecosystem and related research, the majority of funding for research and development (R&D), particularly in terms of development of innovative drug and treatment discovery, tends to originate within the private sector (Sampat & Lichtenberg 2011, Orianna et al. 2016).

Widely-used measures of private sector innovation in drugs and therapies resulting from such investment tend to include the amount spent on R&D, clinical trials, patent applications submitted, and sales. Each

RAND Europe

38

measure on its own provides an incomplete overview of innovation related to private sector research, though each may serve a different, complementary function, for example, spend on R&D is relatively easy to communicate and compare, and some may be more culturally sensitive than others, for example, patent filing tends to be a more robust measure of innovation in the US than in Europe.15 From the perspective of private companies, particularly in the pharmaceutical sector where research into new drugs is characterised by high expenditure and high levels of risk, the focus on expenditure and return on investment is a useful way to understand trends that are significant in understanding performance and sustainability (Schuhmacher 2016, Said & Zerhouni 2014).

At the same time, degrees of innovation in drugs and treatments may also vary widely, from the introduction of new, ‘breakthrough’ treatments to minor changes to existing therapies (Artz et al. 2010). For example, measures of efficacy and effectiveness of drugs (i.e. the extent to which they out-perform comparative therapies under ideal and then normal conditions), the level of medical need for their development, as explored by Annemans et al. (2012), as well as the level of innovation as measured through priority review and first and second class designation, also help to provide additional information in this regard (See for example, the Association of the British Pharmaceutical Industry (ABPI) report in 2005 which provided competitive and performance indicators on behalf of the Pharmaceutical Industry Competitiveness Task Force i.e. PICTF).

Focus on innovation measures for private sector development of biomedical devices, rather than drugs, is less common. While metrics for measuring innovation in this area may be generally similar to that used for drugs and therapies, such as number of patents filed, a systematic review by Oriana et al. (2016) of innovation definitions and measures for biomedical devices shows that device innovations are more often incremental, and that these incremental gains can be more valuable from a user perspective. The overall value of the innovation may require a more multi-faced approach, which includes cost effectiveness benefits for the ultimate consumer. Common metrics for biomedical innovation are shown in Table 3.9 below.

Table 3.9 Measures of biomedical innovation, focus on the pharmaceutical industry


R&D investment Life Sciences Competitiveness Indicators: Pharmaceutical industry spend on research and development

ABPI: R&D expenditure intensity (R&D expenditure as a % of total sales), Distribution of expenditure along the development cycle/’by phase of R&D’ (research, preclinical, clinical, submission, roll out)

Trialling of new medicines

Life Sciences Competitiveness Indicators: Share of patients recruited to global studies (all trial phases), Time from core package received to first patient enrolled in country (all trial phases)

ABPI: Number of clinical trial applications received by regulator, Clinical trial applications by

15 Interview.


39


phase (proxy for clinical trial activity)

Identification of new therapies and drug launches

PICTF: Proportion of world-first patents filed for marketed New Molecular Entities (NMEs) as a proportion of world R&D spend (measures relative productivity of R&D expenditure)

PICTF: Proportion of NMEs launched (relatively crude measure that does not reflect the importance of the NME), Proportion of companies’ NMEs launched in all ‘top 4 markets’: US, Germany, France, UK (measure of successful output)

ABPI: Number of NMEs and New Molecular Approaches (NMAs) launched worldwide (over a long time-horizon measures pace of innovation), Number of NMEs by active substance type (biopharmaceutical or chemical, shows which technology may be advancing more rapidly)

Significance/level of innovation

Efficacy and effectiveness: Measures of efficacy (extent to which an intervention does more good than harm, under ideal circumstances, as compared with one or more alternative interventions) and effectiveness (extent to which an intervention does more good than harm, under normal circumstances, as compared with one or more alternative interventions), often based on projections (Annemans et al. 2012)

PICTF: Number of NME launches receiving FDA priority review (granted by the FDA to drugs offering a significant improvement), Proportion of companies’ NMEs that were first or second launches in class (indicates the significance of the drug discovery)

Cost effectiveness for the consumer (Oriana et al. 2016)

Sales IDEA Pharma: Global sales (measure of funding available for commercialisation efforts)

Sources: ABPI (2005), Annemans et al. (2012), Oriana et al. (2016), and RAND Europe

3.4.1. Innovation through public and private sector research interactions

Though primarily driven by private sector investments, public sector investment in R&D is also an important dimension of innovation in drug development and has strong linkages to private sector innovation. While some public investment has direct benefits related to drug discovery (e.g. when public funding results directly in drug development), the indirect benefits of public funding to therapeutic innovations are considerably more significant (e.g. public funding for basic science and underlying research that informs innovations funded by the private sector) (Sampat & Lichtenberg 2011, Hasket et al. 2014). For example, in a review of new drugs launched in the United States, Sampat and Lichtenberg (2011) find that the direct and indirect impacts of public investment are widespread in drug development and are most significant in more innovative developments (‘priority review’ drugs16 vs ‘standard review drugs). As shown by Hasket et al. (2014), a fuller understanding of the complexities behind the impact of public sector research investment on that of the private sector, including in terms of skillsets of workforces and up-take of advances in basic scientific knowledge, where public sector funding tends to be more concentrated, is more difficult to secure. Measurement challenges include the long time periods for uptake of basic scientific knowledge (it can be decades), and the requirement for more qualitative review (e.g.

16 Drugs that are given ‘priority review’ by the FDA are often perceived as more innovative, as they apply to those applications that, ‘if approved, would be significant improvements in the safety or effectiveness of the treatment, diagnosis, or prevention of serious conditions when compared to standard applications.’ FDA, N.D.

RAND Europe

40

through surveys investigating why firms set up their headquarters in specific locations, etc. (Haskel et al., 2014).

Partnerships with the public sector and other actors are also becoming increasingly important for innovation within the biomedical sector, particularly for pharmaceuticals. Faced with a slowing rate of development of innovative new products, a large series of upcoming patent expirations, and overall decreasing return on investment for R&D, pharmaceutical companies are increasingly looking towards more open innovation models, including partnerships with universities, foundations, and the public sector, particularly at pre-competitive stages (de Vreuh & Crommelin 2017, Deloitte 2018, and Said & Zerhouni 2014). Various measures may be used to capture the degree of innovation relative to these partnerships, which can be broken down into the input, process, output, and outcome stages. In particular, qualitative measures of less tangible partnership qualities, such as degree of trust between partners, are also an important metric of success (De Vreuh & Crommelin 2017). An overview of the more quantitative metrics is captured in Table 3.10 below.

Table 3.10 Measures of public – private collaboration and interactions


National-level investment in R&D

PICTF: National origins of leading 75 global medicines (worldwide sales and company Head Quarter i.e. HQ)

Life Sciences Competitiveness Indicators: Government spend on health research and development

Other sector investment in R&D

Life Sciences Competitiveness Indicators: Non-industry spend on R&D, Share of life sciences academic citations, Share of top 1% (most cited) life sciences academic citations.

Public sector influence on innovation and drug development

Direct public sector influence: Share of public-sector patents assigned to a government agency or for which there is a government interest statement (acknowledgement of government support to patents that result from government funding) for priority and non-priority review (Sampat & Lichtenberg, 2011)

Indirect public sector influence: Patent citing another government-sector patent or a government publication (Sampat & Lichtenberg, 2011), Relationship between public sector science spending and total factor productivity (TFP) of an in industry (Haskel et al., 2014, see discussion of TFP metric on page 11)

Public Private Partnerships

Number of Public Private Partnerships (PPPs) (Thonton et al. 2010)

Effectiveness of PPPs (De Vreuh & Crommelin 2017):

Inputs: Access to resources, Number and diversity of partners

Processes: Exchange of information between partners, quality of PPP intranet, Relevant and high-quality research, Number of clinical trials and patients included, Number of biological samples

Output: Number of projects continuing after PPP funding, number of co-authored publications, Number of articles with international collaboration, Proportion of long-distance collaborative publication, Partnership Ability Index, Quantity and quality of scientific publications, Number of new technologies (patents and other IP)

Outcome: Attractiveness for foreign investors, Partnership continuation, Number and size of


41


new partnerships or other R&D collaborations, Number of partners joining follow-up projects, Number of new products in clinical development based partially on knowledge generated in PPPs, Number of visits to EXPASY server, New processes, Number and size of new projects, solutions and innovations.

Impacts: Process impacts (cost and time savings, reduction in risk, reduction in attrition rate, reduced need of animal testing), New product development (new medicines to market, other products to market), Industry impacts (new businesses, growth of existing businesses), Health systems benefits (improved access to new treatments, more effective use of healthcare budgets or reduced costs), Policy (new policies, clinical guidelines, regulations of legislation changed), Licence revenues, New product effectiveness, Financial performance.

Sources: De Vreuh & Crommelin (2017), Haskel et al. (2014), Sampat & Lichtenberg (2011), Thonton et al. (2010), and RAND Europe

3.5. Regulation and patient access

The regulatory system impacts biomedical innovation through setting standards for drug and device approval, clinical trial design, manufacturing, and the publication of guidance documents for clinical staff and patients. New models of regulatory approval, such as the use of real-world evidence and adaptive licensing for new drugs, have the potential to facilitate patient access to new medications (Eichler et al. 2015; Schneeweiss et al. 2016); however, the benefits of speeding access to innovative new medicines must be carefully weighed against the potential safety risks of introducing drugs to market with more limited evidence behind them (Davis, Lexchin, Jefferson, Gøtzsche, & McKee 2016).

Although there is a wealth of literature addressing the drug approval process and potential changes to facilitate patient access to biomedical innovation (see for example Bouvey, Jonsson, Longson, Crabb & Garner 2016; Homes et al. 2018; Schulthess, Chlebus, Bergström & Van Baelen 2014), we found no clear indices or frameworks addressing the impact of the regulatory system on the biomedical innovation ecosystem or patient access to biomedical innovation. However, we identified some individual indicators that could provide a partial accounting of these characteristics. Considering changes over time in any of the indicators would present a fuller picture of whether the United States is experiencing improvement in regulation and patient access to biomedical innovation. The indicators fall into three categories: process measures; development of new standards, tools, and methods; and patient safety.

• Process measures: Indicators of the performance of the regulatory system derive largely from two sources: official oversight bodies of the regulatory agencies and pharmaceutical industry associations.17 These metrics tend to emphasise process measures, such as the number of

17 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019. Pharmaceutical Industry Competitiveness Task Force. (2006). Competitiveness and Performance indicators 2005: http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf

https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index


http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf


RAND Europe

42

approvals, time to approval, or number of meetings held with the pharmaceutical industry, that indicate whether the agency is meeting performance standards for key regulatory processes. These measures can be useful for understanding the impact of regulation regarding biomedical innovation, especially if refined measures that capture approvals of novel medications or investigational new drugs (INDs) are included.

• Developing new standards, tools and methods for the regulation and evaluation of innovations: One role of regulatory bodies is to develop new methods and tools for oversight, such as the previously mentioned adaptive licensing models (Eichler et al. 2015). They also routinely work with pharmaceutical companies on developing best practices for conducting and analysing clinical trials, practices which are continually evolving to keep pace with innovations in science and medicine. Indicators on the development of new standards, tools, and methods are routinely collected by oversight agencies, but although this is a key aspect of the system’s impact on biomedical innovation, the metrics that are published offer little insight into how these elements affect biomedical innovation. The metrics again reflect process concerns, and measure elements such as the number of publications produced or conferences attended. Some of the metrics on research performance detailed in section 3.2 could potentially be relevant for assessing this aspect of the regulatory environment, in particular, metrics that focus on the impact of research on practice (Table 3.4), or the engagement of the scientific community with the research, such as that detailed in Lavis et al. (2003).

• Patient safety: Measures of patient safety provide indications of how the system functions to ensure safety throughout the approval and post-approval process, including the review and approval of drugs and devices, oversight of manufacturing, and decisionmaking among healthcare professionals. The safety of the biomedical system and that of the approval, manufacturing, and delivery process is reflective of the quality of the public’s access to biomedical innovation.

Ideally, a scorecard, such as the one proposed by FasterCures, would balance measures of processes and patient safety in order to capture the tension that can exist between the push for speed and efficiency on the one hand, and quality and safety on the other. We summarise the existing indicators for measuring the impact of the regulatory system on biomedical innovation in Table 3.11 below.

Table 3.11 Existing indicators for measuring regulation of biomedical innovation ecosystem


Process measures FDA Center for Drug Evaluation and Research (CDER).18 Cumulative approvals of drugs and biologics, Number of approvals and tentative approvals of novel drugs, Number of new commercial investigational drugs (INDs) received, Number of industry-requested meetings scheduled with FDA CDER.

18 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019.




43


PICTF measures.19 Overall time from first protocol submission to final medicines regulatory approval, Proportion of studies approved by research ethics committees without deferral, Average time from first world application for market authorisation to approval in the particular market.

Developing new standards, tools and methods for regulation and evaluation of innovations

FDA CDER.20 Number of newly published policy documents, Number of advisory committee meetings held, Number of publications and conference presentations.

Patient safety FDA CDER.21 Number of adverse drug events by severity, Number of drug recalls, Number of drug shortages prevented, Number of ongoing drug shortages, Number of new drug shortages.

Source: RAND Europe analysis

Patient access to biomedical innovation is sometimes framed as an issue of supply and demand (Ensor & Cooper 2004); however, equity of access to innovation is an important consideration from a social justice and system perspective. Health services researchers look beyond basic market access for innovative new drugs and devices and examine barriers to uptake of innovation, such as regional socioeconomic and demographic characteristics (Groeneveld, Laufer & Garber 2005) and insurance type (Epstein, Ketcham, Rathore, & Groeneveld 2012).

The existing indicators measuring patient access to biomedical innovation may be grouped into three categories: supply of biomedical innovations; demand for biomedical innovations; and equity of access. Table 3.12 presents the patient access indicators identified in the literature and through expert interviews.

Table 3.12 Existing and potential indicators for measuring patient access to biomedical innovations


Supply of biomedical innovations

PICTF measures.22 Number of novel drugs (NMEs) approved

Interviews: time to market, as measured by time from first paper publishing the target to approval of effective therapy in any market.

FDA CDER.23 Number of drug shortages prevented, number of ongoing drug shortages, number of new drug shortages.

19 Pharmaceutical Industry Competitiveness Task Force. (2006). Competitiveness and Performance indicators 2005: http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf 20 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019. 21 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019. 22 Pharmaceutical Industry Competitiveness Task Force. (2006). Competitiveness and Performance indicators 2005: http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf 23 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019.









RAND Europe

44


Demand for biomedical innovations

PICTF measures.24 Number of units sold 1 and 3 years after launch, % by value of national pharmaceutical market accounted for by NMEs launched within last five years

Equity of access Indicators from health services research: Access to biomedical innovations by insurance type,25 demographic groups and socioeconomic characteristics.26

ABPI.27 Share of patients recruited to global clinical research studies, by disease type

Used by the FDA CDER.28 Number of and time to approval for generics

Source: RAND Europe analysis.

24 Pharmaceutical Industry Competitiveness Task Force. (2006). Competitiveness and Performance indicators 2005: http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf

25 Epstein et al. (2012).

26 Nathan et al. (2019).

27 The Association of the British Pharmaceutical Industry. 2019. Science and Innovation. http://www.abpi.org.uk/facts-and-figures/science-and-innovation/

28 US Food & Drug Administration. (March 31, 2019). FDA-TRACK: CDER Index. https://www.fda.gov/about-fda/fda-track-agency-wide-program-performance/fda-track-cder-index. Accessed June 27, 2019.


http://www.abpi.org.uk/facts-and-figures/science-and-innovation/

http://www.abpi.org.uk/facts-and-figures/science-and-innovation/



45

4. Discussion

The biomedical innovation ecosystem is a complex and multifaceted concept. Many models of the ecosystem, or elements of it, take a linear characterisation. Though this can be useful for understanding processes within the ecosystem, a systems-led approach, which takes into account the multiple and complex interactions between actors and within the system, may better characterise and assist in assessing the health and development of the overall biomedical innovation ecosystem. Building on both the linear model and the concepts set out by Baxter et al. (2013) and Wagner et al. (2018a; 2018b) in the NETS and 4DM frameworks, RAND Europe proposes the below conceptualization, highlighted in Figure 4.1, of the biomedical innovation ecosystem. This conceptualization provides a customised high-level overview of key actors and their interactions within the system, based on the literature review undertaken as part of this study, in order to help frame findings and facilitate identification of a fuller range of indicators to cover the system.

This will require further development and refinement by FasterCures, reflecting on their aims and priorities, but hopefully will help to stimulate an initial discussion on the scope and perspective they intend to take regarding the ecosystem, which will guide identification and selection of appropriate indicators.

RAND Europe

46

Figure 4.1 Initial conceptualization

Source: RAND Europe

In developing a scorecard, a key consideration will be to ensure a balanced set of indicators is developed, reflecting the different elements of the ecosystem as set out in the framework or conceptualisation used. Any one indicator in isolation provides little useful information, and placing too much emphasis on a limited set of indicators can provide misleading information and potentially drive undesirable behaviours. Developing a holistic and balanced set of indicators across elements of the ecosystem prioritised by FasterCures will be crucial to the success of their work.

It is also important to note that, given the complex, non-linear nature of any ecosystem, measurement of good performance across a set of indicators is no guarantee of desired outcomes. The complex interaction of factors and the role of the wider social, economic and political context mean that putting in place all of these elements may not be sufficient to deliver a step change in prevention and treatment. However, identifying key characteristics of an effective system, and measuring (and improving) performance on these characteristics, can be expected to improve the likelihood of achieving these desired aims.

Another type of balance to consider in the development of a set of metrics is the relative role of such quantitative indicators by comparison to qualitative information. Not everything one wants to achieve in relation to the ecosystem may be readily quantified, and in any case qualitative evidence – particularly narrative information – can be powerful in communicating with diverse audiences. Also important are


47

practical considerations regarding the quality and availability of data. Even if something can be quantified, it may not have been so far, and the resources required for primary data collection can be significant.

Table 4.1 summarises the different indicator types and domains identified in this paper, integrating measures identified in different areas of the literature where there are similarities in indicator type or intent. An important next step for FasterCures will be to identify which areas and which indicator types are in scope, and of those in scope, which are the highest priority for the scorecard in terms of their relevance to the project’s goals. Based both on this assessment of relevance, informed by the systems diagram above, and practical considerations regarding data availability and resources for data collection and analysis, FasterCures can start to prioritise which metrics to explore and develop further.

Table 4.1 Overview of indicators identified and issues they address

Issues addressed

Types of measures and examples

Productivity and capacity of the academic research system

Measures of research productivity: e.g. number and quality of outputs including but not limited to publications; new research tools and resources generated; relevance of research conducted; targeting of future research

Level of investment: e.g. government spend on health research and development, funding received in different areas

Measures of research capacity: e.g. number and quality of researchers trained, collaboration and networking (academic and wider), career outcomes for researchers and students trained

Improved study designs and methods: e.g. improved methods for recruitment of study participants, improved Institutional Review Board (IRB) processes, improved approaches to cross-disciplinary working

Integrity of academic research system

Relevance: does research address questions relevant to clinicians and patients (e.g. proportion of primary research studies which are funded based on a systematic review of existing evidence)

Rigour: does research have appropriate design and methods (e.g. proportion of studies for which protocols and analysis plans are published at study inception)

Access: is research fully published and accessible (e.g. proportion of registered trials published)

Usability: Research reports are unbiased and useable

Ethical oversight: Improvements in ethical conduct of research and human subjects’ protection

Productivity and capacity of the private sector research system

Spend: Measures of pharmaceutical/other relevant industry spend on R&D

Significance/level of innovations developed: e.g. measures of efficacy and effectiveness of new interventions; number of NME launches receiving FDA priority review (granted by the FDA to drugs offering a significant improvement); Cost effectiveness for the consumer.

Global performance: e.g. global sales; National origins of leading 75 global medicines; profit and revenues

Interaction with public sector: e.g. number and effectiveness of PPPs; public sector influence on innovations in drug development; industry/institute co-publications or co-patents

Measures of innovative/ economic

New products and process developed (and patented, licensed, used); Proportion of NMEs launched and scope of launch (internationally); venture capital access

Outcomes for companies: New businesses developed and benefits to existing companies in terms

RAND Europe

48

Issues addressed


outputs of profitability, new clients, competitive advantage, efficiency etc.; % of sales revenues from new products/services

Economic outcomes: Job creation, workforce development and increased economic competitiveness of region/country

Measures of progress and efficiency in translation

Process markers: Different points in the translational process which can be used as markers to assess the level of progress, or the time for progress between them: (e.g. discovery, proof of concept, prototype development). Can also be used to measure effectiveness and efficiency of specific elements of the process (e.g. percentage of clinical trials that fail to meet expected completion dates, or time from first screened patient to last screened patient)

Supply of biomedical innovations: e.g. number of novel drugs approved; number of drug shortages

Demand for biomedical innovations: e.g. number of units sold 1 and 3 years after launch, % by value of national pharmaceutical market accounted for by NMEs launched within last five years

Regulatory process measures: e.g. overall time from first protocol submission to final medicines regulatory approval, proportion of studies approved by research ethics committees without deferral, average time from first world application for market authorisation to approval in market.

Policy and regulatory processes

Engagement: Interaction of researchers with policy and decisionmakers through invitations, meetings, committees.

Impact on policy: Citation of research in, or other influence on, policy documents including guidelines; resulting changes in policy and improvement in public services (including outside of health - e.g. STEM education);

Regulatory processes and approvals: Cumulative approvals of drugs and biologics, number of approvals and tentative approvals of novel drugs, number of new commercial investigational new drugs (INDs) received, number of industry-requested meetings scheduled with FDA CDER.

Health system changes and processes

Clinician engagement and relationships: Increased clinician engagement in research; more evidence-based practice; improved patient-clinician relationships

Clinical practice: Impact on professional training or development in health; impact on practice including efficiency/cost savings; implementation of new interventions; practice reflects evidence-based guidelines; making processes more efficient or resilient etc.;

Impact on health and well-being (of a group of patients, or more widely)

Patient safety and quality of care

Patient safety: Number of adverse drug events by severity, number of drug recalls, number of drug shortages prevented, number of ongoing drug shortages, number of new drug shortages.

Improvements in patient safety, quality of care and outcomes (e.g. Patient report Outcome Measures i.e. PROMs, QALY, Disability Adjusted Life Year (DALY) measures, outcome measures by condition, mortality and morbidity)

Community engagement and equity

Equity of access: Access to biomedical innovations by insurance type, demographic groups and socioeconomic characteristics; share of patients recruited to global clinical research studies, by disease type and other characteristics; number of and time to approval for generics; improved reimbursement practices and policies for providers

Community engagement and empowerment: Number and range of dissemination and outreach activities; increased public understanding of and engagement with science and research (e.g.


49

Issues addressed


participation in clinical trials); more positive attitudes towards research and researchers; improved health literacy and empowerment of health care consumers

Population health improvements (reflecting improvements in prevention and health promotion alongside treatment)

Source: RAND Europe analysis

One important consideration in the prioritisation process will be the appropriate balance between process and output orientation of the indicators selected, depending on the nature of the system change you wish to drive. For example, traditional metrics for academic research focus on outputs – for example numbers and quality of publications, other outputs such as patents, or capacity-generating outputs such as PhDs trained. However, it may be that metrics related to process, particularly in terms of research integrity and reproducibility, merit more attention and may be equally relevant to the overall health of the biomedical innovation ecosystem.

It is also worth exploring whether there might be opportunity for synergies with other efforts in this area (e.g. FDA TRACK, Federal RePORTER). By working with ongoing efforts, FasterCures can maximise the use of their resources, and also increase their scope for impact by differentiating this work from existing exercises.

A final key point to note is that any set of indicators will have a limited shelf life. Once a framework and set of metrics is developed, work is not complete. This selection of metrics will need continual review and monitoring. As the ecosystem changes and evolves, some measures may become less relevant, and other new priorities and corresponding metrics may emerge. New data collection efforts may enable new metrics to be introduced, as may emerging technologies for data capture and analysis – for example new developments in text mining and data analytics. To maintain the relevance and applicability of the scorecard, regular reviews and updates will be required, reflecting on the changing nature of not just the research and innovation landscape, but also emerging evaluation approaches.

51

References

Aase, G., Roth, E., & Swaminathan, S. (2018, April). Taking the measure of innovation with conversion

metrics. McKinsey Quarterly. Retrieved 25 June 2019, from https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/taking-the-measure-of-innovation

Adams, R., Bessant, J., & Phelps, R. (2006). Innovation management measurement: A review. International Journal of Management Reviews, 8(1), 21–47.

Adner, R. (2006). Match your innovation strategy to your innovation ecosystem. Harvard Business Review, 84(4), 98. Retrieved 21 June 2019, from https://hbr.org/2006/04/match-your-innovation-strategy-to-your-innovation-ecosystem

Adner, R., & Kapoor, R. (2010). Value Creation in Innovation Ecosystems: How the Structure of Technological Interdependence Affects Firm Performance in New Technology Generations. Strategic Management Journal, 31(3), 306–333. https://doi.org/10.1002/smj.821

Adner, R., & Kapoor, R. (2016). Innovation ecosystems and the pace of substitution: Re-examining technology S-curves. Strategic Management Journal, 37(4), 625–648. https://doi.org/10.1002/smj.2363

Altmetric (homepage). Retrieved 1 August 2018, from https://www.altmetric.com/

Anderson, M. S., Ronning, E. A., De Vries, R., & Martinson, B. C. (2007). The Perverse Effects of Competition on Scientists’ Work and Relationships. Science and Engineering Ethics, 13(4), 437–461. https://doi.org/10.1007/s11948-007-9042-5

Annemans, L., Cleemput, I., Hulstaert, F., Simoens, S., (2012). Comparative effectiveness research and measuring the level of pharmaceutical innovation in the EU. Journal of Comparative Effectiveness Research, 1 (1). DOI: 10.2217/cer.11.1

Anthony, S. D. (2013, March 21). How to Really Measure a Company’s Innovation Prowess. Harvard Business Review. Retrieved 20 August 2019, from https://hbr.org/2013/03/how-to-really-measure-a-compan

Arnold, E. (2012). Understanding long-term impacts of R&D funding: The EU framework programme. Research evaluation, 21(5), 332–343.

https://www.mckinsey.com/business-functions/strategy-and-corporate-finance/our-insights/taking-the-measure-of-innovation

https://hbr.org/2006/04/match-your-innovation-strategy-to-your-innovation-ecosystem

https://hbr.org/2006/04/match-your-innovation-strategy-to-your-innovation-ecosystem

https://doi.org/10.1002/smj.821

https://doi.org/10.1002/smj.2363

https://www.altmetric.com/

https://doi.org/10.1007/s11948-007-9042-5

https://hbr.org/2013/03/how-to-really-measure-a-compan

https://hbr.org/2013/03/how-to-really-measure-a-compan

RAND Europe

52

Artz, K.W., Norman, P.M. Hatfield, D.E., and Cardinal, L.B., (2010). A longitudinal study of the impact of R&D, patents, and product innovation on firm performance. The Journal of Product Innovation Management (27). https://onlinelibrary.wiley.com/doi/full/10.1111/j.1540-5885.2010.00747.x

Association of the British Pharmaceutical Industry (ABPI), (2005). Pharmaceutical Industry Competitiveness Task Force (PICTF): Competitiveness and Performance Indicators 2005: Retrieved 27 June 2019, from http://www.abpi.org.uk/media/1220/competitiveness-task-force.pdf.

Association of the British Pharmaceutical Industry (ABPI), (2019). Science and innovation: Retrieved 27 June 2019, from https://www.abpi.org.uk/facts-and-figures/science-and-innovation/

Atun R, de Jongh T, Secci F, Ohiri K, Adeyi O. (2010). Integration of targeted health interventions into health systems: a conceptual framework for analysis. Health Policy Plan, 25(2):104 –11.

Atun RA, Kyratsis I, Jelic G, Rados-Malicbegovic D, Gurol-Urganci I. (2007). Diffusion of complex health innovations — implementation of primary health care reforms in Bosnia and Herzegovina. Health Policy Plan, 22(1):28 –39.

Aymerich, M., Carrion, C., Gallo, P., Garcia, M., López-Bermejo, A., Quesada, M., & Ramos, R. (2012). Measuring the payback of research activities: A feasible ex-post evaluation methodology in epidemiology and public health. Social Science & Medicine, 75(3), 505–510. https://doi.org/10.1016/j.socscimed.2012.03.044

Balas, E. Boren, S. (2000). ‘Managing Clinical Knowledge for Health Care Improvement.’ In: van Bemmel JH, McCray AT, eds. Yearbook of Medical Informatics. Stuttgart: Schattauer Verlagsgesellschaft mbH, 65–70.

Banzi, R., Moja, L., Pistotti, V., Facchini, A., and Liberati, A. (2011). Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Research Policy and Systems, 9(26). http://dx.doi.org/10.1186/1478-4505-9-26

Bauman, A. E., Nelson, D. E., Pratt, M., Matsudo, V., & Schoeppe, S. (2006). Dissemination of Physical Activity Evidence, Programs, Policies, and Surveillance in the International Public Health Arena. American Journal of Preventive Medicine, 31(4, Supplement), 57–65. https://doi.org/10.1016/j.amepre.2006.06.026

Baxter, K., Horn, E., Gal-Edd, N., Zonno, K., O’Leary, J., Terry, P. F., and Terry, S. F. (2013). ‘An End to the Myth: There Is No Drug Development Pipeline.’ Science Translational Medicine, 5 (171): 171cm1-171cm1. https://doi.org/10.1126/scitranslmed.3003505

Bennett, S., Paina, L., Ssengooba, F., Waswa, D., and M’Imunya, J.M. (2013). The impact of Fogarty International Center research training programs on public health policy and program development in Kenya and Uganda. BMC Public Health, 13:1–11. http://dx.doi.org/10.1186/1471-2458-13-770

Bernard Becker Medical Library (n.d.), Assessing the impact of research, Retrieved 20 August 2019, from https://becker.wustl.edu/impact-assessment

https://onlinelibrary.wiley.com/doi/full/10.1111/j.1540-5885.2010.00747.x

https://onlinelibrary.wiley.com/doi/full/10.1111/j.1540-5885.2010.00747.x



https://www.abpi.org.uk/facts-and-figures/science-and-innovation/

https://doi.org/10.1016/j.socscimed.2012.03.044

http://dx.doi.org/10.1186/1478-4505-9-26

https://doi.org/10.1016/j.amepre.2006.06.026

https://doi.org/10.1126/scitranslmed.3003505

http://dx.doi.org/10.1186/1471-2458-13-770

https://becker.wustl.edu/impact-assessment


53

Bone, F., Hopkins, M. M., Rafols, I., Tang, P., Molas-Gallart, J., Davey, G., and Carr, A. M. (2017). User Guide to DARE: A Diversity Approach for Research Evaluation, Retrieved 20 August 2019, from http://www.sussex.ac.uk/spru/research/dare/data/DARE_User_Guide_2017.pdf.

Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for Information Science and Technology, 64(2), 217-233.

Bouvy, J. C., Jonsson, P., Longson, C., Crabb, N., & Garner, S. (2016). Health technology assessment in the context of adaptive pathways for medicines in Europe: challenges and opportunities. Clinical Pharmacology & Therapeutics, 100(6), 594-597.

Bowen S, Zwi AB. (2005). Pathways to ‘evidence-informed’ policy and practice: a framework for action. PLoS Med 2(7):e166.

Bruns, A. (2013). R&D: An Ecosystem Evolves. Site Selection Online. Retrieved 29 June 2019, from Site Selection website: https://siteselection.com/LifeSciences/2013/nov/research-and-development.cfm

Brusoni, S., & Prencipe, A. (2013). The Organization of Innovation in Ecosystems: Problem Framing, Problem Solving, and Patterns of Coupling. In R. Adner, J. E. Oxley, & B. S. Silverman (Eds.), Collaboration and Competition in Business Ecosystems (Vol. 30, pp. 167–194). Bingley: Emerald Group Publishing Ltd.

Bunn, F., Trivedi, D., Alderson, P., Hamilton, L., Martin, A., and Iliffe, S. (2014). The impact of cochrane systematic reviews: a mixed method evaluation of outputs from Cochrane Review Groups supported by the UK National Institute for Health Research. Systematic Reviews, 3(125). http://dx.doi.org/10.1186/2046-4053-3-125

Buxton M, Hanney S, Morris S, Sundmacher L et al. (2008). Medical Research: What’s it worth? London: UK Evaluation Forum.

Buxton, M., and Hanney, S. (1996) How can payback from health services research be assessed? Journal of Health Services Research & Policy, 1(1):35–43.

Canadian Academy of Health Sciences (CAHS). (2009). Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa, ON: CAHS.

Canadian Health Services and Policy Research Alliance (CHSPRA), Impact Analysis Working Group (2018). Making an impact: A shared framework for assessing the impact of health services and policy research on decision-making. Ottawa (ON): CHSPRA.

Carayannis, E. G., & Campbell, D. F. J. (2009). ‘Mode 3’ and ‘Quadruple Helix’: Toward a 21st century fractal innovation ecosystem. International Journal of Technology Management, 46(3–4), 201–234. https://doi.org/10.1504/IJTM.2009.023374

Chalmers I, Glasziou P. (2009). Avoidable waste in the production and reporting of research evidence. Lancet, 374(9683):86-9.

Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P.A., and Oliver, S. (2014). How to increase value and reduce waste when

http://www.sussex.ac.uk/spru/research/dare/data/DARE_User_Guide_2017.pdf

https://siteselection.com/LifeSciences/2013/nov/research-and-development.cfm

https://siteselection.com/LifeSciences/2013/nov/research-and-development.cfm

http://dx.doi.org/10.1186/2046-4053-3-125

https://doi.org/10.1504/IJTM.2009.023374

RAND Europe

54

research priorities are set. The Lancet, 383(9912), 156–165. https://doi.org/10.1016/S0140-6736(13)62229-1

Chen, H. (2014). The supporting system of the science park: Case study of Tus Park. Proceedings of the 2014 World Technopolis Association International Training Workshop, 241–248. Daejeon, Republic of Korea.

Chesbrough, H., Kim, S., & Agogino, A. (2014). Chez Panisse: Building an Open Innovation Ecosystem. California Management Review, 56(4), 144–171. https://doi.org/10.1525/cmr.2014.56.4.144

Connell, J.P. and Kubish, A.C. (1998). Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects, and problems. The Aspen Institute.

Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. A. (2003). Translation of highly promising basic science research into clinical applications. The American Journal of Medicine, 114(6), 477–484. https://doi.org/10.1016/S0002-9343(03)00013-5

Cordero, R. (1990). The measurement of innovation performance in the firm: An overview. Research Policy, 19(2), 185–192.

Davis, C., Lexchin, J., Jefferson, T., Gøtzsche, P., & McKee, M. (2016). ‘Adaptive pathways’ to drug authorisation: adapting to industry?. BMJ, 354, i4437.

de Vrueh, R.L.A., and Crommelin, J.A., (2017). Reflections on the Future of Pharmaceutical Public-Private Partnerships: From Input to Impact. Pharmaceutical Research, 34 (10). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5579142/

Deloitte Access Economics (2011). Returns on NHMRC funded Research and Development. Canberra: Deloitte Access Economics.

Deloitte, (2018). Embracing the future of work to unlock R&D productivity: Measuring the return from pharmaceutical innovation 2018. Deloitte Centre for Health Solutions:Retrieved 27 June 2019, from https://www2.deloitte.com/uk/en/pages/life-sciences-and-healthcare/articles/measuring-return-from-pharmaceutical-innovation.html

Dembe AE, Lynch MS, Gugiu PC, & Jackson RD. (2014). The translational research impact scale: development, construct validity, and reliability testing. Evaluation & the Health Professions, 37: 50–70.

Deshpande, A., Guthrie, S., Cox, C., & Bertscher, A. (2018). Development of metrics to measure the impact of Tommy's work. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/research_reports/RR2587.html

Donovan, C., Butler, L., Butt, A.J., Jones, T.H., & Hanney, S.R. (2014). Evaluation of the impact of National Breast Cancer Foundation-funded research. The Medical Journal of Australia, 200:214–18. http://dx.doi.org/10.5694/mja13.10798

Dougherty, D., & Conway, P. H. (2008). The ‘3T’s’ road map to transform US health care: the ‘how’ of high-quality care. Jama, 299(19), 2319–2321.

https://doi.org/10.1016/S0140-6736(13)62229-1

https://doi.org/10.1016/S0140-6736(13)62229-1

https://doi.org/10.1525/cmr.2014.56.4.144

https://doi.org/10.1016/S0002-9343(03)00013-5

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5579142/

https://www2.deloitte.com/uk/en/pages/life-sciences-and-healthcare/articles/measuring-return-from-pharmaceutical-innovation.html

https://www2.deloitte.com/uk/en/pages/life-sciences-and-healthcare/articles/measuring-return-from-pharmaceutical-innovation.html

https://www.rand.org/pubs/research_reports/RR2587.html

http://dx.doi.org/10.5694/mja13.10798


55

Drew, C.H., Pettibone, K.G., Ruben, E. (2013). Greatest ‘HITS’: A new tool for tracking impacts at the National Institute of Environmental Health Sciences. Research Evaluation, 22:307–15. http://dx.doi.org/10.1093/reseval/rvt022

Drolet, B. C., & Lorenzi, N. M. (2011). Translational research: understanding the continuum from bench to bedside. Translational Research, 157(1), 1-5.

Durst, S., & Poutanen, P. (2013). Success factors of innovation ecosystems – Initial insights from a literature review. Co-Create, 27–38.

Durst, S., & Poutanen, P. (2013). Success factors of innovation ecosystems – Initial insights from a literature review. Co-Create, 27–38.

Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237–256.

Edwards, M. A., & Roy, S. (2017). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental Engineering Science, 34(1), 51-61.

Eichler, H. G., Baird, L. G., Barker, R., BloechlDaum, B., BørlumKristensen, F., Brown, J., Chua, R., Del Signore, S., Dugan, U., Ferguson, J., Garner, S., Goettsch, W., Haigh, J., Honig, P., Hoos, A., Huckle, P., Kondo, T., Le Cam, Y., Leufkens, H., Lim, R., Longson, C., Lumpkin, M., Maraganore, J., O'Rourke, B., Oye, K., Pezalla, E., Pignatti, F., Raine, J., Rasi, G., Salmonson, T., Samaha, D., Schneeweiss, S., Siviero, PD., Skinner, M., Teagarden, JR., Tominaga, T., Trusheim, MR., Tunis, S., Unger, TF., Vamvakas, S., & Garner, S. (2015). From adaptive licensing to adaptive pathways: delivering a flexible life‐span approach to bring new drugs to

patients. Clinical Pharmacology & Therapeutics, 97(3), 234-246.

El Turabi, A. Hallsworth, M., Ling, T., and Grant, J. (2011). A Novel Performance Monitoring Framework for Health Research Systems: Experiences of the National Institute for Health Research in England. Health Research Policy and Systems, 9:13.

Ensor, T. and Cooper, S. (2004). Overcoming barriers to health service access: influencing the demand side. Health policy and planning, 19(2), pp.69-79.

Epstein, A.J., Ketcham, J.D., Rathore, S.S. and Groeneveld, P.W. (2012). Variations in the use of an innovative technology by payer: the case of drug-eluting stents. Medical care, 50(1), p.1.

Evans, D., Coad, J., Cottrell, K., Dalrymple, J., Davies, R., Donald, C, et al. (2014). Public involvement in research: assessing impact through a realist evaluation. Health Services and Delivery Research, 2(36).

Expert Panel for Health Directorate of the European Commission’s Research Innovation Directorate General (2013). Review of Public Health Research Projects Financed under the Commission’s Framework Programmes for Health Research. Brussels: European Commission.

http://dx.doi.org/10.1093/reseval/rvt022

RAND Europe

56

Federal Drug Administration (FDA). (n.d.) Priority Review:Retrieved 27 June 2019, from https://www.fda.gov/patients/fast-track-breakthrough-therapy-accelerated-approval-priority-review/priority-review.

Feld, B. (2012). Startup Communities: Building an Entrepreneurial Ecosystem in Your City. Retrieved 20 August 2019, from https://www.wiley.com/en-gb/Startup+Communities%3A+Building+an+Entrepreneurial+Ecosystem+in+Your+City-p-9781118483312

Frenkel, A., & Maital, S. (2014). Mapping national innovation ecosystems: Foundations for policy consensus. Edward Elgar Publishing.

Frey, B. S., & Osterloh, M. (1997). Sanktionen oder Seelenmassage? Motivationale Grundlagen der Unternehmensfuehrung. Betriebswirtschaft-Stuttgart -, 57(3), 307–321.

Frontier Economics (2014). Rates of return to investment in science and innovation. A report prepared for the UK Department for Business, Innovation and Skills (BIS).

Fukuda, K., & Watanabe, C. (2008). Japanese and US perspectives on the National Innovation Ecosystem. Technology in Society, 30(1), 49–63. https://doi.org/10.1016/j.techsoc.2007.10.008

Gilmour, J.B. (2006). Implementing OMB’s Program Assessment Rating Tool (PART): Meeting the Challenges of Integrating Budget and Performance. OECD Journal on Budgeting, 7(1), 1–40.

Glasgow, R. E. (2007). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science of Dissemination and Implementation in the Service of Public Health: Bethesda, MD.

Glasgow, R. E., Green, L. W., Taylor, M. V., & Stange, K. C. (2012). An evidence integration triangle for aligning science with policy and practice. American Journal of Preventive Medicine, 42(6), 646-654.

Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327.

Glasziou P, Altman DG, Bossuyt P, et al. (2014). Reducing waste from incomplete or unusable reports of biomedical research. Lancet, 383(9913):267-76.

Glasziou P, Chalmers I. (2018). Research waste is still a scandal — an essay. BMJ, 363 doi: https://doi.org/10.1136/bmj.k4645

Godin, B. (2009). National Innovation System: The System Approach in Historical Perspective. Science, Technology, & Human Values, 34(4), 476–501. https://doi.org/10.1177/0162243908329187

Graham I. D. et al. (2006). ‘Lost in Knowledge Translation: Time for a Map?', The Journal of Continuing Education in the Health Professions, 26(1):13 – 24.

Grant, J., Green, L., Mason, B. (2003). ‘Basic research and health: a reassessment of the scientific basis for the support of biomedical science.’ Research Evaluation, 12: 217–24

https://www.fda.gov/patients/fast-track-breakthrough-therapy-accelerated-approval-priority-review/priority-review

https://www.wiley.com/en-gb/Startup+Communities%3A+Building+an+Entrepreneurial+Ecosystem+in+Your+City-p-9781118483312



https://doi.org/10.1016/j.techsoc.2007.10.008

https://doi.org/10.1136/bmj.k4645

https://doi.org/10.1177/0162243908329187


57

Green LW, Orleans CT, Ottoson JM, Cameron R, Pierce JP, Bettinghaus EP. (2006). Inferring strategies for disseminating physical activity policies, programs, and practices from the successes of tobacco control. American Journal of Preventive Medicine, 31(4S):S66 –S81.

Green, L., Ottoson, J., Garcı´a, C., and Hiatt, R. (2009). Diffusion theory and knowledge dissemination, utilization, and integration in public health. Annual Review of Public Health, 30: 151–74.

Grether, M., Eickelberg, O., Mall, M. A., Rabe, K. F., Welte, T., & Seeger, W. (2014). New metrics for translational research. The Lancet Respiratory Medicine, 2(8), e13-e14.

Grimes, D.R., Bauch, C.T. and Ioannidis, J.P.A (2018). Modelling science trustworthiness under publish or perish pressures. Royal Society Open Science.

Group of Eight (2012). Excellence in Innovation: Research Impacting our Nation’s Future: Assessing the Benefits. Adelaide, SA: Australian Technology Network of Universities.

Guthrie, S., Bienkowska-Gibbs, T., Manville, C., Pollitt, A., Kirtley, A., and Wooding, S. (2015). The impact of the National Institute for Health Research Health Technology Assessment programme, 2003–13: a multimethod evaluation. Health Technology Assessment 19(67).

Guthrie, S., d'Angelo, C., Ioppolo, B., Shenderovich, Y., and McInroy, G. R. (2018). Evidence synthesis on measuring the distribution of benefits of research and innovation. The Royal Society. https://www.rand.org/pubs/research_reports/RR2610z1.html

Guthrie, S., Krapels, J., Lichten, C. A., and Wooding, S. (2016). 100 Metrics to Assess and Communicate the Value of Biomedical Research: An Ideas Book. Association of American Medical Colleges. https://www.rand.org/pubs/research_reports/RR1606.html

Guthrie, S., Wamae, W., Diepeveen, S., Wooding, S., and Grant, J. (2013). Measuring research: A guide to research evaluation frameworks and tools. Santa Monica, CA: RAND Corporation, 2013. https://www.rand.org/pubs/monographs/MG1217.html

Hannes, L. (2014). Why the public sector should stop selecting innovation projects | Discuto. Retrieved 29 June 2019, from https://www.discuto.io/de/consultation/3193?page&equal;1

Hanney S, Buxton M, Green C, Coulson D, Raftery J. (2007). An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technology Assessment, 11(53). http://dx.doi.org/10.3310/hta11530

Hanney, S. R., Castle-Clarke, S., Grant, J., Guthrie, S., Henshall, C., Mestre-Ferrandiz, J., Pistollato, M., Pollitt, A., Sussex, J., & Wooding, S. (2015). How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice. Health research policy and systems, 13(1).

Hao, X. (Janet), van Ark, B., & Ozyildirim, A. (2017). Signposts of Innovation: A Review of Innovation Metrics (SSRN Scholarly Paper No. ID 2970648). Retrieved 20 August 2019, from Social Science Research Network website: https://papers.ssrn.com/abstract=2970648

Harris J.R., Cheadle A., Hannon P.A., et al. (2012). A framework for disseminating evidence-based health promotion practices. Preventing Chronic Disease, 9:E22.

https://www.rand.org/pubs/research_reports/RR2610z1.html


https://www.rand.org/pubs/monographs/MG1217.html

https://www.discuto.io/de/consultation/3193?page&equal

http://dx.doi.org/10.3310/hta11530

https://papers.ssrn.com/abstract=2970648

RAND Europe

58

Haskel, J., Hughes, A. and Bascavusoglu-Moreau, E., (2014). The Economic Significance of the UK Science Base. A Report for the Campaign for Science and Engineering, UK Innovation Research Centre.

Havelock RG. (1969). Planning for innovation through dissemination and utilization of knowledge. Centre for Research on Utilization of Scientifıc Knowledge, Institute for Social Research, University of Michigan.

Heneghan C, Mahtani KR, Goldacre B, Godlee F, Macdonald H, Jarvies D. (2017). Evidence based medicine manifesto for better healthcare. BMJ 357:j2973. doi:10.1136/bmj.j2973 pmid:28634227F

Herper M (2014). The cost of creating a new drug now $5 billion, pushing big pharma to change. Retrieved 20 August 2019, from https://www.forbes.com/sites/matthewherper/2013/08/11/how-the-staggering-cost-of-inventing-new-drugs-is-shaping-the-future-of-medicine/#4a2c873313c3 Higher Education Funding Council for England (HEFCE) (2015). Research Excellence Framework 2014: Overview Report by Main Panel A and Sub-panels 1 to 6. London: HEFCE.

Hill, S (2017). The fifth A of assessing impact. http://stevenhill.org.uk/the-fifth-A-of-assessing-impact/

Hirsch, G., Trusheim, M., Cobbs, E., Bala, M., Garner, S., Hartman, D., Isaacs, K., Lumpkin, M., Lim, R., Oye, K., Pezalla, E., Saltonstall, P., and Selker, H. (2016). Adaptive Biomedical Innovation: Evolving Our Global System to Sustainably and Safely Bring New Medicines to Patients in Need. Clinical Pharmacology & Therapeutics, 100(6), 685–698. https://doi.org/10.1002/cpt.509

Holmes Jr, D.R., Hance, R., Jensen, T.S.S., Schwartz, D.A., Kaplan, A., Farb, A., Zuckerman, B., Leon, M., Waklowiak, J., Mack, M.J. and Shuren, J., 2018. The 21st Century Cures Act and Early Feasibility Studies for Cardiovascular Devices: What Have We Learned, Where Do We Need to Go?. JACC: Cardiovascular Interventions, 11(21), pp.2220-2225.

Honig, P. K., & Hirsch, G. (2016). Adaptive Biomedical Innovation. Clinical Pharmacology & Therapeutics, 100(6), 574–578. https://doi.org/10.1002/cpt.513

Hughes A, Martin B. (2012). ‘Council for Industry and Higher Education, UK Innovation Research Centre. Enhancing Impact. The Value of Public Sector R&D’, Summary report (Pubd online), Retrieved 20 August 2019, from http://www.uk-irc.org/resources/reports/enhancing-impact-the-value-of-public-sector-rd-summary-report/

Institute of Medicine (IOM) (2013). The CTSA Program at NIH: opportunities for advancing clinical and translational research. Washington, DC: The National Academies Press.

Ioannidis, J. P., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014). Increasing value and reducing waste in research design, conduct, and analysis. The Lancet, 383(9912), 166-175.

Ioannidis, J.P.A. (2005). Why most published research findings are false. PLoS Medicine, 2(8): e124. https://doi.org/10.1371/journal.pmed.0020124

https://www.forbes.com/sites/matthewherper/2013/08/11/how-the-staggering-cost-of-inventing-new-drugs-is-shaping-the-future-of-medicine/#4a2c873313c3

https://www.forbes.com/sites/matthewherper/2013/08/11/how-the-staggering-cost-of-inventing-new-drugs-is-shaping-the-future-of-medicine/#4a2c873313c3

http://stevenhill.org.uk/the-fifth-A-of-assessing-impact/

https://doi.org/10.1002/cpt.509


http://www.uk-irc.org/resources/reports/enhancing-impact-the-value-of-public-sector-rd-summary-report/

http://www.uk-irc.org/resources/reports/enhancing-impact-the-value-of-public-sector-rd-summary-report/

https://doi.org/10.1371/journal.pmed.0020124


59

Ioannidis, J.P.A. (2018). Thousands of scientists publish a paper every five days. Nature Comment, 561: 167-169. doi: 10.1038/d41586-018-06185-8

Iyer, B., & Davenport, T. H. (2008, April 1). Reverse Engineering Google’s Innovation Machine. Harvard Business Review, (April 2008). Retrieved 20 August 2019, from https://hbr.org/2008/04/reverse-engineering-googles-innovation-machine

Jackson, D. J. (2011). What is an Innovation Ecosystem? Retrieved 20 August 2019, from https://www.semanticscholar.org/paper/What-is-an-Innovation-Ecosystem-By-Jackson/a4ca09537e99faad7616bd8d88be2fc2f3a4ccfc

Johnson, K., Hays, C., Center, H., & Daley, C. (2004). Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation Program Planning, 27, 135–149

Johnson, T., Joyner, M., DePourcq, F., Drezner, M., Hutchinson, R., Newton, K., Uscinski, K. C., & Toussant, K. (2010). Using research metrics to improve timelines: proceedings from the 2nd Annual CTSA Clinical Research Management Workshop. Clinical and translational science, 3(6), 305-308.

Jones, R., & Wilsdon, J. (2018, July 12). The Biomedical Bubble. Retrieved 25 June 2019, from nesta website: https://www.nesta.org.uk/report/biomedical-bubble/

Karsh, B. T. (2004). Beyond usability: Designing effective technology implementation systems to promote patient safety. Quality and Safety in Health Care, 13, 388–394.

Khoury, M. J., Gwinn, M., Yoon, P. W., Dowling, N., Moore, C. A., & Bradley, L. (2007). The continuum of translation research in genomic medicine: how can we accelerate the appropriate integration of human genome discoveries into health care and disease prevention? Genetics in Medicine, 9(10), 665.

Kingdon JW. (1984). Agendas, alternatives, and public policies. Boston: Little, Brown.

Kingdon JW. (2010). Agendas, alternatives, and public policies. Updated 2nd ed. Boston: Longman.

Kok, M. O and Schuit, A. J. (2012). ‘Contribution Mapping: A Method for Mapping the Contribution of Research to Enhance its Impact.' Health Research Policy and Systems, 10/21: 1 – 16.

Krapels, J., Morgan Jones, M., Castle-Clarke, S., Kryl, D. and Younossi, O. (2015). Developing a research impact performance management system for The Research Council, Oman: Final Report. Santa Monica, CA: RAND Corporation. RR-833/1-TRC. As of 1 August 2018: https://www.rand.org/pubs/research_reports/RR833z1.html

Kreuter MW, Casey CM, Bernhardt JM. (2012). Enhancing dissemination though marketing and distribution systems: a vision for public health. In: Brownson RC, Colditz G, Proctor EK, eds. Dissemination and implementation research in health: translating science to practice. New York: Oxford University Press.

Lander B. Atkinson-Grosjean J. (2011). ‘Translational Science and the Hidden Research System in Universities and Academic Hospitals: A Case Study.' Social Science & Medicine, 72/4: 537 – 44.

Lane J. (2010). ‘Let’s Make Science Metrics More Scientific’, Nature vol. 464 (pp. 488-9).

https://hbr.org/2008/04/reverse-engineering-googles-innovation-machine

https://www.semanticscholar.org/paper/What-is-an-Innovation-Ecosystem-By-Jackson/a4ca09537e99faad7616bd8d88be2fc2f3a4ccfc

https://www.nesta.org.uk/report/biomedical-bubble/

https://www.rand.org/pubs/research_reports/RR833z1.html

RAND Europe

60

Lavis, J., Ross, S., McLeod, C., & Gildiner, A. (2003). Measuring the impact of health research. Journal of Health Services Research & Policy, 8(3), 165-170.

Leach, R. (2014, August 26). To Get More Entrepreneurs, We Must Create Better Ecosystems. Retrieved 3 July 2019, from HuffPost website: https://www.huffingtonpost.com/ray-leach/to-get-more-entrepreneurs_b_5708839.html

Lester JP. (1993). The utilization of policy analysis by state agency offıcials. Science Communication, 14(3):267.

Littman, B. et al. (2007). ‘What's Next in Translational Medicine?', Clinical Science, 112:217 – 27.

Lowson, K., Jenks, M., Filby, A., Carr, L., Campbell, B., & Powell, J. (2015). Examining the implementation of NICE guidance: cross-sectional survey of the use of NICE interventional procedures guidance by NHS Trusts. Implementation Science, 10(1), 93.

Lund H, Brunnhuber K, Juhl C, et al. (2016). Towards evidence based research. BMJ2016;355:i5440. doi:10.1136/bmj.i5440 pmid:27797786

Luoma-aho, V., & Halonen, S. (2010). Intangibles and innovation: the role of communication in the innovation ecosystem. Innovation Journalism, 7(2), 1–20. Retrieved 20 August 2019, from http://journal.innovationjournalism.org/2010/11/intangibles-innovation-role-of.html Mahroum, S., & Al-Saleh, Y. (2013). Towards a functional framework for measuring national innovation efficacy. Technovation, 33(10), 320–332. https://doi.org/10.1016/j.technovation.2013.03.013

Marincola, F. M. (2011). ‘The Trouble with Translational Medicine', Journal of Internal Medicine, 270/2:123 – 27.

Marjanovic, S., Krapels, J., Sousa, S., Castle-Clarke, S., Horvath, V., and Chataway, J. (2015). The NIHR Invention for Innovation (i4i) programme: A review of progress and contributions to innovation in healthcare technologies. Santa Monica, CA: RAND Corporation. Retrieved 20 August 2019, from https://www.rand.org/pubs/research_reports/RR1101.html

Matei, M. M., & Aldea, A. (2012). Ranking National Innovation Systems According to their technical Efficiency. Procedia – Social and Behavioral Sciences, 62, 968–974. https://doi.org/10.1016/j.sbspro.2012.09.165

May C, Finch T. (2009). Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology 43(3):535–54.

May C, Murray E, Finch T, et al. (n.d.) Normalization process theory on-line users’ manual and toolkit. www.normalizationprocess.org.

Meagher, L., Lyall, C., and Nutley, S. (2008). Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Research Evaluation, 17(3):163–73. http://dx.doi.org/10.3152/095820208X331720

Medical Research Council (MRC) (2013). Outputs, Outcomes and Impact of MRC Research: 2012 Report. Swindon: MRC.

https://www.huffingtonpost.com/ray-leach/to-get-more-entrepreneurs_b_5708839.html

https://www.huffingtonpost.com/ray-leach/to-get-more-entrepreneurs_b_5708839.html

http://journal.innovationjournalism.org/2010/11/intangibles-innovation-role-of.html

https://doi.org/10.1016/j.technovation.2013.03.013


https://doi.org/10.1016/j.sbspro.2012.09.165

http://www.normalizationprocess.org

http://dx.doi.org/10.3152/095820208X331720


61

Medical Research Council (MRC) (2018). About researchfish. Retrieved 1 August 2018, from https://mrc.ukri.org/funding/guidance-for-mrc-award-holders/researchfish/about-researchfish/

Meijer, I. (2012). Societal Returns of Scientific Research. How Can We Measure it? Leiden: Centre for Science and Technology Studies, Leiden University.

Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. (2008). Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2):21–37.

Mercan, B., & Goktas, D. (2011). Components of innovation ecosystems: a cross-country study. International Research Journal of Finance and Economics, 76(16), 102–112. Retrieved 20 August 2019, from http://blog.ub.ac.id/arifhidayat/files/2012/06/Components-of-Innovation-Ecosystems.pdf

Mezzourh, S., & Nakara, W. (2012). New Business Ecosystems and Innovation Strategic Choices in SMEs. The Business Review, Cambridge, 20, 176–182.

Molas-Gallart, J., D’Este, P., Llopis, O., & Rafols, I. (2015). Towards an alternative framework for the evaluation of translational research initiatives. Research Evaluation, 25(3): 235-243.

Morgan Jones, M and Grant J (2013). ‘Making the Grade. Methodologies for Assessing and Evidencing Research Impact’, in 7 Essays on Impact: DESCRIBE Project Report. Dean et al., eds., Exeter, UK: Exeter University Press, 25–43.

Morris, L. (2008). Innovation Metrics: The Innovation Process and How to Measure It. Retrieved from http://www.innovationlabs.com/Measuring_Innovation.pdf

Morris, Z. S., Wooding, S., & Grant, J. (2011). The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the Royal Society of Medicine, 104(12), 510-520.

Mostert, S.P., Ellenbroek, S., Meijer, I., van Ark, G., and Klasen, E.C. (2010). Societal output and use of research performed by health research groups. Health Research Policy and Systems, 8:30. http://dx.doi.org/10.1186/1478-4505-8-30

Mugabushaka A, Papazoglou T. Jeffery K, Dvořák J. (2012). ‘Information systems of research funding agencies in the ‘era of the Big Data’. The case study of the Research Information System of the European Research Council’, E-Infrastructures for Research and Innovation: Linking Information Systems to Improve Scientific Knowledge, Proceedings of the 11th International Conference on Current Research Information Systems pp. 103–12. Prague, Czech Republic.

Munafo, M.R. et al. (2017). A manifesto for reproducible science. Nature Human Behaviour (1): 0021. DOI: 10.1038/s41562-016-0021

Murray E, Treweek S, Pope C, et al. (2010). Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Medicine, 8:63.

Nathan, A. S., Geng, Z., Dayoub, E. J., Khatana, S. A. M., Eberly, L. A., Kobayashi, T., Puglieses, S, Adusumalli S, Giri, J, and Groeneveld, P. W. (2019). Racial, Ethnic, and Socioeconomic

https://mrc.ukri.org/funding/guidance-for-mrc-award-holders/researchfish/about-researchfish/

http://blog.ub.ac.id/arifhidayat/files/2012/06/Components-of-Innovation-Ecosystems.pdf

http://blog.ub.ac.id/arifhidayat/files/2012/06/Components-of-Innovation-Ecosystems.pdf

http://www.innovationlabs.com/Measuring_Innovation.pdf

http://dx.doi.org/10.1186/1478-4505-8-30

RAND Europe

62

Inequities in the Prescription of Direct Oral Anticoagulants in Patients with Venous Thromboembolism in the United States. Circulation: Cardiovascular Quality and Outcomes, 12(4), e005600.

National Research Council (NRC) (1999). Evaluating federal research programs: Research and the Government Performance and Results Act.

National Science Foundation (NSF) (2010). STAR METRICS: New Way to Measure the Impact of Federally Funded Research. National Science Foundation, press release, 1 June.

Nieva VF, Murphy R, Ridley N, et al. (2005). From science to service: a framework for the transfer of patient safety. In: Henriksen K, Battles JB, Marks ES, Lewin DI, eds. (2005). Advances in patient safety: from research to implementation (Vol. 2: Concepts and methodology). Rockville MD: Agency for Healthcare Research and Quality.

Niosi, J. (2010). Building National and Regional Innovation Systems. Retrieved from https://ideas.repec.org/b/elg/eebook/14006.html

Office for Life Sciences (2018). Life Sciences Competitiveness Indicators:Retrieved 227 June 2019, from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/706963/life-sciences-competitiveness-indicators-2018.pdf.

Oh, D.-S., Phillips, F., Park, S., & Lee, E. (2016). Innovation ecosystems: A critical examination. Technovation, 54, 1–6. https://doi.org/10.1016/j.technovation.2016.02.004

Organisation for Economic Co-operation and Development. (1997). National Innovation Systems. Retrieved 20 August 2019, from https://www.oecd.org/science/inno/2101733.pdf

Oriana, C., Armeni, P., Boscolo, P.R., Cavazza, M., Jommi, C., Tarricone, R., (2016). De innovatione: The concept of innovation for medical technologies and its implications for healthcare policy-making. Health Policy and Technology, 5 (1). https://doi.org/10.1016/j.hlpt.2015.10.005

Owen N, Glanz K, Sallis JF, Kelder SH. (2006). Evidence-based approaches to dissemination and diffusion of physical activity interventions. American Journal of Preventive Medicine, 31(4S):S35–S44.

PBRF Working Group (2002). Wellington: Performance-based Research Fund, Ministry of Education and Transition, Tertiary Education Commission.

Penfield, T., Baker, M. J., Scoble, R., & Wykes, M. C. (2014). Assessment, evaluations, and definitions of research impact: A review. Research Evaluation, 23(1), 21-32.

Pollitt, A., Potoglou, D., Patil, S., Burge, P., Guthrie, S., King, S., Wooding, S., & Grant, J. (2016). Understanding the relative valuation of research impact: a best–worst scaling experiment of the general public and biomedical and health researchers. BMJ Open, 6(8), e010916.

Pozen R, Kline H. (2011). Defining success for translational research organizations. Science Translational Medicine, 3: 94cm20.

Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. (2009). Implementation research in mental health services: an emerging science with conceptual, methodological, and

https://ideas.repec.org/b/elg/eebook/14006.html

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/706963/life-sciences-competitiveness-indicators-2018.pdf

https://doi.org/10.1016/j.technovation.2016.02.004

https://www.oecd.org/science/inno/2101733.pdf

https://doi.org/10.1016/j.hlpt.2015.10.005


63

training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36(1):24 –34.

Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65-76.

Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver, N. L. (2008). A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice, 14(2), 117–123.

Raftery, J., Hanney, S., Greenhalgh, T., Glover, M., & Blatch-Jones, A. J. (2016). Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme. Health Technology Assessment, 20(76): 1-254.

Richter A. W., et al. (2006). ‘Boundary Spanners' Identification, Intergroup Contact, and Effective Intergroup Relations.' Academy of Management Journal, 49/6: 1252 – 69.

Rohrbeck, R., Hölzle, K., & Gemünden, H. G. (2009). Opening up for competitive advantage – How Deutsche Telekom creates an open innovation ecosystem. R&D Management, 39(4), 420–430. https://doi.org/10.1111/j.1467-9310.2009.00568.x

Ross JS, Tse T, Zarin DA, Xu H, Zhou L, Krumholz HM (2012). Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ. 344:d7292.

Rothwell, R. (1994). Towards the Fifth generation Innovation Process. International Marketing Review, 11(1), 7–31. https://doi.org/10.1108/02651339410057491

Royal Netherlands Academy of Arts and Sciences (2010). Standard Evaluation Protocol 2009–2015. Protocol for Research Assessment in the Netherlands. Amsterdam: Association of Universities in the Netherlands.

Rycroft-Malone J, Wilkinson J, Burton CR, Harvey G, McCormack B, Graham I, et al. (2013). Collaborative action around implementation in Collaborations for Leadership in Applied Health Research and Care: towards a programme theory. Journal of Health Services Research & Policy, 18(Suppl. 3), 13–26. http://dx.doi.org/10.1177/1355819613498859

Rye, C. B., & Kimberly, J. R. (2007). The adoption of innovations by provider organizations in health care. Medical Care Research and Review, 64(3), 235–278.

Said, M., and Zerhouni, E., (2014), The role of public–private partnerships in addressing the biomedical innovation challenge. Nature Reviews Drug Discovery, 13.

Sainty M. (2013). Research impact: a United Kingdom Occupational Therapy Research Foundation perspective. British Journal of Occupational Therapy, 76:528–37.

Samila, S., & Sorenson, O. (2010). Venture capital as a catalyst to commercialization. Research Policy, 39(10), 1348–1360. https://doi.org/10.1016/j.respol.2010.08.006

Sampat, B.N., and Lichtenberg, F.R., (2011). What are the respective roles of the public and private sectors in pharmaceutical innovation? Health Affairs, 30 (2).

https://doi.org/10.1111/j.1467-9310.2009.00568.x

https://doi.org/10.1108/02651339410057491

http://dx.doi.org/10.1177/1355819613498859

https://doi.org/10.1016/j.respol.2010.08.006

RAND Europe

64

Sarli, C. C., Dubinsky, E. K., & Holmes, K. L. (2010). Beyond citation analysis: a model for assessment of research impact. Journal of the Medical Library Association, 98(1), 17–23. doi:10.3163/1536-5050.98.1.008

Schneeweiss, S., Eichler, H.-G., GarciaAltes, A., Chinn, C., Eggimann, A.-V., Garner, S., Goettsch W., Lim R., Löbker W., Martin D., Müller T., Park BJ., Platt R., Priddy S., Ruhl M., Spooner A., Vannieuwenhuyse B. & Willke, R. J. (2016). Real World Data in Adaptive Biomedical Innovation: A Framework for Generating Evidence Fit for Decision-Making. Clinical Pharmacology & Therapeutics, 100(6), 633–646. https://doi.org/10.1002/cpt.512

Schuhmacher, A., (2016), Changing R&D models in research-based pharmaceutical companies. Journal of Translational Medicine, 14 (105). https://doi.org/10.1186/s12967-016-0838-4

Schulthess, D., Chlebus, M., Bergstrm, R., & Van Baelen, K. (2014). Medicine adaptive pathways to patients (MAPPs): using regulatory innovation to defeat Eroom’s law. Chinese clinical oncology, 3(2).

Seo, J. (2014). Building the platform to bring new ideas to startups in the creative economy. Proceedings of the 2014 World Technopolis Association International Training Workshop, 249–259. Daejeon, Republic of Korea.

Spaapen, J., van Drooge, L., Propp, T., van der Meulen, B., Shinn, T., Marcovich, A., et al. (2011). Social Impact Assessment Methods for Research and Funding Instruments Through the Study of Productive Interactions (SIAMPI): Final Report on Social Impacts of Research. Amsterdam: Royal Netherlands Academy of Arts and Sciences.

St-Germain, V. (2010, December 10). The Results-Oriented Organisation: A Critical Look from a Bertalanffyan Perspective. Presented at the BCSSS Lectures: Uncommon Sense in Thinking Society, Vienna, Austria. Retrieved 20 August 2019, from http://www.bcsss.org/wp-content/uploads/2011/09/pdf61.pdf

Stiles, P. G., Boothroyd, R. A., Snyder, K., & Zong, X. (2002). Service penetration by persons with severe mental illness: How should it be measured? Journal of Behavioral Health Services & Research, 29(2), 198.

Sung, N. S., Crowley Jr, W. F., Genel, M., Salber, P., Sandy, L., Sherwood, L. M., Johnson S.B., Catanese V., Tilson H., Getz K., Larson E.L., Scheinberg D., Reece E.A., Slavkin H., Dobs A., Grebb J., Martinez R.A., Korn A., & Rimoin D. (2003). Central challenges facing the national clinical research enterprise. Jama, 289(10), 1278–1287.

Sussex, J., Feng, Y., Mestre-Ferrandiz, J., Pistollato, M., Hafner, M., Burridge, P., and Grant, J. (2016). Quantifying the economic impact of government and charity funding of medical research on private research and development funding in the United Kingdom. BMC Medicine, 14(32).

Swann, J. et al. (2007). ‘Modes of Organizing Biomedical Innovation in the UK and US and the Role of Integrative and Relational Capabilities.' Research Policy, 36/4: 529 – 47.


https://doi.org/10.1186/s12967-016-0838-4

http://www.bcsss.org/wp-content/uploads/2011/09/pdf61.pdf

http://www.bcsss.org/wp-content/uploads/2011/09/pdf61.pdf


65

Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice: models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337-350.

Talmar, M., Walrave, B., Podoynitsyna, K. S., Holmström, J., & Romme, A. G. L. (2018). Mapping, analyzing and designing innovation ecosystems: The Ecosystem Pie Model. Long Range Planning. https://doi.org/10.1016/j.lrp.2018.09.002

Tassey, G. (2010). Rationales and mechanisms for revitalizing US manufacturing R&D strategies. The Journal of Technology Transfer, 35(3), 283–333. https://doi.org/10.1007/s10961-009-9150-2

The National Academies of Science Engineering Medicine. 2017. ‘Mapping and Connecting the Biomedical Innovation Ecosystem: Health and Medicine Division.’ Retrieved 20 August 2019, from http://nationalacademies.org/hmd/Activities/Research/DrugForum/Mapping-and-Connecting.aspx.

TIDIRH Working Group. (2011). Interacting elements of integrating science, policy, and practice. In: Training institute for dissemination and implementation research in health. Conference proceedings. Chapel Hill NC.

Trochim, W. (2010). Translation Won’t Happen Without Dissemination and Implementation: Some Measurement and Evaluation Issues. 3rd Annual Conference on the Science of Dissemination and Implementation. Bethesda, MD.

Trochim, W., Kane, C., Graham, M., Pincus, H. A. (2011). Evaluating Translational Research: A Process Marker Model. Clinical and Translational Science, 4(3):153-162.

Turner, K. M. T., & Sanders, M. R. (2006). Dissemination of evidence-based parenting and family support strategies: Learning from the triple p-positive parenting program system approach. Aggression and Violent Behavior, 11(2), 176–193.

University Health Network (2008). Accountability to Our Patients How we measure up – the Balanced Scorecard. Retrieved 20 August 2019, from https://www.uhn.ca/corporate/AboutUHN/Purpose_Performance/Pages/balanced_scorecard.aspx

University of Sussex (2019) The Framework, University of Sussex. Retrieved 20 August 2019, from http://www.sussex.ac.uk/spru/research/dare/Framework.html

Wagner, J. A., Dahlem, A. M., Hudson, L. D., Terry, S. F., Altman, R. B., Gilliland, C. T., DeFeo, C. , & Austin, C. P. (2018a). Application of a Dynamic Map for Learning, Communicating, Navigating, and Improving Therapeutic Development. Clinical and Translational Science, 11(2), 166–174. https://doi.org/10.1111/cts.12531

Wagner, J., Dahlem, A. M., Hudson, L. D., Terry, S. F., Altman, R. B., Gilliland, C. T., DeFeo, C., & Austin, C. P. (2018b). A dynamic map for learning, communicating, navigating and improving therapeutic development. Nature Reviews Drug Discovery, 17(2), 150. https://doi.org/10.1038/nrd.2017.217

https://doi.org/10.1016/j.lrp.2018.09.002

https://doi.org/10.1007/s10961-009-9150-2

http://nationalacademies.org/hmd/Activities/Research/DrugForum/Mapping-and-Connecting.aspx

http://nationalacademies.org/hmd/Activities/Research/DrugForum/Mapping-and-Connecting.aspx

https://www.uhn.ca/corporate/AboutUHN/Purpose_Performance/Pages/balanced_scorecard.aspx

http://www.sussex.ac.uk/spru/research/dare/Framework.html

https://doi.org/10.1111/cts.12531

https://doi.org/10.1038/nrd.2017.217

RAND Europe

66

Wallner, T., & Menrad, M. (2011). Extending the innovation ecosystem framework. Proceedings of XXII ISPIM Conference. Retrieved 20 August 2019, from https://pdfs.semanticscholar.org/2b2c/9911cea2259692286c178a9fbc5d8d550157.pdf

Wandersman A, Duffy J, Flaspohler P, et al. (2008). Bridging the gap between prevention research and practice: the interactive systems framework for dissemination and implementation. American Journal of Community Psychology, 41(3– 4):171– 81.

Ward V, House A, Hamer S. (2009). Developing a framework for transferring knowledge into action: a thematic analysis of the literature. Journal of Health Services Research & Policy, 14(3):156–64.

Ward V, Smith S, Carruthers S, House A, Hamer S. (2010). Knowledge brokering. Exploring the process of transferring knowledge into action. Leeds: University of Leeds.

Ward V, Smith S, House A, Hamer S. (2012). Exploring knowledge exchange: a useful framework for practice and policy. Social Science & Medicine, 74(3):297–304.

Weiss, A.P. (2007). Measuring the impact of medical research: moving from outputs to outcomes. American Journal of Psychiatry, 164(2), 206–214. doi: 10.1176/appi.ajp.164.2.206

Weiss, C. (1995). Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families. In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.

Wellcome Trust (2014). Assessment Framework Report 2012/13. London: Wellcome Trust.

Wellcome Trust. (2014). The UK’s innovation ecosystem. Retrieved 20 August 2019, from https://wellcome.ac.uk/sites/default/files/wtp057817_1.pdf

Wessner, C. W. (2007). Innovation Policies for the 21st Century: Report of a Symposium. https://doi.org/10.17226/11852

Westfall, J., Mold, J., Fagnan, L. (2007). Practice-based research – ‘Blue Highways’ on the NIH roadmap. JAMA 297: 403–6.

Williams, V., Eiseman, E., Landree, E., Adamson, D. (2009). Demonstrating and Communicating Research Impact. Preparing NIOSH Programs for External Review. Ann Arbor, MI: University of Michigan, RAND.

Willis, C. D., Riley, B., Stockton, L., Viehbeck, S., Wutzke, S., & Frank, J. (2017). Evaluating the impact of applied prevention research centres: results from a modified Delphi approach. Research Evaluation, 26(2), 78-90.

Wilsdon, J. (2016). The metric tide: Independent review of the role of metrics in research assessment and management. Sage.

Wooding, S., Nason, E., Starkey, T., Hanney, S., and Grant, J. (2009). Mapping the Impact: Exploring the Payback of Arthritis Research. Santa Monica, CA: RAND Corporation.

Wooding, S., Pollitt, A., Castle-Clarke, S., Cochrane, G., Diepeveen, S., Guthrie, S., Horvitz-Lennon M., Larivière V., Morgan Jones M., Chonaill S.N., O'Brien C., Olmsted S.S., Schultz D.,

https://pdfs.semanticscholar.org/2b2c/9911cea2259692286c178a9fbc5d8d550157.pdf

https://wellcome.ac.uk/sites/default/files/wtp057817_1.pdf

https://doi.org/10.17226/11852


67

Winpenny E., Pincus H.A., & Grant J. (2014). Mental Health Retrosight: Understanding the Returns from Research (Lessons from Schizophrenia): Policy Report. Rand Health Quarterly, 4(1).

Wratschko, K. (2009). Empirical Setting: The pharmaceutical industry. Strategic Orientation and Alliance Portfolio Configuration. New York, NY: Springer.

Yordanov, et al. (2015). Avoidable waste of research related to inadequate methods in clinical trials. BMJ, 350:h809

69

Annex A. Methods

The team has undertaken an accelerated evidence review in order to capture and summarise the existing literature regarding metrics and frameworks that measure the biomedical innovation ecosystem. This has been informed primarily through a rapid literature review, which capitalises on existing, comprehensive reviews, publications, and grey literature known to the research team through previous work. This has been complemented by targeted additional searches on known literature in the area, as well as ‘snowballing’, or identification of references within relevant publications.

The analysis and literature have also been informed by a series of semi-structured interviews with five experts who have provided additional in-depth information and who have helped identify further significant work and publications to form part of the project’s literature review. Experts have been identified based on the existing networks of the research team and in close collaboration with FasterCures.

For the purposes of this study, and in order to facilitate the project’s research and analysis, the research team separated the ‘biomedical innovation ecosystem’ into the following five sets of issues to help guide searches, seek and consult experts, and identify where there may be gaps in the literature.

Figure A.1 Issues identified for research and analysis within the biomedical innovation ecosystem for the purposes of this project

Source: RAND Europe

RAND Europe

70

Scope

The project team has taken the scope of the review to include the identification and presentation of relevant frameworks and metrics for the analysis and understanding of the biomedical innovation system. This includes a non-exhaustive mapping and overview of the systems and metrics already in use and benefits and challenges of each. The recommendation of one specific framework or metric over another, as well as the formulation of new potential frameworks, is considered outside the scope of this review.

Literature review

We were aware of four good quality reviews covering frameworks and metrics in the evaluation of academic biomedical and health research (Raftery et al. 2016, Banzi et al. 2011, Guthrie et al. 2018, Penfield et al. 2014). We therefore focused our analysis initially on these four sources, snowballing from them to analyse specific reference papers where salient details were available. In addition, to ensure adequate coverage of the literature on research integrity and research waste, we reviewed the series of papers in the Lancet on research waste29 and the recent ten-year anniversary commentary on progress by Glasziou and Chalmers (2018). We again snowballed from these core publications to access other relevant literature as needed.

While existing reviews focus primarily on the measurement of public and charitably-funded research and development (R&D), gaps in the known literature include: a focus on wider elements of the biomedical innovation ecosystem, regulatory systems, translational research, interactions between public and private research, and patient access to treatment.

These issue areas have been subject to additional, targeted searches in specific databases (e.g. Google Scholar, PubMed), complemented by a review of relevant grey literature. A list of relevant search terms for each area is listed below.

Table A.1 Search strategy for issue area

Issue area Relevant search terms

Translational research ((knowledge OR research) AND translation AND health) OR ((translational research) AND (indicators OR metrics OR measurement OR assessment))

Regulatory environment Biomedical AND (innovat* OR treatment OR device) AND (regulat* OR legal OR law OR barrier OR challenge)

conceptual[All Fields] AND framework[All Fields] AND ‘biomedical’[All Fields]) AND (‘regulation’[All Fields]))

Private investment and Public-private collaboration

Biomed* AND innov* AND (private* OR indust* OR pharma)

29 https://www.thelancet.com/series/research

https://www.thelancet.com/series/research


71

Issue area Relevant search terms

Biomed* AND innovati* AND public AND private

Patient access to treatment30 (biomedical OR medical) AND ((new OR innov* OR pioneer*) AND (treatment OR device OR drug OR intervention OR procedure) AND (patient OR market OR cost OR price OR equity OR afford*)) AND (framework OR metric OR (access AND (evaluat* OR assess*)))

Ecosystem Biomed* AND innov* AND (ecosystem OR system) OR incentive

Biomed* AND innov* AND (ecosystem OR system) AND (framework OR metric* OR indicat*)

Source: RAND Europe

Screening, extraction and analysis

• Screening: Typically we have screened the first 100 papers by relevance for each search and identified those which are most relevant to the aim of this review – particularly those which articulate specific indicators or metrics that have been used or proposed, or which discuss frameworks and conceptualisations regarding the biomedical innovation ecosystem or components of it. Papers were also prioritised based on year of publication (with more recent publications – e.g. in the last ten years – prioritised) and geographical location (work from the US and comparable R&I systems prioritised). We only received publications in English. Following initial screening based on title and abstract, the full texts of relevant papers were then reviewed.

• Extraction: Researchers recorded data about each reviewed paper (including those initially known by the research team, and additional papers identified), including general information on the publication, information on the key study questions it addresses, and evidence it presents in relation to the issue under analysis (e.g. regulatory environment, ecosystem, etc.).

• Analysis: Relevant evidence has been mapped against initial study questions to identify the main findings of relevance. Evidence has been synthesised in this paper using a narrative synthesis approach.

Interviews

To supplement our review of the literature, we also conducted interviews with five experts. The aim of these interviews was to help target our searches better, identify relevant literature, and capture grey literature or information available in other sources that might not have been easily identified through our searches (e.g. on organisational web pages). Table A.2 provides an overview of the interviews conducted. All interviews were conducted by telephone with the exception of one interview which was conducted face to face. Interviews lasted between 30 and 60 minutes and were unstructured to enable issues of relevance

30 Supplemented by targeted searches by author name for known experts – notably Peter Groeneveld and Janet Woodcock. Also supplemented by publications identified by experts at interview.

RAND Europe

72

to the individual in question to be explored as required, since each individual had different expertise and knowledge. Consent was sought in line with GDPR requirements, including to provide information on their participation as set out in Table A.2.

Table A.2 Summary of interviews conducted

Interviewee Affiliation Areas of expertise

Prof Stephen Hanney Brunel University Expert in health system evaluation, particular expertise in evaluation of academic research and translational research.

Anonymous Anonymous Expert in regulatory systems and their evaluation

Dr Peter W Groeneveld, MD University of Pennsylvania Expert in diffusion of medical innovations, and measurement of quality and outcomes in healthcare settings

Anonymous Anonymous Expert in evaluation of new treatments and in research translation

Phill O’Neill Office for Health Economics Expert in the economics of the pharmaceutical market and benchmarking industry performance

73

Annex B. Overview of a range of indicators used in the evaluation and assessment of academic biomedical and health research

Below we provide a summary of the most relevant indicators used in evaluation and assessment of academic biomedical and health research

Table B.1 Summary of most relevant indicators

Area Indicators

Knowledge production and research targeting

Number and quality of outputs including but not limited to publications; New research tools and resources generated; Relevance of research conducted; Targeting of future research; Funding received

Capacity building Esteem measures; Number and quality of researchers trained; Collaboration and networking (academic and wider); Career outcomes for researchers and students trained

Innovative and economic impact New products and processes developed (and patented, licensed, used); New businesses (spinouts); Benefits to companies in terms of profitability, new clients, competitive advantage, efficiency etc.; Job creation, workforce development and increased economic competitiveness of region/country

Health and health sector benefit Impact on professional training or development in health; Impact on practice including efficiency/cost savings; Regulatory approval and implementation of new interventions; Practice reflects evidence-based guidelines; Making processes more efficient or resilient etc.; Impact on health and wellbeing (of a group of patients, or more widely)

Policy and public services (not limited to health)

Interaction with policy and decisionmakers through invitations, meetings, committees; Citation in or other influence on policy documents, including guidelines; Resulting changes in policy; Improvement in public services (including outside of health – e.g. STEM education); Resulting benefits to society and quality of life

Public engagement Number and range of dissemination and outreach activities; Increased public understanding of and engagement with science and research (e.g. participation in clinical trials); More positive attitudes towards research and researchers

RAND Europe

74

Sources: RAND Europe analysis based on Banzi et al. (2011); CAHS(2009); Pollitt et al. (2016); Bernard Becker Medical Library (n.d.); Lavis et al. (2003); Deshpande et al. (2018); Guthrie et al. (2016); CHSPRA (2018); and Willis et al. (2017)

Each of the sources/frameworks which provides the input to the above table is included below for further reference.

Table B.2 Payback framework

Area Indicators


Journal articles; Conference presentations; Books; Research reports; Other disseminative material; New research lines or know-how

Capacity building Career promotions; PhDs, Masters

Innovative and economic impact Benefits in occupation and economic development, productivity

Health and health sector benefit Health outcomes; QALY; Savings for health care systems


Guidelines and documents addressing policies; Citations; Membership of decision panels

Public engagement N/A

Source: Banzi et al. (2011)

Table B.3 Research impact model

Area Indicators


Knowledge and methodology advancements

Capacity building Networking, leadership, establishment of collaborations and networks

Innovative and economic impact Macroeconomic benefits

Health and health sector benefit Evidence-based practice; Cost-containment, quality of care, knowledge, attitudes and behaviour, health literacy, social capital, equity


N/A


Source: Banzi et al. (2011)

Table B.4 Framework and indicators to measure returns on investment in health research

Area Indicators


Activity indicators, e.g. share of publications, publication counts; Outreach, e.g. co-author analysis, field analysis of citations; Contextual and cultural, e.g. relative activity index Aspiration, e.g. Expanded relative citation impact, relative download


75

Area Indicators

rate, research diffusion; Funding e.g. levels of additional research funding; Infrastructure e.g. infrastructure grants, percentage of activity grants with infrastructure support

Capacity building Personnel, e.g. graduated research students in health-related subjects; numbers of research and research-related staff in Canada; Aspiration indicators, e.g. receptor capacity, absorptive capacity

Innovative and economic impact Health products industry, e.g. number of patents licensed, clustering and co-location, consulting to industry; collaboration with industry, use of research in stage reports by industry; Activity impact, e.g. economic rent (labour rent) – benefit accrued through the action of research as opposed to outputs of research – the economic benefit of employing people in health research rather than in another capacity; Commercialisation, e.g. licensing returns, product sales revenues, valuation of spinout companies, producer rent and spillover effect – economic benefit accrued through sales and revenues of commercialised research findings

Health and health sector benefit Health impacts: Health status, e.g. prevalence, incidence, potential years life lost, quality-adjusted life years (QALYs), patient reported outcome measures; Determinants of health, e.g. examples include obesity, alcohol consumption, waiting times, adherence to clinical guidelines etc.; Health benefit, e.g. health benefit in QALYs per health care dollar, health benefit in patient-reported outcome measures per health care dollar – the net benefit of improving health


Informing decisionmaking: Health related (health care, public health, social care and other health-related systems) e.g. use of research in guidelines, survey of public health policymakers, reported use of research findings outside health, research cited in ongoing health professional education material; Research, e.g. consulting to policy making, requests for research to support policy

Public engagement General public e.g. research cited in advocacy publications, public lectures given; Aspiration indicators, e.g. media citation analysis, citation in public policy documents

Source: CAHS (2009)

Table B.5 Excellence in Research for Australia (ERA)

Area Indicators


Indicators of research volume and activity: covers the total number of outputs (publications, books and so on); Total income and a number of other research items indicative of the volume of research conducted at the institution; Indicators of research quality: a mix of citation data, peer review and peer-reviewed Australian and international research income

RAND Europe

76

Area Indicators

Capacity building Indicators of recognition: a range of esteem measures, such as being the editor of a prestigious work of reference, fellow of a learned academy, or recipient of a nationally competitive research fellowship.

Innovative and economic impact Indicators of research application: concerns the wider impact of research and consists of research commercialisation income and other measures

Health and health sector benefit N/A


N/A


Source: Pollitt et al. (2016)

Table B.6 Research Excellence Framework (REF)

Area Indicators


N/A

Capacity building N/A

Innovative and economic impact Commercial impacts: Impacts where the beneficiaries are usually companies, either new or established, or other types of organisation which undertake activity that creates wealth, e.g. a spinout or new business has been created, and established its viability by generating revenue or profits Industry (including overseas industry) has invested in research and development. The performance of an existing business has been improved A business or sector has adopted a new technology or process. The strategy, operations or management practices of a business have changed. A new product or service is in production or has been commercialised. Highly skilled people have taken up specialist roles (including academic consultancy) in companies or other organisations. Jobs have been created or protected. Social enterprise initiatives have been created. Production impacts: Impacts where the beneficiaries are individuals (including groups of individuals) whose production has been enhanced, e.g. research has increased production, yields or quality or reduced waste. Decisions by regulatory authorities have been influenced by research. Costs of production have been reduced. Husbandry methods have changed. Change in management practices in production businesses.


77

Area Indicators

Health and health sector benefit Economic impacts: Impacts where the beneficiaries are usually the NHS or private health care e.g. Policies that have an impact on economic growth or incentivising productive activity. The costs of treatment or healthcare have reduced. Changing the roles and/or incentives for health professionals and organisations, resulting in improved service delivery. Health and welfare impacts: Impacts where the beneficiaries are individuals and groups (both human and animal) whose quality of life has been enhanced (or potential harm mitigated), e.g. Public health and well-being has improved. A new clinical or lifestyle intervention (for example, drug, diet, treatment or therapy) has been developed, trialled with patients or other groups (for example, prisoners, community samples), and definitive (positive or negative) outcome demonstrated. A new diagnostic or clinical technology has been adopted. Disease prevention or markers of health have been enhanced by research. Impacts on practitioners and services: Impacts where beneficiaries are organisations or individuals, including service users involved in the development and delivery of professional services e.g. Changes to professional standards, guidelines or training have been influenced by research. Practitioners/professionals have used research findings in conducting their work. The quality or efficiency of a professional service has improved. Work force planning has been influenced by research. Forensic methods have been influenced by research.


Impacts on public policy and services: Impacts where the beneficiaries are usually government, public sector, and charity organisations and societies, either as a whole or groups of individuals in society, through the implementation of policies, e.g. Policy debate has been stimulated or informed by research evidence. Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence. The implementation of a policy or the delivery of a public service (but not health) has changed. A new technology or process has been adopted. The quality, accessibility or cost-effectiveness of a public service has been improved. The public have benefitted from public service improvements. Control measures for infections have improved. Changes to social policy have been informed by research. Changes to social policy have led to improved social welfare, equality or social inclusion Impacts on the environment: Impacts where the key beneficiary is the natural or built environment, e.g. policy debate on climate change or

RAND Europe

78

Area Indicators

the environment has been influenced by research. Environmental policy decisions have been influenced by research evidence. Planning decisions have been informed by research. The management of conservation of natural resources has changed. The management of an environmental risk or hazard has changed. Impacts on international development: Impacts where the beneficiaries are international bodies, countries, governments or communities, e.g. International policy development has been influenced by research. International agencies or institutions have been influenced by research. Quality of life in a developing country has improved.

Public engagement Impacts on society, culture and creativity. Impacts where the beneficiaries are individuals, groups of individuals, organisations or communities whose knowledge, behaviours or practices have been influenced, e.g. public understanding has improved through their collaborative involvement with research. Public debate has been stimulated or informed by research.


Table B.7 National Science Foundation (NSF)

Area Indicators


N/A

Capacity building Full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM) Increased partnerships between academia, industry, and others

Innovative and economic impact Development of a diverse, globally competitive STEM workforce Increased economic competitiveness of the United States

Health and health sector benefit Improved well-being of individuals in society


Improved well-being of individuals in society Improved national security. Improved STEM education and educator development at any level Enhanced infrastructure for research and education

Public engagement Increased public scientific literacy and public engagement with science and technology



79

Table B.8 Snowball metrics

Area Indicators


Publications and citations

Capacity building Collaboration (co-authorship) Esteem measures Post-graduate Completion rates Alumni/destination of leavers Skills development

Innovative and economic impact Patenting Licensing income Spinout generation/income Knowledge Transfer Partnership (KTP) numbers Consultancy income

Health and health sector benefit N/A


N/A



Table B.9 Research Outcomes Systems (ROS)

Area Indicators


A range of types of academic output are listed, but only number and descriptions for each have been captured. Further funding received: further funding received as a direct or indirect result of the research. It can include any funding awarded by the Research Councils, the Technology Strategy Board (TSB) or other funding providers and funding-awarded collaborating partners. It can also include scholarships, studentships, fellowships or travel awards. Summary of impact - narrative summary of demonstrable contributions your research has made to academic advances, across and within disciplines, including significant advances in understanding; methods, theory and application.

Capacity building Collaboration partnership: bi-lateral or multi-lateral partnerships, and participation in networks, consortia or other initiatives. Collaborations with other departments/researchers within your institution can also be recorded here. Also includes: Membership or creation of a board; council or panel, committees, and networks set up (e.g. online discussion groups) Staff development: the role of people supported by this grant who have since moved to another position. It includes secondments into and out of the project and also covers any training courses produced. Award/Recognition: Any honours or degrees received.

RAND Europe

80

Area Indicators

Innovative and economic impact IP and exploitation: Intellectual property, exploitation mechanisms. Describe any demonstrable contributions your research has made to the economy, for example fostering global economic performance

Health and health sector benefit Describe any demonstrable contributions your research has made to society, for example fostering global economic performance, increasing effectiveness of public services and policy, and enhancing quality of life.


Policy influence: Please describe how your research has influenced the development of policy in any area of public administration and/or the delivery of public services Describe any demonstrable contributions your research has made to society, for example fostering global economic performance, increasing effectiveness of public services and policy, and enhancing quality of life.

Public engagement Dissemination/Communication: academic and non-academic audiences. Lectures, seminars/workshops, organised events.


Table B.10 Consortia Advancing Standards in Research Administration Information (CASRAI)

Area Indicators


Impact on productivity: quantity of direct knowledge and innovation production, quality of direct or indirect knowledge and innovation production, quantity of derived knowledge and innovation production.

Capacity building Impact on capacity: human capacity, leadership capacity, physical capacity, integration capacity. Examples provided at the level of each subgroup.

Innovative and economic impact Impact on productivity: quantity of direct knowledge and innovation production, quality of direct or indirect knowledge and innovation production, quantity of derived knowledge and innovation production. Impact on society: Direct economic benefits to society, indirect economic benefits to society

Health and health sector benefit Impact on society: broader health benefits


Impact on society: broader social benefits, broader environmental benefits

Public engagement Impact on society: broader cultural benefits



81

Table B.11 Framework for tracking diffusion of research outputs and activities to locate indicators that demonstrate evidence of biomedical research impact

Area Indicators


Biological material identified or developed as a result of the research study. Books or book chapters resulting from the research study. Conference papers resulting from the research study. Conference posters resulting from the research study. Panel discussions resulting from the research study. Presentations submitted by members of a research study in response to a Call for Papers issued by organisers of a meeting or conference. Presentations given by members of a research study in response to a written invitation issued by organisers of a meeting or conference. Lecture provided by research investigators. Supplemental materials provided by research investigators. Meetings or symposia are organised and held by research investigators. Research data generated by the research study. Research data is deposited into a shared repository. Metadata from research data is shared with other parties. Research study findings lead to new direction and/or area of research. Research study findings lead to new field of research or discipline. Supplemental materials are deposited into a shared repository. Database or repository resulting from the research study. Research study invited to submit a funding application to a funding body. Research study secures funding. Non-peer-reviewed journal articles or report or other publication resulting from the research study. Trade publications resulting from the research study. Materials that supplement a publication (published or unpublished). Slides, videos or other materials resulting from the research study. Peer-reviewed journal articles resulting from the research study. Algorithm resulting from the research study. Conceptual framework resulting from the research study. Scientific analysis method resulting from the research study. New approach for researching a disease, disorder or condition resulting from the research study. Novel use of technology resulting from the research study. Theoretical construct resulting from the research study. Software resulting from the research study (alpha or beta). Research study findings represent a paradigm shift in a field. Research study findings lead to change in understanding a problem. Research study findings expand the knowledge in a field. Research study findings lead to a new approach to disease, disorder or condition. Publications generated by the research study are cited in a subsequent reference, including journal articles, books, book chapters, grey literature, dissertations, patents, web sources, and/or data records. Conference theme is organised around the topic related to the research study. Conference panel is created around the topic related to the research study. Research study findings are cited in a presentation. Research study findings are noted by a funding agency Research methodologies developed by the research study are used by others. Examples include:

RAND Europe

82

Area Indicators

• Scientific analysis method • Conceptual framework • Algorithm. Research study leads to change in bench or clinical research practices. New research generated as a result of the research study. New pilot award generated as a result of research study findings. Research study is renewed

Capacity building

Trainees or scholars involved in the research study. Examples of outcomes related to the training/education received are: • Trainees or scholars receive advanced degrees. • Trainees or scholars continue research at different institutions. • Trainees or scholars receive grant award funding for research. • Trainees or scholars engage in translational research Collaborative relationships result from the research study. Research study collaborations that continue after completion of a research study within the university or outside the university. Research study results in new collaborative relationship after completion of a research study within the university or outside the university. Research study members are invited to collaborate on a new research study within the university or outside the university. Training materials developed as a result of the research study. New institute or centre is formalised as a result of a research study. Research study findings result in enhancement of existing resources and expertise. Research study findings result in new laboratory methods or procedures.

Innovative and economic impact

Invention disclosures reported to the university/organisation as a result of the research study. Licence agreements or patents may be executed for inventions deemed to be of potential commercial value. Examples of inventions are: • Diagnostic product • Therapeutic product • Measurement instrument • Computer software. Licence agreement executed for invention or intellectual property generated by the research study. Measurement instrument developed as a result of the research study. Medical device or prototype developed as a result of the research study. Mobile application developed by the research study. Diagnostic application for identification of a disease, disorder or condition developed as a result of the research study. Screening tool for identification of a disease, disorder or condition developed as a result of the research study. Patent executed for invention generated by the research study. Potential new drug identified as a result of the research study. MTA is granted for transfer of tangible property generated by the research study. Licence is granted to outside parties for use of intellectual property or invention generated by the research study. Revenue generated from granting of licensing to outside parties for use of intellectual property or invention generated by the research study. Examples of licences include:


83

Area Indicators

• Number of licences granted. • Requestor status, individual or organisation. • Purpose. • Translations, if applicable. • Adaptations, if applicable. Spinoff company formed as a result of the research study. • Revenue generated. • Duration of spinoff. • Value at termination of spinoff company. • Number of employees hired. Start-up company formed as a result of the research study. • Revenue generated. • Number of employees hired.

Health and health sector benefit

Curriculum guideline refers to the research study as being significant, or for use as recommended, or background reading for more information. Research study findings are cited in teaching materials. Research study is cited in a meta-analysis. Potential new drug shows benefit during pre-clinical development trials. Investigational New Drug (IND) application accepted by FDA Physicians, policy makers or consumers contact research study investigators for more information on research findings. Research study cited in a review. Biological material application generated by the research study shows benefit during clinical trials. Biological material application generated by the research study registered/licensed with FDA. Biological material application generated by the research study used by health care providers and/or consumers. Research study cited in a clinical decision aid. Research study findings are synthesised in consumer health materials. Research study efforts result in recruitment and retention of clinical trial participants. Clinical Trial is successful (Phase I, II, III, or IV). Research study cited in a clinical point-of-care resource. Research study findings result in a clinically effective approach in the management and treatment of a disease, disorder or condition. Clinicians in the field report change in delivery of healthcare based on findings of the research study. Healthcare Common Procedure Coding System (HCPCS) code implemented as a result of the research study. International Statistical Classification of Diseases and Related Health Problems (ICD) code implemented as a result of the research study. Current Procedural Terminology (CPT) code implemented as a result of the research study. Research study findings cited in continuing education materials. Research study findings result in a diagnostic application for identification of a disease, disorder or condition. Research study findings result in a screening tool for identification of a disease, disorder or condition. Research study findings lead to identification of measures for disease prevention. Research study findings lead to eradication of a disease.

RAND Europe

84

Area Indicators

Clinical trial study testing of a medical device generated by the research study. Clinical data generated in support of marketing a medical device, 510(k); Investigational Device Exemption (IDE); or Premarket Approval (PMA) generated by the research study. Medical device generated by the research study shows benefit during clinical trials. Medical device generated by the research study registered/licensed with FDA. Medical device generated by the research study used by health care providers and/or consumers. Mobile application developed by the research study is used by health care providers and/or consumers. Drug generated by the research study shows benefit during clinical trials. Drug generated by the research study registered/licensed with FDA. Drug generated by the research study is listed on a drug formulary list. Drug generated by the research study is listed on the WHO Model List of Essential Medicines. Drug generated by the research study is prescribed by health care providers. Research findings result in a treatment or intervention that is judged as being clinically effective. Patients self-report increase in quality of life. Research study identifies new parameter leading to surveillance and reporting of a disease, disorder or condition. Research study findings lead to public awareness of risk factors of a disease, disorder, condition or behaviour. Research study findings lead to identification of risk factors of a disease, disorder, condition or behaviour. Research study findings result in patient decision materials to assist with healthcare decisionmaking. Research study findings are cited in materials for patients or the public. Research study findings result in increased performance, quality, and consistency in the delivery of health care services. Research study findings lead to enhancement of health promotion activities among community members. Research study findings lead to identification of a lifestyle intervention. Measurement instrument generated by the research study used by consumers. Research study forms partnership with community or other group to address a community-based need. Drug generated by the research study is used by consumers. Research study cited in private insurance benefit plan in support of coverage. Research study cited in a public insurance benefit plan in support of coverage. Research study findings result in clinically effective approach in the standard of care for a disease, disorder or condition. Research study findings result in reduced costs in the delivery of health care services. Research study findings result in a cost-effective intervention for a disease, condition or disorder. Research study findings lead to identification of measures for disease prevention. Research study findings lead to eradication of a disease. Research study findings result in enhancement of existing resources and expertise. Research findings result in a treatment or intervention that is judged as being clinically effective. Patients self-report increase in quality of life. Research study findings result in increased performance, quality and consistency in the delivery of health care services. Research study findings result in increased life expectancy. Research findings result in reduction of disease burden.


85

Area Indicators

Research findings result in reduction of morbidity. Research findings result in reduction of mortality. Research findings results in reduction of prevalence of a disease, disorder or condition. Research findings result in reduction of incidence of a disease, disorder or condition. Drug generated by the research study results in a cost-effective means of management of a disease, disorder or condition.


Research study findings are noted in a trade publication, government documents, industry reports, public policy documents or other publications. Research study members invited to serve on advisory boards. Research study members invited to serve on committees for policy development. Research study cited in development of guidelines issued by organisations. Research study cited in enactment of federal, state or local legislation or regulation. Research study cited in enactment of federal, state or local policy. Research study cited in enactment of non-governmental policy. Policy-making organizations report that research findings resulted in change in policy. Research study cited in enactment of standards, such as American National Standards Institute (ANSI) or the International Standards Organization (ISO). Testimony based on research outputs and/or activity is presented before a legislative body. Research study cited in a guideline issued by a government agency. Research study cited in a guideline issued by a specialty organization related to the field of study. Research study cited in a guideline issued by a non-government organization Best practices guidance document developed as a result of the research study. Research study findings are noted in a trade publication, government documents, industry reports, public policy documents or other publications.

Public engagement

Research investigators invited to provide a presentation based on research findings to the public. Research investigators invited to provide a presentation at a non-professional conference. Research investigators meet with policy-makers and other stakeholders to review research findings. Number of attendees at outreach visits. Other types of outreach activities. Social media accounts (Twitter, blogs, Facebook, etc.) utilized by the research study to disseminate results from the research study. Research investigators create a YouTube video about the research study. Research investigators create a Wikipedia or other wiki entry based on the research study. Website developed for the research study. Mass media publication or broadcast refers to the research study. Research investigators are interviewed by the media. Research study investigators receive feedback as a result of an outreach effort.

Source: Bernard Becker Medical Library (n.d.)

Table B.12 Indicators for measuring the impact of health research

Area Indicators


Number of products published (e.g. peer-reviewed papers) Institute for Scientific Information (ISI) impact factors and half-lives Number of products targeted at specific decisionmakers (e.g. summaries of the take-home

RAND Europe

86

Area Indicators

message from a body of research)

Capacity building

N/A


N/A


N/A


Number of interactions with decisionmakers held at the request of researchers (e.g. academic detailing visits, policy briefings) Decisionmakers’ awareness of research (and its source), knowledge of research (and its source), attitudes towards research (and its source) Decisionmakers’ actual use of research in the context of competing influences on the decisionmaking process Number of information requests by decisionmakers (looking for specific research) Number of web hits by individuals with domain names, suggesting a decisionmaker organization (looking for specific research with potential for more web hits in future) Number of newsletter subscriptions from individuals with mailing addresses for decisionmaker organizations (looking for research in general) Number of interactions with decisionmakers held at the request of decisionmakers (e.g. face-to-face meetings) Number of research projects commissioned by decisionmakers Decisionmakers’ awareness of research organizations’ expertise, knowledge of research organizations‘ expertise, attitudes towards research organizations’ expertise (e.g. objective versus biased) Decisionmakers’ self-reported use of research organizations as an information resource (i.e. for answering questions unrelated to the original reason that the organizations came to their attention) Decision-makers’ actual use of researcher organizations as an information resource Research organizations involve decisionmakers in the research process (i.e. establishing the research agenda, conducting research, and/or identifying implications of the research findings) Decisionmaking organizations involve researchers in the decisionmaking process Decisionmakers’ assessment of how they were involved in the research process Researchers’ assessment of how they were involved in the decisionmaking process Research organizations’ research reflects (at least in part) the research needs and context of decisionmakers Decisionmaking organizations’ decisions reflect (at least in part) the research available to them

Public engagement

N/A

Source: Lavis et al. (2003)


87

Table B.13 Metrics to measure the impact of Tommy’s31 work

Area Indicators


Practitioner and patient perspectives on the relevance of the proposed programme of research (e.g. through survey) Satisfaction of staff and participants (at different levels) with the management and government systems Feedback from clinics/parents (e.g. by survey) on the relevance of research to their practice. Number (and range) of references in materials aimed at patient and staff audiences (e.g. patient group materials, practical toolkits). Long-term increases in the quality of research in the field

Capacity building Key contributions to the creation, development and maintenance of major research resources (e.g. health information systems, software, databases) Academic careers advancement Number of Tommy's researchers in senior research positions – e.g. professors Next destinations of fellows – proportion staying in prenatal research Improved research literacy among health staff Strong and ongoing links with clinicians Number/proportion of Tommy's research publications with an NHS-based author Number and range of NHS staff who are involved in research Number and range of NHS maternity and neonatal staff who consider research a core part of their role Number of NHS staff who have had training in research (of some form). Proportion of centre research conducted with NHS partner Impacts of Tommy’s funded research on improving training in the field of interest Impact of Tommy’s funded research on career development of experts in the field Numbers of available formalised career tracks around maternity research, and uptake of these


Patents Development and sale of spinout companies Licences awarded and brought to market Number of projects with industry partner


Number of citations in systematic reviews Compliance and adherence to research-informed policies and guidelines Addressing barriers to the use of research-informed interventions in the health system Improved patient satisfaction Reduced adverse events/complications Cost savings/cost reduction in delivery of existing services Increased service effectiveness The use of Tommy's research evidence by different groups involved in clinical diagnosis and decisionmaking Patient-reported outcome measures (PROMS) Patient satisfaction and experience surveys

31 Tommy’s is a UK-based charity focused on reducing infant mortality and neonatal deaths.

RAND Europe

88

Area Indicators

Qualitative data on waiting times and service accessibility Adoption of health technologies Cost-effectiveness assessment of introducing a new health technology New funding attributed to intervention Better targeting, accessibility, utilisation and coverage of health services NHS Choices end-user visit rate Improved health of patients Improved quality of care metrics Numbers of lives touched Narrowing of health/health care disparities Improved awareness of preventative measures in the community Prevalence, incidence, QALYs, patient-reported outcome measures Adherence to clinical guidelines Number of industry-funded trials taking place within institute/clinic Total value of industry-funded trials taking place


Interactions between researchers and policymakers/strong and ongoing links with policymakers and advocacy groups Number of citations in policy documents Number of citations in clinical guidelines Number of citations in ongoing health professional education material Number of presentations to policy and decisionmakers Number of invitations from policymakers Number/size of requests for research to support policy Number of citations in curricula for new researchers Number of Tommy's researchers on key boards and committees (e.g. National Institute for Health and Care Excellence i.e. NICE, Public Health England i.e. PHE)

Public engagement Number and range of practitioners engaged in each of the following project stages: prioritisation and idea development; project planning; creation of research materials; data collection; data analysis; reporting. Research cited in advocacy publications Number and range of participants at events Level of response to media coverage Level of participation in clinical trials Recruitment to trial performance Attitudes of research participants towards research

Source: Deshpande et al. (2018)

Table B.14 Metrics to Assess and Communicate the Value of Biomedical Research

Area Indicators


Health Care Professionals (HCPs), Multi-National Corporations (MNCs). Overall and star researchers Mentions in social media Number and size of awards from major funders - NIH, Howard Hughes


89

Area Indicators

Use of infrastructure by other researchers Catalogue of infrastructure Number and total value of awards, repeat funding Success rate of applications High grant application success rate Number of research output downloads

Capacity building Level and networks of co-authorship Number of collaborations on grant applications and projects Total number of collaborators across all projects Number of research projects engaging community organisations Grade Point Average (GPA) of graduates Subject coverage of Personal Development Planning (PDP) Uptake of PDP Feedback on PDP K to R conversion rate Five-year career outcomes for Doctor of Philosophy (PhD) graduates Number of publications per PhD graduate Number of PhD graduates and proportion completing PhD Improved educational attainment/reduced dropout rate Editorships, membership of editorial board of high profile journals Number and type of prizes Number of (international) speaker invitations, conference invitations Career outcomes for researchers Longitudinal data on career progression of students


Level of (local) spending Direct employment effects Number and type of new offices (i.e. could be subsidiary) in area Regional employment data Number of patent applications and awards Number of licensing agreements and Licensing revenue Patent citations Number and size of consultancy agreements Number of new treatments and tools Number of spinouts Venture Capital (VC) money invested in start-ups List/examples of know-how taken up by industry Patents awarded and licensed Existence of IP policy Size of tech transfer office Fraction of indirect costs covered Contract funding from industry Industrial research funding PhD placements/secondments, PhD funding Number of projects in collaboration with an industry partner Costs and benefits of a project or set of projects Number of private sector innovations/products/devices brought to market

Health and health sector Improved health status/life expectancy among the community served/patients

RAND Europe

90

Area Indicators

benefit Number of cutting-edge treatments available – i.e. those that were developed elsewhere but in use here Number and description of treatment improvements developed in house Percentage, number, and range of types of clinicians (e.g. nurses, Allied Health Professionals i.e. AHPs) on research projects. Quality of care metrics Number of uses of research infrastructure in clinical practice


Requests and invitations from policy makers Number of citations on clinical guidelines Number of citations in policy documents Number of research/policy secondments

Public engagement Number of staff engaged in outreach Number of people attending outreach events, and their perceptions Level of participation in clinical trials Existence of specifically-tailored material for different community groups Number of research projects engaging community organisations for the entire duration of research projects Existence of community material aimed at addressing health disparities

Source: Guthrie et al. (2016)

Table B.15 A framework for assessing the impact of health services and policy research on decisionmaking

Area Indicators


Partnered (co-invested) funding investments [name of organisation, investment dollar ($) amount and per cent (%) total, area of research]

Number (#) of organizations whose primary purpose is not scholarship or education that support health services and policy researchers and/or trainees (over time).

Leveraged funding from follow-on funding.

Important problems warranting Health Service and Population Research (HSPR) attention are co-identified with decisionmakers [number (#) and description of type of problem].

Number (#) and type of HSPR funding programs/projects according to HSPR priority theme areas

Trend in funding investments over time for HSPR [per cent (%) growth of HSPR funding over time, open and strategic, and by HSPR priority theme areas].

Capacity building Trend in number (#) and per cent (%) growth of HSPR applicants and funded researchers at all career stages (e.g. masters, doctoral, postdoctoral, new investigator, mid-career, clinician scientist, embedded researcher).

Number (#), per cent (%), and type of HSPR trainees (e.g. PhD, etc.).

Number (#) of HSPR trainees graduated.


91

Area Indicators

Trend in number (#) and per cent (%) of health services and policy researchers working in healthcare delivery, coordination, or policy organizations.

Number (#), per cent (%) and type of HSPR trained staff (e.g. trainees, others) in healthcare delivery, coordination, or policy organizations.

Number (#) of health services and policy researcher and NGO (non-governmental organization) collaborations


N/A


Trend in number (#) and per cent (%) of health services and policy researchers working in healthcare delivery, coordination, or policy organizations.

Number (#), per cent (%) and type of HSPR-trained staff (e.g. trainees, others) in healthcare delivery, coordination, or policy organisations.

Number (#) of health services, policy researcher and NGO collaborations.

Number (#) and per cent (%) of policies that cite research evidence

Research evidence directly informing agenda setting, priority setting, policy debates, briefings: e.g. invited policy papers and consultancies, information requests by decisionmakers, invited meetings, and interactions with decisionmakers.

Research directly underpinning policy decisions (e.g, legislation, regulation, programmmes, practice, behaviour, service delivery).

Evidence of participation of researchers in the process of making decisions (e.g. participation in policy networks, boards, advisory groups).

Number (#) and per cent (%) of policies with use of HSPR evidence in their development.

Number (#) of public service and broader public sector organisations formally requiring the use of research to inform health services and policy (over time).


Number (#) and per cent (%) of policy/decisionmakers’ self-reported use of research.

Public engagement Number (#) of end users satisfied with engagement with researchers.

Number (#) of HSPR projects that include meaningful participation of patients or members of the public as appropriate

Number (#) of HSPR researchers engaged in capacity development with end-user audiences.

Number (#) and per cent (%) of end users that reported HSPR evidence was useful.

Source: CHSPRA (2018)

Table B.16 An approach to evaluate the impact of an applied prevention research centre

Area Indicators

Knowledge production and Stability of centre funding for indirect/overhead costs

RAND Europe

92

Area Indicators

research targeting Number or percentage of centre projects driven by expressed policy/practice needs of engaged policy/practice organizations Evidence that research, policy and practice perspectives have jointly informed centre priorities and activities Number of centre projects involving co-production of knowledge with knowledge users

Capacity building N/A


Number or percentage of centre projects driven by expressed policy/practice needs of engaged policy/practice organizations Evidence that research and policy and practice perspectives have jointly informed centre priorities and activities Number of centre projects involving co-production of knowledge with knowledge users Use of research in informing public health programmes and policy: input from public health practitioners/policymakers about what research has been used to inform their work


Number or percentage of centre projects driven by expressed policy/practice needs of engaged policy/practice organizations Evidence that research and policy and practice perspectives have jointly informed centre priorities and activities Number of centre projects involving co-production of knowledge with knowledge users Use of research in informing public health programmes and policy: input from public health practitioners/policymakers about what research has been used to inform their work Citations in public policy documents Evidence of centre contributions to supporting processes and groups (e.g. steering groups, ministerial working groups, and government committees)


N/A

Public engagement Number and type of knowledge exchange activities undertaken by the centre

Source: Willis et al. (2017) 32

32 Indicators identified as high importance and high feasibility to measure in modified Delphi exercise.

93

Annex C. TRIS indicators

Table C.1 TRIS indicators

Domain Sub-domain Indicators

Rese

arch

-rela

ted

impa

cts

Research direction and resources

Research needs are identified, such as gaps in research, unanswered questions, or identification of new areas for investigation

New methods are developed for collecting and storing data and/or building new database systems

New research methods or techniques are developed or extended

Translational research concepts, definitions, and terminology are better defined or elucidated (e.g. the nomenclature of T1–T4 to describe phases of translational research)

Effective new research networks and collaborations are formed

Qualified research personnel are recruited

Research management and conduct

Improved methods for recruitment of study participants are developed and/or implemented

IRB processes are improved (e.g. shortening the time to approval, ease of completion, and appropriateness of decisions)

New research projects and teams are formed

Translational research studies are successfully completed

Improved retention and continuity of research team/teams is achieved

Improvements in the quality or quantity of teamwork across disciplines occur

Process efficiencies in the conduct of research are achieved

Researchers or staff are recognised for leadership in the field

A particular translational study or the research unit receives an award

Internal communication among individual researchers and among research units is improved

More of the unit’s researchers and staff serve on regional or national research organisations (e.g. NIH study sections)

Significant positive changes in the research environment and culture occur

Barriers in the research process are identified and strategies for overcoming those barriers are developed

There is an increased emphasis on conducting practice-based research (in ‘‘real-

RAND Europe

94


world’’ clinical settings)

Research methods

New analytical techniques are developed

New or improved mixed-methods approaches are used

New or improved methods for conducting studies spanning multiple disciplines are established

New methods of synthesising results from varying disciplines are implemented

New ways of measuring outcomes from translational research studies are adopted

Research results

The number and rate of grant submissions among translational researchers increases

The number and rate of grant awards among translational researchers increases

Novel or innovative discoveries are made

New scientific knowledge and/or techniques result

Efficacy of a new treatment (or new application of an established treatment) is demonstrated

New medical or research devices or products are developed

Patent/patents for a biomedical device or product are obtained

New biomarkers to accelerate delivery of health care are identified and/or validated

A discovery (e.g. drug, device, or measurement instrument) is brought to market

Research dissemination

The number of publications authored or co-authored by translational researchers increases

The average impact factor of journal articles authored or co-authored by translational researchers increases

The participation of translational researchers in making presentations at national conferences increases

The total number of citations for articles authored or co-authored by our researchers increases

Media coverage for articles or projects involving the unit’s researchers increases

There is greater diffusion of knowledge and techniques in a broad context (e.g. in community-based health care practice)

More of the unit’s researchers serve as journal editors or members of journal editorial boards

Scientific advances by a unit’s researchers (e.g. study results) reach a broader array of audiences


95

Domain Sub-domain Indicators Tr

ansla

tiona

l im

pact

s

Translational impacts

Findings and discoveries from bench science are incorporated into studies involving animals or humans

Findings and discoveries from clinical trials are incorporated into clinical guidelines, or otherwise accepted as good medical practice

Improvements in the delivery of effective and efficient health care services are made

The effectiveness of different treatment and intervention choices are determined in a manner that is useful for clinicians

Improvements result in better quality of patient care

The incidence of medical errors decreases

Increased training in translational research methods occurs among health care providers and support personnel

Health information technologies are enhanced

Techniques are put into place that effectively constrain costs and enhance cost effectiveness

Translational research training development increases

New prevention techniques are incorporated into medical practice

Medical practice becomes more evidence-based

Patient outcomes improve

Positive health behaviours expand (within particular populations)

Clinical guidelines are developed and promulgated

Advancements in personalised health care are made based upon genetic sequencing

The patient–clinical relationship is demonstrably strengthened

The translation of research findings into clinical practice and outcomes accelerates

Soci

etal

impa

cts

Policy development

Policies and procedures for protection of human subjects are strengthened

There is improved regulation of new technologies and devices

Clinical and translational science is conducted consistent with recognised ethical principles

Community-based health programmes are developed

There are Improved reimbursement practices and policies for providers

Self-efficacy and empowerment among health care consumers increases

Research findings help to inform the decisionmaking process and policy development

Community improvement

Improvements are made in health literacy of consumers and specific populations

The health status of consumers and specific populations is improved

Economic benefits are obtained (e.g. greater employment opportunities, medical cost savings)

Job growth and development expand in a particular area or region

Effective public health initiatives are established in a particular area or region

Disparities in health and the provision of health care are reduced

RAND Europe

96


Organisational coordination raises awareness and stimulates enhanced community programming concerning health issues

New knowledge regarding sustainable development helps to promote and protect the health of communities

Consumer resources and behaviour

Consumer understanding and support of translational research increase

Consumers feel more empowered about health issues and their ability to affect change

There is improved communication and understanding about health risks among consumers

Health education and health literacy expand

Source: RAND Europe

Existing indicators to measure the biomedical innovation ... · IT Information Technology KRL...

Documents

Transcript of Existing indicators to measure the biomedical innovation ... · IT Information Technology KRL...