Assessing the Impact of Health Research and Innovation: A Systems Approach

CASRAI Canada ReConnect14 November 20, 2014

Dr. Kathryn Graham Director, Performance Management and Evaluation

Alberta Innovates Health Solutions

Overview

1. Understanding health research and innovation impact: A Canadian approach

2. Promising tools for addressing system needs

3. Building capacity to optimize the Canadian environment in assessing impact

1. Understanding Health Research and Innovation Impact: A

Canadian Approach

Impact

Impact Assessment Defined

Assesses the changes that can be linked to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. …...

(Source: ISRIA 2014; World Bank; GAO 2012)

“Making a Difference”

Why Assess Impact?

A Canadian Approach: Systems Level Framework

CAHS Impact Framework

(Source: CAHS, 2009)

CAHS Impact Framework (con’t)

Five Impact Categories: 1. Advancing Knowledge2. Capacity Building3. Informing Decision Making4. Health Benefits5. Broad Economic and Social Benefits

CAHS Impact Framework (con’t)

Library of 66 Impact Indicators

CAPACITY BUILDING Indicator Description Level of Application Category

PERSONNEL

Graduatedresearch students in health‐ relatedsubjects

* Number of graduated PhD/ MSc/MD, annual, year over year* Should be able todisaggregate to subjects, gender, etc.

* Not recommended atthe individual level* Can be used atinstitutional level* Most usefulprovincially or nationally

* As an aspiration, we would like to track the success of training programs in producing outstanding scientists and the progress that all research graduates make* Could be done in part using the StatisticsCanada National Graduate Survey

2. Promising Tools for Addressing System Needs

System Needs and Challenges

• “Large-scale social change requires broad cross-sector coordination, yet the social sector remains focused on the isolated intervention of individual organizations….

Source: FSG website – Collective Impact

• The funding ecosystem consists of multiple funders investing in multiple research projects with many contributions. As funders we are challenged to tell policy and decision makers about the overall impact of our research and innovation investments

Macro Level View

Source: Trochim, W., Kane, C., Graham, M., and Pincus, H. (2011). Evaluating Translational Research: A Process Marker Model. Clinical and Translational Sciences 4, 153-162.

Systems Approach: Horizontal

Systems Approach: Measurement

Source: FSG Social Impact Advisors

1. Understand the context

2. Identify the assessment purpose

3. Measure: Define Indicators of success

4. Develop the design, methods and data collection

5. Communicate and use findings

6. Manage the assessment

Common Approach: Alberta Protocol for Research Impact Assessment

AIHS: Common Language for Classifying Investments

Health Research Classification System

Systems Common Language: Classifying Investments

INFORMING DECISION MAKING COMMERCIALIZATION

$309 Leveraged

$51 Invested

LEVERAGED FUNDING (Millions) CAPACITY BUILDING

AIHS: Measuring Impacts Quantitatively (ROI Dashboard)

AIHS: Measuring Impacts Qualitatively

AIHS was part of a three- year international study to improve the translation of mental health research from bench to bedside using 25 years worth of data.

In 2011-12, AIHS invested over $11 million and supported projects in mental health…

95

“Research that had researchers span boundaries had higher impact ”

Alberta Innovates Health Solutions

• Across 4 Research Pillars

• 12 Areas of Strategic Focus

Research Capacity

• $20m invested

• $28m in attracted funding

•209 research investigators

• 37 international recruitments

Res

ea

rch

Res

ult

s Knowledge Pool

1196

C

oll

ab

ora

tio

ns

wit

h o

the

r R

es

ear

ch

ers

Health System Collaborations

• Task forces• Primary care networks

24 partnerships with Industries

•17 Reports of invention/invention disclosures

Collaborations with Government

• Change in policies

collaborations with Non-profits

• Guidelines

Health Care•Improvements to health care effectiveness• Improvements to health care efficiency & accessibility

Improvements in:• Therapeutics • Diagnostic tools• Clinical practice guidelines

Determinants of Health• Personal behaviour• Social/cultural determinants• Environmental

Improvements in health and wellbeing • Morbidity• Mortality• Quality of life

Economic & Social Impact

Impacts feed back into inputs for future research

National/ Global Research

Research activity

That produces results

That influences decision making in….

That affects healthcare and health risk factors

That contribute to changing health, social and economic wellbeing

Health Status and function, well-being, economic conditions

Research Diffusion

AdvancingKnowledge

Capacity Building

InformingDecisionMaking

HealthImpact

SocioeconomicImpact

• 750 publications

• 49% peer review

• 32% abstracts

•552 public outreach activities

• 107 at an international level

• Invited talks

•11 Patent applications • 6 Spin-off companies

The Public• Media events

AIHS: Measuring Across the CAHS Framework

Systems Common Tool: Measuring Impact

• A research portal to capture and analyze common research outputs so impacts can be tracked over time and across funders

• Can reduce administrative burden and be cost effective

• Provides an opportunity to capture “true” impact and contributions

3. Building Capacity to Optimize the Canadian Environment in

Assessing Impact

AIHS website for work on health research impact and the international school http://www.aihealthsolutions.ca/performance.

Impact Assessment: Alberta

Key Messages• Share frameworks and tools to support the

optimization of research and innovation impact in Canada

• Common approach - CAHS Impact Framework- Alberta Protocol

• Common language - CASRAI

• Common tools and measurement systems

• Continue collaborative efforts to measure the impacts of the Canadian health research and innovation system

Thank You

Dr. Kathryn GrahamDirector of Performance Management and EvaluationAlberta Innovates - Health Solutionskathryn.graham@albertainnovates.ca

Questions?1. Is there interest in

implementing a common approach, language, tools, and measures in Canada in the assessing impact and that can be compared internationally?

2. What are the advantages of this approach?

3. What are the disadvantages of this approach?