When developing an evidence-informed theory driven program · Capability, Opportunity, Motivation...
93
Making implementation science practical: an implementation primer for researchers October 4, 2018
Transcript of When developing an evidence-informed theory driven program · Capability, Opportunity, Motivation...
Making implementation
science practical: an
implementation primer
for researchers
October 4, 2018
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
The three stages of expertise
Source: Wardley, S (2008) The three
stages of expertise [Blog post]
http://blog.gardeviance.org/2008/04/t
hree-stages-of-expertise.html)
2
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Day 1 Recap
• KT is an umbrella term that includes many phases
• We use models, theories, and frameworks to apply KT
• When developing an evidence-informed theory driven program:
o define ‘what’ and ‘who’ needs to change
o identify and assess barriers and facilitators to change and use this
information to identify‘ how’ change will be implemented
• What makes research KT research/how to develop KT research
questions?
3
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Models, Theories and Frameworks
MTF Description Reference
KTA
.
Knowledge to Action model is a process
model that defines and describes
Knowledge Translation processes and
outlines strategies for building Knowledge
Translation capacity
Graham, D., et al. (2006). Lost in
translation: Time for a map?. Journal of
continuing education in the health
professions, 26 (1): 13-24.
COM-B
Capability, Opportunity, Motivation –
Behaviour is a behavior change theory.
Michie, S. et al. (2011). The behaviuor
change wheel: A new method for
characterizing and designing behaviour
change interventions. Implementation
Science, 6:42.
TDF
Theoretical Domains Framework is a
framework that was developed to make
behavior change theories more accessible
in relation to implementation practices.
Cane, J., et al. (2012). Validation of the
theoretical domains framework for use in
behaviour change and implementation
research. Implementation Science, 7(1).
CFIR
Consolidated Framework for Implementation
Research is a framework that can be used
to understand the implementation context.
Damschroder, L., et al. (2009). Fostering
implementation of health services
research findings into practice: a
consolidated framework for advancing
implementation science. Implementation
Science, 4:50.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Models, Theories and Frameworks
MTF Description Reference
QIF
.
Quality Implementation Framework is a
framework that provides a synthesized
overview of critical steps of implementation
that can be used as a guide for practice and
research.
Wandersman, et al. (2012). The Quality
Implementation Framework: A Synthesis
of Critical Steps in the Implementation
Process. American Journal of
Community Psychology, 50(3-4):462-80.
doi: 10.1007/s10464-012-9522-x.
ISF
Interactive Systems Framework is a
framework that outlines the
people/organizations/contexts needed to
move effective prevention into the field.
Wandersman, A., et al. (2008). Bridging
the gap between prevention research
and practice: The interactive systems
framework for dissemination and
implementation. American journal of
community psychology, 41(3-4): 171-
181.
RE-AIM
A framework for evaluating public health
interventions that assesses 5 dimensions:
reach, efficacy, adoption, implementation,
and maintenance. These dimensions occur
at multiple levels (e.g., individual, clinic or
organization, community) and interact to
determine the public health or population-
based impact of a program or policy.
Glasgow RE, Vogt TM, Boles SM:
Evaluating the public health impact of
health promotion interventions: the RE-
AIM framework. Am J Public Health.
1999, 89: 1322-1327.
10.2105/AJPH.89.9.1322.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Outline for day 2
6
1.0 What do you need to consider when implementation planning?
7
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Learning Objectives
• Explain the importance of identifying the
“Who,” “Where,” and “How” when planning
for implementation
8
Stuart Miles/freedigitalphotos.net
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Implementation planning is essential
“In preparing for battle I have
always found that plans are
useless, but planning is
indispensable.”
- Dwight D. Eisenhower
9
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Where is “implementation” on the KTA process model?
10
Source: Graham ID et al. JCHEP 2006;26:13-24.
Needs
unpacking!
Requires
another
framework to
operationalize
this step.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
The Quality Implementation Framework
1. Setting & assessment strategies
o Needs assessment
o Fit assessment
o Capacity/readiness assessment
o Possible adaptations
o Obtain buy-in & foster supportive environment
o Build general organizational capacity
o Recruit/maintain staff
o Train staff
2. Create structure for implementation
o Create implementation teams
o Develop implementation plan
3. Ongoing structure and support
o Technical assistance/coaching/supervision
o Process evaluation
o Feedback mechanism
4. Improving future applications
o Learning from experience
11
• WHO o Create implementation teams
o Recruit/maintain staff
o Train staff
o Obtain buy-in & foster supportive environment
• WHERE o Needs assessment
o Fit assessment
o Capacity/readiness assessment
o Possible adaptations
• HOW o Technical
assistance/coaching/supervision
o Build general organizational capacity
o Use implementation tools
Source: Adapted from Meyers, Durlak, & Wandersman (2012). American Journal of Community
Psychology; 50:462-08. doi: 10.1007/s10464-012-9522-x.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Questions to consider when planning to implement a program
12
WHO is involved in implementation efforts?
WHERE is implementation happening?
HOW you monitor and evaluate implementation?
HOW will you sustain, spread or scale up implementation?
2.0 Who is involved in implementation
13
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Learning Objectives
• Explain the importance of identifying all the
different roles involved in the
implementation process (i.e., who will
support implementation, who will be part of
the implementation team, etc.)
14
Stuart Miles/freedigitalphotos.net
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Source: Wandersman A, et
al. Bridging the gap between
prevention research and
practice: the interactive
systems framework for
dissemination and
implementation. Am J
Community
Psychol. 2008;41(3–4):171–
181
Delivery System
Support System
Synthesis & Translation System
Delivers interventions/integration
strategies at the frontline
Synthesizes evidence and creates
knowledge products
Develops capacity and provides support
for implementation
Funders
Policy-
makers
Primary research
Program Recipients
The target of the
interventions/integration strategies
15
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
WHO is doing the implementing?
Who is on the implementation team?
Who will need to support
implementation?
16
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Creating implementation teams
• What is an implementation team?
o A core group of individuals accountable for guiding implementation, sustainability, and scale-up of ETPs
o NOT advisory groups, committees or representatives providing periodic input
• Who should be on an implementation team?
o 3-5 individuals who have dedicated time to support collaborative work (collective problem-solving, communication, goal setting, planning and task coordination)
17
Source: The National Implementation Research Network’s Active Implementation Hub (NIRN). Module
3:Implementation Teams http://implementation.fpg.unc.edu/module-3/
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Implementation teams have many essential functions
18
Metz & Naoom , University of North Carolina , National Implementation Research Network, 2014
• Can promote implementation of the core components of the ETP
Know the ETP
• Can guide implementation efforts based on best practices
Know and apply implementation science
• Can work at multiple levels of the system to create enabling environments and influence change
Promote and participate in systems changes
• Can use data to determine implementation quality and inform improvements
Know and apply monitoring and evaluation
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Who else is involved in implementation?
Who needs to be considered and engaged in implementation?
• Across the system?
• At the formal leadership level?
• At the informal leadership level?
• At the practice level?
What exactly is their stake in the ideal practice or implementation challenge?
• What are their viewpoints?
• Can they affect the plan?
• Are they affected by the plan?
19
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How can they be engaged based on their stake?
• What drives and motivates the stakeholder?
Values: Things people care most about
• Who are the stakeholders accountable/ committed to?
Accountabilities: Commitments to people and to the people behind the
ideas that matter to them.
• What might the stakeholder risk losing if things change/stay the same?
Losses: Resistance to change stems from fear of losing something important (e.g., identity, comfort, reputation, time,
status, security, independence, resources).
• How could the stakeholder benefit if things change/stay the same?
Benefits: Things people stand to gain from the change/initiative.
20
Source: Heifetz, Grashow, & Linsky (2009). The Practice of Adaptive Leadership: Tools and Tactics
for Changing Your Organization and the World. Harvard Business Press.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Questions or comments about implementation teams?
21
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #1: Identify the different stakeholders needed for
implementation
22
1
Think about your project and identify
different stakeholder groups you may
need to engage to support
implementation. Write the stakeholder
group, their role in the implementation
process, and your rationale for selecting
them in the table in your Activity
Workbook.
We will discuss responses as a larger
group. 2
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #1 discussion: Identify the different stakeholders
needed for implementation
• Did you have any challenges or “aha” moments when identifying
stakeholders need for implementation?
23
3.0 Where is implementation happening?
24
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Learning Objective
1. Discuss the importance of assessing the
implementation context
2. Describe how to use the Consolidated
Framework for Implementation Research
(CFIR) to assess the context
3. Describe how to proactively plan for
adaptations based on context
25
Stuart Miles/freedigitalphotos.net
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Implementation can happen at many different levels
26
Country/province Organization Provider Patient/family
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Why is it important to consider context?
27
Evidence-based practice effectiveness
Implementation strategy
effectiveness
Implementation quality
Enabling Context
Significant Health
Outcomes
Source: National Implementation Research Network.
http://nirn.fpg.unc.edu/learn-implementation/implementation-defined
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Surveys
Interviews/
focus groups
Observations
Discussions with
key stakeholders
Literature Reviews
How do you assess context?
28
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
The Consolidated Framework for Implementation Research
29
Inner Setting
Structural characteristics
Networks and communications
Culture
Implementation climate
Readiness
Outer Setting
Patient needs/resources
Cosmopolitanism
Peer pressure
External policies/incentives
Individual Characteristics
Knowledge & Beliefs
Self-efficacy
Stage of Change
Identification with Organization
Personal Attributes
Process
Planning
Engaging
Executing
Evaluating
Intervention Characteristics
Intervention Source
Evidence Strength & Quality
Relative Advantage
Adaptability
Trialability
Complexity
Design Quality & Packaging
Cost Source: Damschroder et al. (2009). Implementation Science; 4:50.
doi: 10.1186/1748-5908-4-50
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Does the ETP fit the context of where you are
implementing?
30
Intervention Characteristics
Intervention Source
Evidence Strength & Quality
Relative Advantage
Adaptability
Trial ability
Complexity
Design Quality & Packaging
Cost
Source: Damschroder et al. (2009).
Implementation Science; 4:50.
doi: 10.1186/1748-5908-4-50
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Does the ETP fit the inner context of where you are
implementing?
Inner Setting
Structural characteristics
Networks and communications
Culture
Implementation climate
Readiness for implementation
31
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is the implementation climate?
Inner Setting Implementation
Climate
Tension for change
Compatibility
Relative priority
Organizational incentives & rewards
Goals and feedback
Learning climate
32
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is the readiness for implementation?
Inner Setting Readiness for
Implementation
Leadership engagement
Available resources
Access to knowledge & information
33
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Does the ETP fit the outer context of where you are
implementing?
Outer Setting
Patient needs/resources
Cosmopolitanism
Peer pressure
External policies/incentives
34
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Context and people influence adaptations
35
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Questions or comments about assessing the
implementation context?
36
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #2: Consider the context
37
1
Thinking about your project, use the
CFIR table in the Activity Workbook to
identify 5 contextual level barriers and
facilitators to implementing your
evidence-based practice.
We will discuss responses as a larger
group.
2 In your small group, discuss how you may
use this information about the context to
proactively adapt your implementation
strategies
3
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #2 discussion: Consider the context
• Did you have any challenges or “aha” moments when considering the
context for implementation?
38
4.0 How do you evaluate implementation?
39
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Learning Objectives
1. Describe when and how to
evaluate (distinguish between
formative, process and
outcome evaluations)
2. Discuss evaluation
frameworks and indicators
40
Stuart Miles/freedigitalphotos.net
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Where does ‘evaluation’ fit in the KTA process model?
41
Process
Outcome
Impact
Formative
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is a formative evaluation?
What it shows:
• Needs
• How target audience receives information
• Ideal champions/ spokespeople
• Challenges in using program content
Why it is useful:
• Making sure there’s a need
• Clarification
• Make revisions
• Maximizes the likelihood that the program will succeed
When to use it:
• Program development or revisions
• Examples:
• Needs assessment
• Usability testing
• Implementation planning
• Stakeholder engagement
42
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is a process evaluation?
43
What does it show?
• Activities conducted
• Characteristics of activities conducted
• (e.g., how many people are participating in the program and how many people are not)
Why is it useful?
• Identifies challenges in reaching the target population
• Allows programs to evaluate how well their plans, procedures, activities, and materials are working and to take timely, corrective actions
When is it done?
• As soon as the program begins
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is an outcome evaluation?
44
What it shows:
• The degree to which a program is meeting its goals
• For example, changes in knowledge, skills, attitudes, beliefs, behaviour, patient/other outcomes
Why it is useful:
• To make adaptations
• Is program moving toward achieving goals
When to use it:
• After the program has made contact with at least one person or one group of people in the target population
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is an impact evaluation?
45
What it shows:
• The degree to which the program has met its ultimate goals
Why it is useful:
• Allows programs to learn from their successes and failures and to incorporate what they have learned into their next project
• Provides evidence of success for use in future requests for funding
When to use it:
• For ongoing programs
• For one-time programs
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Why is it important to monitor the process of
implementation by measuring implementation quality (IQ)?
46
• Better IQ Better Outcomes
• Measuring IQ Better IQ
• Measure IQ Unpack black box
• IQ to show funders what you’ve done
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
1. Dosage
2. Adherence/ Fidelity
3. Adaptation
4. Quality of delivery
5. Participant responsiveness
6. Reach
How do we measure implementation quality?
47
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
48
How much of the original was delivered?
1. Dosage
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
How close was the implementer to the originally intended form?
2. Adherence/
Fidelity
49
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
What changes were made during the implementation process?
3. Adaptation
50
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
What was the quality of delivery (affective quality)?
4. Quality of delivery
51
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
How interested and receptive were participants?
5. Participant responsiveness
52
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we measure implementation quality?
Who is being reached (how representative they are of the target audience)?
6. Reach
53
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Observation Implementer Self-Report
Administrative Documents
Participant Self-Reports
Where can we get information to measure implementation
quality?
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Different frameworks can be used to guide evaluation
55
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf
Evaluation steps/components to consider
• RE-AIM (Glasgow, R. E. et al., 1999)
• PRECEDE-PROCEED (Green, L. 1968)
Implementation factors to assess in your evaluation plan
• CFIR (Damschroder, et al., 2009)
• PARiHS (Harvey & McCormack,1998)
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
RE-AIM is a commonly used evaluation framework
56
Source: Introduction to program evaluation for public health programs: Centers for Disease Control and Prevention, 2011. http://www.cdc.gov/eval/guide/cdcevalmanual.pdf; Glasgow, 1999
• Number, proportion, or representativeness of individuals participating
Reach
• Impact of the intervention on outcomes Effectiveness
• Number, proportion, or representativeness of settings Adoption
• How well the intervention was implemented Implementation
• Sustainability of the individual changes, program delivery, and outcomes
Maintenance
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we evaluate Implementation?
57
Source: Durlak & DuPre (2008). AM Community Psychol; 41(3-4):327-50. doi: 10.1007/s10464-008-9165-0
• Dosage
• Adherence/ Fidelity
• Adaptation
• Quality of delivery
• Participant responsiveness
• Program reach
Measure how well the strategies were implemented (i.e., implementation quality)
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we evaluate Reach?
Measure the number, proportion, or representativeness of individuals
who were recipients of the ideal practice
• # of potential participants approached
• # of participants deemed ineligible to participate
• # of participants that actually participated
• % of all eligible invited participants who accepted participation
• Characteristics of participants compared with nonparticipants
58
Source: National Cancer Institute (2013). Research-Tested Intervention
Programs (RTIPs). https://rtips.cancer.gov/rtips/reAim.do
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we evaluate Effectiveness?
Measure impact of the ETP on outcomes
• Did the ETP make a difference?
• How much of a difference (effect size or the magnitude of change)?
• Did some program recipients benefit more than others (subgroup
analysis)?
59
Source: National Cancer Institute (2013). Research-Tested Intervention
Programs (RTIPs). https://rtips.cancer.gov/rtips/reAim.do
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we evaluate Adoption?
Measure number, proportion, or representativeness of settings/sites that decided to adopt the ETP
• # of settings in a given population qualified to host the ETP
• # of settings that were interested in participating
• # of settings that were not appropriate for the study
• # of settings that met criteria and chose to participate
• % of the total number of available settings that actually participate
• Characteristics of participating settings compared with nonparticipating settings
60
Source: National Cancer Institute (2013). Research-Tested Intervention
Programs (RTIPs). https://rtips.cancer.gov/rtips/reAim.do
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do we evaluate Maintenance?
61
Maintenance Sustainability…… Stay tuned!
Measure sustainability of the ETP (ideal practice, strategies and
outcomes)
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
When and what are you evaluating for your project?
What evaluation framework will you use?
62
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Questions or comments about evaluation?
63
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #3: Evaluate the process and outcome
64
1
Thinking about your project, describe
your plan for evaluation by thinking
through the questions in your Activity
Workbook:
1. Describe how you will evaluate the
process of implementation
2. Describe how will you evaluate the
outcomes of implementation
We will discuss responses as a larger
group. 2
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #2 discussion: Evaluate the process and outcome
• Did you have any challenges or “aha” moments when considering the
process and outcome evaluation?
65
5.0 How do you sustain the program?
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Learning Objectives
1. Define sustainability, scale-
up, and spread
2. Discuss frameworks and
resources for sustainability,
scale-up, and spread
67
Stuart Miles/freedigitalphotos.net
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What happens after you have successfully implemented?
68
Sustainability
Intermediary
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Why does sustainability matter?
69
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner. 70
Source: Graham ID et al. JCHEP 2006;26:13-24.
Where is sustainability on the KTA process model?
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Why plan for sustainability?
o Planning for sustainability
makes it more likely that you
will sustain the intervention
o Not sustaining the
implementation intervention
decreases the chances that
the intervention will be adopted
in the future
71
Source: Flagan & Flay (2009). Health Educ Behav;36(1):9-23. doi: 10.1177/1090198106291376; Johnson, Hays,
Center, & Daley (2004) Building capacity and sustainable prevention innovations: a sustainability planning model.
Evaluation and Program Planning; 27:135-49. doi: 10.1016/j.evalprogplan.2004.01.002; Bumbarger & Perkins
(2008). Journal of Children’s Services; 3:55–64
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Related terms
72
Davies B. In Knowledge Translation in Health Care, second
edition. Wiley, In Press
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What do we know about sustainability?
• Literature on sustainability
remains theoretical, with little
guidance on how to sustain
• 4 systematic reviews
o 11.5% (n=24) of the articles
provided a definition of
sustainability
o Few models of sustainability
have been rigorously tested
Source: Stirman et al. (2012). Implementation Science; 7:17. doi: 10.1186/1748-5908-7-17;
Gruen et al. (2008). Lancet; 372(9649):1579-89. doi: 10.1016/S0140-6736(08)61659-1;
Ament et al. (2015). BMJ Open; 5(12). doi: 10.1136/bmjopen-2015-00807; Tricco et al.,
(2016). Implementation Science; 11:55. doi: 10.1186/s13012-016-0421-7
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Defining sustainability
1. After a defined period of time
2. a program or implementation strategies continue to be delivered;
3. behavior change aligned with ideal practice is maintained;
4. the implementation strategies (2) and ideal practice (3) may evolve or adapt,
5. while continuing to produce benefits for individuals/systems.
74
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
1. After a defined period of time
• Refers to a specific period of
time after implementation
• Defined by implementers
• Time period depends on the
ideal practice or program and
the outcomes of interest
75
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
2. A program or implementation strategies continue to
be delivered
• An organization or community
continues to deliver
implementation strategies
76
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
3. Behavior change aligned with ideal practice is
maintained
• New ways of working become
the norm
• An individual continues to follow
the recommendations of the
evidence-based program,
guideline, or practice
77
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
4. Implementation strategies and/or ideal practice may
evolve or adapt
• The implementation strategies
may evolve in response to
changes inside or outside of the
host organization or community
• The evidence-based practice
may evolve in response to
changes in the host
organization, a community, or
new evidence.
78
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
5. While continuing to produce benefits for
individuals/systems
• The program or ideal practice
continues to:
o Show a positive impact on
health outcomes
o Reduce costs for
organization/community
o Display advantages over
previous practices or programs
o Display advantages over new
practices or programs
79
Source: Moore, Mascarenhas, Bain, & Straus (2017).
Implementation Science; 12(1):110.
doi: 10.1186/s13012-017-0637-1.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
1) Define what you are sustaining;
2) Identify barriers and facilitators to sustainability;
3) Select sustainability strategies; and
4) Develop a sustainability plan.
How do you plan for sustainability?
80
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
But remember - sustainability is dynamic
• Principles of dynamic sustainability
o Continuous improvement
o Measures progress
o Make sure it fits the context
o Engage stakeholders
o Intervention effects do not have to drop over time
Source: Chambers, Glasgow, & Stange (2013).
Implementation Science; 8:117. doi: 10.1186/1748-5908-8-117.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
National Health Service – Sustainability model
82
Source: Maher, Gustafson, & Evans (2010) NHS Sustainability Model and
Guide. The NHS Institute for Innovation and Improvement. England, United
Kingdom
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do you measure sustainability (maintenance)?
83
• Reach
o Continue to reach the intended target audience?
• Effectiveness
o Does the program continue to produce positive effects?
o Are these effects maintained over time?
• Adoption
o Are sites continuing to implement the program?
• Implementation
o Are sites continuing to implement with high quality?
Dosage
Adherence/Fidelity
Adaptation
Quality of delivery
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Spread vs. Scale up
Spread
Horizontal diffusion or actively disseminating best practice and knowledge and implementing each intervention in every available care setting Source: Institute for Healthcare Improvement (2008).
Scale up
Vertical diffusion or deliberate,
systematic approaches to
increasing the coverage, range,
and sustainability of services Source: Eaton et al. (2011). Lancet; 378(9802):1592-
603). doi: 10.1016/S0140-6736(11)60891-X
84
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is Spread?
• Diffusion: “The process by
which an innovation is
communicated through
certain channels over time
among the members of a
social system”
Source: Rogers (2003). Diffusion of
innovations. New York: Free Press.
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What are examples of strategies that can be used to
spread an ETP?
Develop educational materials
• Develop and format manuals, toolkits, and other supporting materials in ways that make it easier for stakeholders to learn about the ETP and how to implement the ETP
Through a dissemination organization
• Identify or start a separate organization that is responsible for disseminating the ETP. It could be a for-profit or non-profit organization
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
What is Scale up?
• Deliberate efforts to increase
the impact of innovations
successfully tested in pilot or
experimental projects so as
to benefit more people and
to foster policy and program
development on a lasting
basis.
87
Source: Simmons, Ruth, Fajans, Peters & Ghiron, Laura
( 2007) . Scaling up health service delivery: from pilot
innovations to policies and programmes / edited by Ruth
Simmons, Peter Fajans, and Laura Ghiron. Geneva : World
Health Organization.
http://www.who.int/iris/handle/10665/43794
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
How do you scale up?
88
Institute for Healthcare improvement (IHI) Framework for Going to Full
Scale Can be used to scale up health interventions.
Source: Barker, Reid, & Schall (2016). Implementation Science; 11(1):12.
doi: 10.1186/s13012-016-0374-x
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Do you have questions about sustainability, scale-up, or
spread?
89
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #4: Apply what you’ve learned to your project
90
1 Thinking about your own project, work
through the questions and fill in the
answers in the spaces provided in your
Activity Workbook.
We will discuss responses as a larger
group. 2
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Activity #4 discussion: Apply what you’ve learned to your
project
o How do you plan to embed what you’ve learned in the workshop to
your implementation science proposal?
o What are some of your anticipated challenges and how might these
be addressed?
91
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Day 2 review
Presentation 1.0: What do you
need to consider when planning
for implementation?
Presentation 2.0: Who is involved
in implementation?
Presentation 3.0: Where is
implementation happening
Presentation 4.0: How do you
evaluate implementation?
Presentation 5.0: How do you
plan for sustainability, scale-up,
and spread?
92
© Copyrighted by St. Michael’s Hospital 2017.
The materials are intended for non-commercial use only. No part of the materials
may be used for commercial purposes without the written permission of the
copyright owner.
Thank You!
93
