Hollingsworth.J.rogers.doing Institutional Analysis-Implications for the Study of Innovations
Innovations in Prevention & Implementation Science: Implications for Improvement in Schools
description
Transcript of Innovations in Prevention & Implementation Science: Implications for Improvement in Schools
1
Innovations in Prevention &
Implementation Science:
Implications for Improvement in
SchoolsC Hendricks Brown, Northwestern University Feinberg School of Medicine
Sheppard Kellam, Johns Hopkins UniversityJeanne Poduska, American Institutes for Research
Juan Villamar, Northwestern University Feinberg School of MedicineGracelyn Cruden, Northwestern University Feinberg School of Medicine
2
Acknowledgments: for Ce-PIM • National Institute on Drug Abuse: CENTER FOR
PREVENTION IMPLEMENTATION METHODOLOGY FOR DRUG ABUSE AND SEXUAL RISK BEHAVIOR (PI Brown P30 DA027828)
• Substance Abuse and Mental Health Services Administration
SAMHSA Ce-PIM Partnership
3
Acknowledgements: Work to DateFunding Agencies:• IES•NIDA•NIMH•NICHD
4
Process in this Workshop: Active
Engagement
5
1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems
2. Public Education and Public Health: Toward a System of Systems
3. Systems Thinking and Partnerships in Daily Life4. Illustrative Innovations of Ce-PIM5. Summary: Building an Integrated Structure to
Support Children and Youth Reaching their Full Potential
Workshop Outline
6
1. A Challenging Tour of Implementation Science and its Home in Service Delivery Systems The Perspective at Ce-PIMA. Implementation as an action strategy working on systems changeB. Why Study Implementation?C. Approaches to Moving Effective Interventions into PracticeD. Three Interacting and Evolving Components of ImplementationE. A Systems Approach involving 3 interacting components Small group exercise
7
Ce-PIM Mission: Building scientific and methodologic rigor into implementation research and practice.AIMS: 1. Develop Systems Methods (Systems Science, Engineering,
Computational) Methods for Implementation Science2. Infuse systems methods into the field of Implementation
Research3. integrating methods within the practice of prevention
implementation at the federal, state, county, and local levels.
8
A. What is Implementation Research?According to NIH (2008):The use of strategies to adopt and integrate evidence-based health interventions and change practice patterns within and across specific systems
Action OrientedWithin Settings or SystemsAND collects data
Chambers DA. Advancing the science of implementation: A workshop summary. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1-2):3-10.
9
B. Why Study Implementation?1. We know a lot about what works 10K reviewed studies in What Works Clearinghouse
2. We are short on implementation action strategies to put what works into practice: WWC Practice Guides:
Reducing Behavior Problems in the Elementary School Classroom 2008
3. It takes too long for research to affect practice
10
Closing the Gap between What We “Know” Works
and What We Do.“17 Year Gap” in Health Care
11
17 Years in Implementing an Statistical MethodologyGeneralized Estimating Equations (GEE: Brown et al., APMH 2012)
12
•Is the gap between research and practice similar in education to that existing in health?Types of Gaps?As long as? As important to shorten? Which way?As resistant to change?
13
C. Approaches to Moving Effective Programs into Practice
Making a Program
Work
Does a Program Work?
Could a Program Work?
Traditional Research Pipeline for
Implementation
14
IOM 2009
Landsverk, Brown et al. 2012
Aarons et al., 2011
Implementation
Exploration
Adoption / Preparation
Implementation
Sustainment
Effectiveness Studies
EfficacyStudies
Preintervention
Traditional Translation Pipeline
Rea
l Wor
ld R
elev
ance
Local knowledge
Generalized knowledge
Intervention
15
A Focus on the Intervention itself is Not Sufficient•The use of effective interventions without implementation strategies is like serum without a syringe; the cure is available, but the delivery system is not
Fixsen, Blase, Duda, Naoom, Van Dyke,2010.
16
D. Three Interacting and Evolving Components of Implementation (Chambers et al., 2013 Imp Sci)• Intervention: Program, Practice, Policy, Principles• Practice Setting: Delivery Support System• Ecological System: Population and Community/Cultural Context
17
Key Parameters on Implementation ScienceA. Transitions of Research Questions:
Effectiveness – Can a Program Work in real life systems? Implementation – Making a Program Work
B. 4 Stages: Exploratory, Adoption, Implement w/ Fidelity, SustainabilityC. Focus on Interactions of Program, Delivery System, and EcologyD. Science:
Produce Generalizable Knowledge rather than Local KnowledgeMeasurement, Modeling, and Testing Using Rigorous Methods
E. Partnerships
18
Definition of System
A set of connected parts forming a complex whole
19
Group Exercise: Produce a System Map of the Educational System Three Steps:1. Who needs to be at the table? What are the components
needed represented?2. How do they Interact with one another?3. Where would you intervene to affect intervention usage?
Education Policy, i.e. Common Core Standards
Identify one person from your small group to bring this back to the group
20
Follow-up Question on Systems:Would your list of players and location intervention differ based on implementing
New policyReplacement of an existing curriculumRemoval of a expulsion policy
21
Summary of Part 1• Multi-goal, Interacting, Dynamic and Multilevel
System
• System View Looks at Interactions between components.
• Behavior of a system is not linear: complex.
• A useful way to understand a system is to intervene and study its behavior.
2. Public Education and Public Health: Toward a
System of Systems
Sheppard G. Kellam, MDCenter for Prevention Implementation Methodology
Johns Hopkins Bloomberg School of Public Health
Society for Research on Educational EffectivenessWashington, DC, September 4-5, 2014
23
The Baltimore Education and Prevention Partnership
The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials.
Trials were directed at helping children master obeying rules of behaving, attending, academic learning, socializing appropriately in 1st grade classroom.
Interventions were tested separately in 1st generation (our focus today), then together in later trials.
24
Two Dimensional View of Mental Health
•Social Adaptational Status—level of success or failure in main social fields at each stage of life
•Psychological/physical Well-Being—includes affect, cognitive development, i.e., internal condition of the individual
25Kellam/Langevin, 2003
DEVELOPMENTALEPIDEMIOLOGY:
directed at early proximal targets
MORE IMMEDIATE RISK: directed at
more recent proximal targets
COMMUNITY PREVENTION:
directed at Community & School proximal targets
COMMUNITY / SOCIETAL: directed at
Policies & Laws as proximal targets
Prevention Research and Service Strategies
26
Universal
Selective
IndicatedRxMed, MH,Soc Welfare
Levels of Prevention
and TreatmentEntire
Population
27
Current Problems in System Coordination
•Prevention and treatment not integrated•Service Agencies are Silos—minimal collaboration
•No shared information system monitoring developmental progress
•Mystery as to who gets “referred” for services•Little follow-up of “no-shows”
28
Early Risk in Prevention Research Over the last four decades much has been learned about early risk factors and paths leading to drug abuse, and other behavioral, mental health, and school problems.
Most if not all are strongly related to school failure, also a major risk factor for later drug abuse, alcohol, tobacco, depression, anti-social and other problem outcomes.
Aggressive, disruptive behavior as early as 1st grade has been repeatedly found a risk factor for later drug and alcohol abuse and disorders, delinquency, violence, tobacco use, high risk sex, school failure and other high risk behaviors.
29
When a child misbehaves…
30
The Baltimore Education and Prevention Partnership
• The Baltimore City Public School System (BCPSS) has collaborated in 3 generations of education and prevention field trials.
• They were directed at helping children master key social task demands in 1st grade classroom.
• Interventions were tested separately, then together.
• The 1st generation will be our main focus here, where the Good Behavior Game (GBG) was tested by itself and the children were followed up at ages 19-21 and beyond.
31
High Risk Children in Well vs. Poorly-Managed Classrooms (control classrooms only)
• If the top 25% of children on aggressive behavior were in disrupted classrooms, their risk of severe aggressive behavior by middle school was up to 59 times the average child’s.
• If similar children were in well-managed classrooms, the risk was up to 2.7 times the average child’s.
32Kellam et al., Addiction Science & Clinical Practice; 6:73-84, 2011
33
The Role of School Information Systems in our New System of Systems• Following individual children over time, schools and
communities is required for assessing their progress, needs and for integrating services across agencies
• Information systems tracking each child exist in most school districts and can be extended and used to integrate services and monitor progress
• Confidentiality is by law guaranteed by school authorities and can be expanded to include data as needed for services
34
Partnerships for Building and Overseeing the New System of Systems •Analyze political constituencies•Engage and work through trust with each•Learn their mission and vision• Identify mutual self-interests and•Encourage their coming together as oversight committee
35
Whole Group Activity: Partnerships•What do we mean by “who” needs to be included?
•The nature of the process of engaging and developing the base of support/collaboration.
36
3. Systems Thinking and Partnerships in
Daily Life Jeanne Poduska, Sc.D.
American Institutes for Research
Society for Research on Educational EffectivenessWashington, DC, September 4-5, 2014
37
Systems Concepts • Introduction of programs and action strategies requires adaptation to systems, e.g. staffing, fiscal allocations
•We operate in complex systems: multi-level, multi-actor
•Change occurs locally (even if simultaneously)
•Systems thinking focuses on interactions
Working Definition of Scaling-Out
“Scaling out is the deliberate use of strategies to implement and sustain evidence-based interventions through or across settings to promote the greatest public health impact”.
Ce-PIM Workgroup on Scaling Out, unpublished, 2014
38
Scaling Out Perspective on Implementation Research
39Scale Out Across Diverse ContextsLocal knowledge
Generalized knowledge
Scale Up
ExploreAdopt/
Prepa
re
Implem
entSusta
in
Baltimore
Houston
NE urban
NE rural
40
AIR’S mission is to conduct and apply the best behavioral social science research and evaluation towards improving peoples’ lives, with a special emphasis on the disadvantage.
41Chambers et al, 2013 Implementation Science
Prevention Science into….
Johns Hopkins: 1983-present
RCT 1: GBG | MLRCT 2: GBG + curricula | family
AIR: 2003-2007RCT 3: GBG + reading + family Develop model of training and support for GBG
Baltimore ’83-’07
Case Study: Good Behavior Game at AIR
Implementation Science
RCT 1: -GBG: 2 models of PD-Implement/Sustain-”Adaptation”
Service contract to develop local GBG capacity: Face-to-face training and support
Houston ‘08- present
Implementation Science
Service contract: Face-to-face training and support
Under review: GBG for Middle School
Nebraska urban ’11- present
Implementation Science
Development grant: Develop and pilot web-based tools to deliver training to teachers
Under review: Develop and pilot web-based tools for coaching and on-going support to teachers
Nebraska rural ’13- present
JHSPH American Institutes for Research
Domitrovich et al., 2008; Poduska et al., 2009; Poduska et al., 2012; Poduska & Kurki, 2014
Scaling Out Perspective on Implementation Research
43Scale Out Across Diverse ContextsLocal knowledge
Generalized knowledge
Scale Up
ExploreAdopt/
Prepa
re
Implem
entSusta
in
Baltimore
Houston
NE urban
NE rural
Who needs to be at table at AIR?
Leadership:CEO/President
Board of DirectorsDir, Ed Program
LegalContracts
Human Resources
Information Technology
(IT)
Financial
Institutional Review Board
(IRB)
Publication Services
Communications
Web Services
??
??
Colleagues:Researchers
Practitioners
Data Use
Professional Development
Adolescence
Ed Tech
Early Childhood
Middle School
Social Emotional Learning
Mental Health ServicesQuantitative Analysts
Qualitative Analysts
45
Additional Systems Concepts
•Systems have developmental trajectories •Systems are adaptive, fluid •Adaptation occurs locally •A note about silos: people may know one another, talk with one another and not be working towards a common goal
46
4. Innovations: Activities of Ce-PIM
Juan Villamar, MS.EdGracelyn Cruden, M.A.
C Hendricks Brown, Ph.D.
Center for Prevention Implementation MethodologyDepartment of Psychiatry and Behavioral Sciences
Northwestern University
47
5. Improve efficiency and effectiveness of implementation delivery
Implementation Agency or Agencies
Target
Developer/ Purveyor
Broker, Intermediary
1. Person, Place, and Time, Reach, Engagement
2. Delivery medium
3. Simplify and support intervention delivery
4. Training, fidelity monitoring and feedback
6. Intervention R&D
7. Monitoring Large-Scale Implementation
Funder / Oversight
8. Modeling and instituting implementation and oversight policies
General Model to Support Implementation of Evidence Based InterventionBrown et al, 2013 JAIDS
48
Using a mobile application to improve fidelity of the GBG – Rapid Development of a Prototype
49
Visualization Tool – Classroom Disruptiveness Comparison Model
2 classrooms, 24 students
Chance that students will act disruptively and their disruptiveness will spread.
50
Major Gap – Sustainable Partnerships •Partnerships between content experts and technology developers
“What do you recommend we use?” VS “Tell me what you want to see in a report”
•Bring to the table those players that will interact with the report FIRST
•Partnerships between Research and PracticeIterative testing with feedback loops
•Technology System, Educational System, Public Health SystemIntegrated System?
Understanding the System: Policy• Problem: Dissemination of Evidence-Based Interventions• Who can pay, sustainably USPSTF• How to influence this powerful entity?
• Understand how they came to power, to whom they report, and who is required to listen
• Review their decision making process• Concrete challenge: # of prevention programs in parenting that are
linked to primary care nearly 0 (despite IOM 2009 report)
• Is primary care even the best environment? What about schools, who have a larger denominator?
52
Strategic Engagement• Identify areas for involvement
• Short term – • engage with AHRQ and USPSTF; identify brokers
• Long term – • Scaling out research designs (primary care)• Research design for increased knowledge utilization (ex: harmonized
measures)• Publications in strategic journals
• Is there a similar influential agent in Education? How would you begin to engage in the short and long term?
53
Innovation: Opening integration between primary care and research
AHRQ
USPSTF
NIH
Schools- Nurses- School Wide
Programs- Information Systems
Primary Care Clinics
CMMI grants ODP
54
Implementation Science can be conducted with high rigor through:•Measurement•Modeling•Testing
CAL-OH Head-to-Head Randomized Implementation Trial• Single evidence-based intervention:
• Multidimensional Treatment Foster Care (MTFC)
• 2 alternative strategies of implementing this same programStandard implementation (Stnd)
Community Development Team (CDT)
• Randomize 51 Counties to implementation strategy*• Evaluate implementation success using Stages of Implementation Completion
(Saldana IS 2014)• Faster implementation• more (# families served), • better (fidelity)
Chamberlain P, et al. (2008). Engaging and Recruiting Counties in an Experiment on Implementing Evidence–Based Practice in California. Administration and Policy in Mental Health and Mental Health Services Research, 35(4): 250-260.
55
Existing Implementation
Supports for County, Agency,
Group Home
MTFC Intervention
MTFC Implementation
Supports for County, Agency, Clinicians,
Parent
Two-Arm TrialsEffectiveness vs.
Implementation
56
MTFC Intervention MTFC
Intervention
CDTImplementation
Supports for County
Control Condition
Standard Implementation
Supports for County
Youth Youth
Youth Youth
57
Randomize 51 Counties in CA and OH to Implementation Strategy and Time (Cohort)
Randomized Roll-Out Design*
40 CA Counties
26 Wait LIsted
CDT
Stnd
Wait Listed
13 Wait LIsted
COHORT 1 COHORT 2 COHORT 3 COHORT 4
*Brown, et al. 2009 Ann Rev PH
11 OH Counties
Summary of Findings (Brown et al., under review)• Mixed Results
• No evidence that • CDT affected rate of adoption• CDT changed speed of implementation
• Evidence that • CDT increased numbers
of families served• CDT counties completed
implementation more thoroughly58
0 5 10 15 20 25 30
05
10
15
20
25
30
Figure 4. Comparison of PlacementQuantiles for CDT and IND Counties
CDT
IND
Number Served Quantiles for CDT versus IND (EQQ Plot)
59
5. Building an Integrated Structure to Support
Children and Youth Reach their FULL Potential
Opportunities, Strengths, Next Steps and Closing
60
1. Implementation Science involves a systematic study of actions within and across systems
2. Requires partnerships that need to built on mutual self interest 3. Systematic way to build such partnerships that coalesce the
interests of diverse systems, including public education and public health
4. Implementation lives inside the organizations and communities that are responsible for delivering or overseeing services.
5. There are rigorous methodologic approaches to Implementation Science that address different stages of implementation with well define measures, modeling approaches and research designs.
61
Contact Information:• C Hendricks Brown,
[email protected] • Sheppard G Kellam, [email protected] • Jeanne Poduska, [email protected] • Juan Villamar, [email protected]• Gracelyn Cruden, [email protected]
Slides and References are Available:http://cepim.northwestern.edu/products/presentations-and-workshops/