Dean Fixsen

Dean L. Fixsen, Karen A. Blase,

Leah Bartley, Michelle Duda,

Sandra Naoom, Allison Metz,Barbara Sims, Melissa Van Dyke

National Implementation Research Network

Frank Porter Graham Child Development Institute

University of North Carolina at Chapel Hill

Blueprints 2012

Evidence-basedprograms: A failedexperiment or the futureof human services?

Copyright © Dean L. Fixsen and Karen A. Blase, 2012

During the Golden Age ofresearch in human services,the field has been dominatedby the randomized, controlledexperimental paradigm

The Golden Age


A key lesson from the GoldenAge is that the effects ofsocial programs in practicehover near zero, adevastating discovery forsocial reformers

The Golden Age


A consequence of these findingsis the recognition of theimportance of implementationresearch in overall evaluations

The Golden Age


Rossi, P. H., & Wright, J. D.(1984). Evaluation Research: AnAssessment. Annual Review ofSociology, 10, 331-352.

Summarized the Golden Age thatbegan with Kennedy in 1962,flourished during Johnson’sGreat Society programs, andended with Reagan in 1982

The Golden Age


0

100

200

300

400

500

600

700

800

900

≤1972 1974 1976 1978 1980 1982

0

50

100

150

200

250

300

CumulativeCouples

CumulativeHomes

Fixsen, Blase, Timbers, & Wolf (2001)

Teaching–Family Replications


Follow Through Programs

Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, andmath computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure isadapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press.


The “evidence-based movement” is aninternational experiment to makebetter use of research findings intypical service settings.

The purpose is to produce greaterbenefits to children, families,individuals, and society.

The New Golden Age


The evidence-based programmovement

Evidence-based

Program

Movement

The New Golden Age


What defines “evidence”Two or more high quality researchstudies using randomized groupdesigns (within subject designs)

Preferably done by two or moreindependent research groups

Preferably summarized in meta-analyses of findings across studies

Evidence-based


For the past decade theNational Institutes of Healthhave spent over $100 billion ayear on research to developevidence-based programs

Other federal agencies (e.g. IES; USAID)and philanthropies add to this total eachyear

Evidence-based


In 2008 NIRN documented 32websites and review articles

Assess the quality of “evidence” byexamining research methods

Name some programs as “evidence-based”and others as promising or other

N = 700 deemed to be “evidence-basedprograms” based on external reviews

Blueprints website N = 11 ModelPrograms (out of 900 reviewed)

Evidence-based



Evidence-based

ProgramMovement

The New Golden Age


What is a “program?”Clear description of the program

Philosophy, values, principlesInclusion – exclusion criteria

Clear essential functions that definethe program & linked to outcomesOperational definitions of essentialcomponents (do and say)Practical performance assessment

Highly correlated (0.70+) with outcomes

Programs


About 18% of outcome studies(N=1,200+) assessed theindependent variable

About 7% linked essentialcomponents to outcomes

Few studies measure fidelityFewer yet link fidelity tooutcomes

Programs

Dane & Schneider, 1998; Durlak & DuPre, 2008


359 outcome studies in 8journals

32% used a treatment manual

22% supervised treatment agents

18% measured protocol adherence

6% did all three

55% did none of the above

Moncher & Prinz (1991)

Programs


Know a lot about Scientificrigor

Rigor is not used bypractitioners to impact the livesof people

Know little about ProgramsPrograms are used bypractitioners to impact the livesof people

Programs



Evidence-based

Program

Movement

The New Golden Age


Letting it happenRecipients are accountable

Helping it happenRecipients are accountable

DO IT YOURSELF APPROACHES TOMOVING SCIENCE TO SERVICE

Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate,& Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke(2010)

Movement


Prevention programs in 5,847schools; 2004-2005 school year

Avg. 9 innovations per school

7.8% were evidence-based

3.5% used with fidelity

US Department of Education, 2011

Movement


Evidence-base Actual SupportsYears 1-3

OutcomesYears 4-5

Every TeacherTrained

Fewer than 50% ofthe teachersreceived sometraining

Fewer than 10% ofthe schools used theCSR as intended

Every TeacherContinuallySupported

Fewer than 25% ofthose teachersreceived support

Vast majority ofstudents didnot benefit

Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006

Longitudinal Studies of a Variety of Comprehensive School Reforms

Movement


$500 million invested in “FamilySupport Services” 1993-1998No implementation supports beyondwhatever TA was offered by statesNo fidelity criteria insisted upon by thedevelopers (e.g. Kinney, Haapala, Booth)National evaluation = not effective

Over 25% was spent on in-office interventionswith parents or children (< 0 fidelity)An implementation failure labeledHomebuilders as an intervention failure

Past Federal Funding


Current Federal Funding$100 billion for innovativeprograms (USDE)

$63 billion for maternal healthprograms (USAID)

$4 billion for homevisitingprograms (ACF)

Little/ no funding forimplementation supports forthese program initiatives


The “evidence-based movement” is aninternational experiment to makebetter use of research findings intypical service settings.

The purpose is to produce greaterbenefits to children, families,individuals, and society.

The New Golden Age

National Assessment of Education Progress (NAEP)

40 Years ofVariation Around a Mediocre Mean


Best Data Show These Methods, WhenUsed Alone, Are Insufficient:Diffusion/ Dissemination of information

Training

Passing laws/ mandates/ regulations

Providing funding/ incentives

Organization change/ reorganization

About 5% to 20% Realize Intended Benefits

Implementation Science


Focusing on methodologicalrigor to move science toservice is not sufficient

Relying on passive/unplannedmeans to move science toservice is not sufficient

Movement


SCIENCE SERVICEGAP

Science “to” Service

The New Golden Age


Complex ProblemsHuman services involveinteraction-based sciencesand services

Inherently more complexthan atom-based sciences

e.g., atom-based ingredientsdon’t talk back or run away


PractitionersIn human services, thePRACTITIONER IS THEINTERVENTION

Everyone / everything else needs tobe aligned to provide effectivesupports so all practitioners canproduce desired outcomes for allrecipients of services


SCIENCE SERVICEGAP

DiffusionDissemination

Translation

IMPLEMENTATION

Implementation

Science “in” Service


“In theory there is no differencebetween theory and practice;

in practice, there is.”

Albert Einstein

Implementation


The ChallengeScience to Service Gap

What is known is not what is usedto help children, families,individuals, and communities

Implementation Gap What is adopted is not used with fidelity

and good outcomes for consumers.

What is used with fidelity is not sustainedfor a useful period of time.

What is used with fidelity is not used on ascale sufficient to impact social problems.


Implementation“Implementation has never been a

national goal per se, but goalsthat can be reached only byeffectively implementing newtechnology have been inherent inmany national programs.”

(Hough, 1975)



Review and synthesis of theimplementation evaluationliterature (1970 – 2004)

Multi-disciplinary

Multi-sector

Multi-national


Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).Implementation Research: A Synthesis of the Literature. Tampa, FL: University ofSouth Florida, Louis de la Parte Florida Mental Health Institute, The NationalImplementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/

ImplementationResearch:A Synthesis ofthe Literature



Formula for Success

Effective intervention

X

Effective implementation

=

Effective outcomes

0.99

.009

0.01

X

Brown & Flynn, 2002Clancy, 2006


An intervention is one thing

Implementation is somethingelse altogether

Like serum and a syringeVery different evidence basesEach is necessaryNeither one is useful without theother



ACTIVE ImplementationFrameworks and Best Practices



Implementation ScienceLetting it happen and Helping it happen

MOVE SCIENCE TO SERVICEWITH EXPERT HELP

Making it happenPurposeful use of implementation

practices and scienceImplementation teams are accountable

Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate,& Kyriakidou (2004); Fixsen, Blase, Duda, Naoom, & Van Dyke(2010)


Implementation TeamMinimum of three people (four or

five preferred) with the expertise topromote effective, efficient, andsustainable implementation,organization change, and systemtransformation work

Tolerate turnover; teams aresustainable even when the playerscome and go (Higgins, Weiner, &Young, 2012; Klest & Patras, 2011)


Implementation Team

Organization SupportsManagement (leadership, policy)

Administration (HR, structure)

Supervision (nature, content)

Practitioner/Staff Competence

State and Community Supports Regional Authority Supports

Impl

emen

tatio

n Te

amSimultaneous, Multi-LevelInterventions


Implementation Team

ImplementationTeam

PrepareOrganizations

PreparePractitioners andStaff

Work withResearchers

Assure Implementation

Prepare Regions AssureIntendedBenefits

Create Readiness

Parents andStakeholders

© Fixsen & Blase, 2009

20%80%


All Individuals &Families

Practitionersand Staff

Innovation outcomes result from adultinteractions with children, families, & individuals

Capacity Development:Establishing & sustaining aninfrastructure for implementationto assure full and effective usesof effective practices in allagencies

Innovations

AgencyImplementation

Teams

RegionalImplementation

Teams

StateImplementation

Team

StateDepartmentLeadership

Impl

emen

tatio

n O

utco

mes


Impl. Team NO Impl. Team

Effective

Effective use ofImplementationScience & Practice

IMPLEMENTATION

INTE

RVE

NTI

ON 80%, 3 Yrs 14%, 17 Yrs

Balas & Boren, 2000Fixsen, Blase,Timbers, & Wolf, 2001

Implementation Team

Letting it HappenHelping it Happen

Substantial Return on Investment


Costs and SavingsImplementation Costs & Savings

(Inflation Adjusted)

60

70

80

90

100

110

120

1 Yr Pre During Post

Year 1

Post

Year 2

Post

Year 3

Ch

an

ge

in

Bu

dg

et

(Pe

rce

nt)

Invest inImplementation

Capacity

ImproveEffectivenessand Efficiency


Making it Happen

Implementation DriversCommon features of successfulsupports to help make full andeffective uses of a wide varietyof innovations


Performance Assessment(fidelity)

Coaching

Training

Selection

Integrated &Compensatory

Com

pete

ncy

Drive

rs

Systems Intervention

Facilitative Administration

Decision Support Data System

Organization Drivers

AdaptiveTechnical

Leadership Drivers

Implem

entat

ion

Drivers

Consistent uses ofInnovations Interventions

meetImplementation

Reliable Benefits

Training

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

N = 7 N = 6 N = 3 N = 7 N = 8 N = 9

A B C D E F

Workshops

Perc

en

t o

f T

I C

om

po

nen

ts

Dem

on

str

ate

dPRE

POST

Collins, S. R.,Brooks, L.E.,Daly, D.L.,Fixsen, D.L.,Maloney, D.M., &Blase, K. A.(1976)

Staff Training

OUTCOMES(% of Participants who Demonstrate Knowledge, Demonstrate

new Skills in a Training Setting,and Use new Skills in the Classroom)

TRAININGCOMPONENTS

Knowledge SkillDemonstration

Use in theClassroom

Theory andDiscussion 10% 5% 0%

..+Demonstrationin Training

30% 20% 0%

…+ Practice &Feedback inTraining

60% 60% 5%

…+ Coaching inClassroom

95% 95% 95%

Joyce and Showers, 2002

Training, Coaching, Performance


Teaching-Family Model

0%

10%

20%

30%

40%

50%

Homes

1.0

1.2

1.5

1.7

1.92.0

1.1

1.4

1.6

1.8

1.3

Delinquency (Outcome)

Teaching (Fidelity)

rs = – .94

Perc

ent

Pare

ntal

-Tea

chin

g W

ith Y

outh

s

Mean Self R

eportedD

elinquency Ratings

1 2 3 4 5 6

Bedlington, et al. (1988)

0 10 20 30 40 50 60

1234567

Mean89101112

Mean131415161718

Mean19202122232425

Mean

CONTROL

% Recidivism

Highly Competent& CompetentTherapists

N=12; 204 Families

13% Recidivism

Borderline &Not CompetentTherapists

N=13; 223 Families

28% Recidivism

Abo

ve F

idel

ityC

riter

iaB

elow

Fid

elity

Crit

eria

Functional Family Therapists (WSIPP)

Control Group 22% Recidivism


N = 100 county MH Clinics

Best of the MH Clinics (10%)Sustain EBPs for 50 months

Rest of the MH Clinics (90%)Sustain EBPs for 24 months

Glisson et al., 2008

Organization Supports

Organization supports matter!


0102030405060708090

100

Tx Unit Strategy Org. Development Strategy

% H

omes

Ope

n 6+

Yrs

.

N = 84 N = 219

Fixsen, Blase, Timbers, & Wolf (2001)

Organization Supports


Stages of Implementation

Exploration

Installation

Initial Implementation

Full Implementation

Implementation occurs in stages:

InterventionOutcomes

0% 100%

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

Type III ErrorDobson & Cook, 1980


Data fromDevelopers & Implementers

Assertive CommunityTreatment

Collaborative forAcademic, Social, andEmotional Learning

Dialectical BehavioralTherapy

Functional FamilyTherapy

Incredible Years

Life SkillsMultidimensionalTreatment Foster Care

Multisystemic TreatmentNurse-Family PartnershipNutrition Model Programfor ElderlyPositive Behavior SupportSchool-based mentalhealthSupported EmploymentFederation of Families forChildren’s Mental HealthNational Alliance ofMulticultural BehavioralHealth Associations

Blase et al., 2005

Impl Team Activities

ImplementationStages

Explore Install Init Impl

Assessment 97% 1% 2%

Planning 20% 32% 48%

Selection/Training 3% 31% 66%

Coaching 8% 6% 86%

Evaluation 3% 23% 73%

Org Development 11% 16% 73%

System Intervention 37% 30% 33%

N = 579 items(Concept Mapping; NominalGroup Process; Interviews)

Outcomes of EBPs

Effective InterventionPractices

Practitioners

Staff

Administrators

Boards

Implementation Teams

ImplementationProcesses

ImplementationPerformance Assessments

(fidelity)

InterventionPerformance Assessments

(fidelity)

ExplorationInstallationInitial ImplementationFull Implementation

WHAT

HOW

WHO

Systems trump programs!…Patrick McCarthy, Annie E. Casey

"All organizations are designed,intentionally or unwittingly, to achieveprecisely the results they get.”

…R. Spencer Darling

Systems


System Supports

Innovative practices do not farewell in existing organizationalstructures and systems

Organizational and systemchanges are essential tosuccessful use of innovations

Expect it

Plan for it


The Challenge

Systems are fragmented and arecharacterized by highly variable,often ineffective, and sometimesharmful services to consumers

Dean Fixsen and Karen BlaseUniversity of North Carolina at Chapel Hill

Rob Horner and George SugaiUniversity of Oregon; University of Connecticut

Barbara Sims and Michelle DudaUniversity of North Carolina at Chapel Hill


System SupportsSupply side: Effective programs gowhere they are most welcome

Let it happen; Help it happen

Islands of excellence

Demand side: Effective programsgo where they are most needed

Make it happen

A sea of change


Existing System

Effective InnovationsAre Changed to

Fit The System

Existing System IsChanged To SupportThe Effectiveness Of

The Innovation

Effective Innovation

System Supports

Compliance and Crises, Urgent, TimeSensitive!!• Services not meeting Standards• Deal with urgent and high profile issues

Best PracticesImplemented FullyWith Good OutcomesDisturb the System

System Supports & Stability• Regulatory roles• Basic Data Systems• Financing and FiscalAccountability• Accreditation/ LicensingStandards• HR rules and regulations• Safety Standards• Work with Legislature• Inclusion of Stakeholders

System Supports & Stability

Mandates, System Supports,

Foundational Polices & Regulations

Leadership Responsibilities and Leverage PointsLeadership Responsibilities and Leverage PointsThanks to Tom Bellamy


Implementation Team

StateManagement

Team

PractitionersInnovations

Children, Families

Syst

emC

hang

e

Adaptive Challenges• Duplication• Fragmentation• Hiring criteria• Salaries• Credentialing• Licensing• Time/ scheduling• Union contracts• RFP methods• Federal/ State laws

“Ext

erna

l” S

yste

m C

hang

e Su

ppor

t

Prac

tice

Info

rmed

Polic

yPolicy Enabled

Practice

System Reinvention


Capacity Building

YEARS

AM

OU

NTS

Funding

Capac

ity

Implementation TeamsOrganization ChangeSystem Reinvention


Children, families, andindividuals cannot benefit fromservices they do notexperienceFor the EBP Movement to besuccessful, we must implementevidence-based programs andsustain/ improve their benefitson a socially significant scale

Challenges

www.implementationconference.org

2013


For More InformationDean L. Fixsen, Ph.D.

[email protected]

Karen A. Blase, [email protected]

Frank Porter Graham Child Development Institute

University of North Carolina

Chapel Hill, NC

http://nirn.fpg.unc.edu/www.scalingup.org


HTTP://NIRN.FPG.UNC.EDU

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).Implementation Research: A Synthesis of the Literature. Tampa, FL: University ofSouth Florida, Louis de la Parte Florida Mental Health Institute, The NationalImplementation Research Network (FMHI Publication #231).

ImplementationResearch:A Synthesis ofthe Literature


StayConnected!

www.scalingup.org

SISEP @SISEPcenter

For more on Implementation Sciencehttp://nirn.fpg.unc.edu



Thank You for your SupportAnnie E. Casey Foundation(EBPs and culturalcompetence)William T. GrantFoundation(implementation literaturereview)Substance Abuse andMental Health ServicesAdministration(implementation strategiesgrants; nationalimplementation awards)Centers for Disease Control& Prevention(implementation research)

National Institute of MentalHealth (research and traininggrants)Juvenile Justice andDelinquency Prevention(program development andevaluation grantsOffice of Special EducationPrograms (Scaling upCapacity DevelopmentCenter)Administration for Childrenand Families (Child WelfareLeadership; CapacityDevelopment)Duke Endowment (ChildWelfare Reform)

Dean Fixsen

Documents

Transcript of Dean Fixsen