19th Annual RTC Conference
Presented in Tampa, February 2006
1
CASRC Multiple Stakeholder PerspectivesMultiple Stakeholder Perspectives
on Evidence-Based Practiceon Evidence-Based Practice
ImplementationImplementation
Gregory A. Aarons, Ph.D.Gregory A. Aarons, Ph.D.1,2,31,2,3
Karen Karen ZagurskyZagursky, B.A. , B.A. 1,31,3
Larry Palinkas, Ph.D. Larry Palinkas, Ph.D. 1,41,4
11Child and Adolescent Services Research Center (CASRC)Child and Adolescent Services Research Center (CASRC)11ChildrenChildren’’s Hospital San Diego, CAs Hospital San Diego, CA
22San Diego State UniversitySan Diego State University33University of California, San DiegoUniversity of California, San Diego
44University of Southern CaliforniaUniversity of Southern California1,1,
3020 Children’s Way, MC 5033
San Diego, CA 92123
(858) 966-7703 ext. 3550
http://casrc.org
CASRC
AcknowledgementsAcknowledgements
NIMH NIMH –– R03 R03 MH070703MH070703 (Aarons) Concept (Aarons) ConceptMapping of Readiness for Evidence-BasedMapping of Readiness for Evidence-BasedPracticePractice
NIMH - R01 MH072961 (Aarons) Mixed-NIMH - R01 MH072961 (Aarons) Mixed-Methods Study of a Statewide EBPMethods Study of a Statewide EBPImplementationImplementation
NIMH P30 MH068579 (Proctor) Pilot Study:NIMH P30 MH068579 (Proctor) Pilot Study:Aarons PI Organizational Receptivity toAarons PI Organizational Receptivity toEvidence-Based PracticeEvidence-Based Practice
CASRC
AgendaAgenda
The need for effective implementationThe need for effective implementation
Barriers and Facilitators to Implementing EBPBarriers and Facilitators to Implementing EBP
Study methodsStudy methods
ResultsResults
What does it all mean?What does it all mean?
CASRC
EBP Implementation is HappeningEBP Implementation is Happening
Effective implementation of EBPs into real-world serviceEffective implementation of EBPs into real-world servicesettings is important for improving service quality andsettings is important for improving service quality andoutcomes for youth outcomes for youth (Hoagwood & Olin, 2002; Jensen, 2003)(Hoagwood & Olin, 2002; Jensen, 2003)
Some (but not many) implementation improvementSome (but not many) implementation improvementmethods are being tried methods are being tried (Haynes & Haines, 1998)(Haynes & Haines, 1998)
–– Abstracting servicesAbstracting services
–– Evidence-based clinical guidelinesEvidence-based clinical guidelines
–– Incentives for better care systemsIncentives for better care systems
–– Increasing effectiveness of quality improvement programsIncreasing effectiveness of quality improvement programs
Research is testing some factors associated withResearch is testing some factors associated withimplementation but multiple stakeholder perspectives areimplementation but multiple stakeholder perspectives arenot well defined not well defined (NIMH R01, R03, PI: Aarons; R01 Webster Stratton,(NIMH R01, R03, PI: Aarons; R01 Webster Stratton,R01 Chaffin, R34 Shipp, )R01 Chaffin, R34 Shipp, )
CASRC We are Learning about ImplementationWe are Learning about Implementation
Some barriers to implementation have beenSome barriers to implementation have beenidentifiedidentified
–– e.g., lack of funds for continuing education (Simpson,e.g., lack of funds for continuing education (Simpson,2002).2002).
We know little about the most effective mannerWe know little about the most effective mannerin which to implement EBPsin which to implement EBPs
–– ((HenggelerHenggeler, Lee, & Burns, 2002; Morgenstern, 2000), Lee, & Burns, 2002; Morgenstern, 2000)
New models of implementation have beenNew models of implementation have beendevelopeddeveloped
–– (Aarons, 2005; (Aarons, 2005; FrambachFrambach & & SchillewaertSchillewaert, 2002; Klein,, 2002; Klein,ConnConn,& ,& SorraSorra, 2002)., 2002).
CASRC
Implementation is ComplexImplementation is Complex
Implementation should be evidence-basedImplementation should be evidence-based
Implementation is a multilevel issue (Dixon et al., 2001).Implementation is a multilevel issue (Dixon et al., 2001).–– PoliciesPolicies
–– AgenciesAgencies
–– ProgramsPrograms
–– Administrative staffAdministrative staff
–– CliniciansClinicians
–– ConsumersConsumers
Clear, comprehensive, measurable, and testableClear, comprehensive, measurable, and testableimplementation models are needed to guide research onimplementation models are needed to guide research onorganizational changeorganizational change
There are few empirical studies addressing these issues inThere are few empirical studies addressing these issues inyouth mental health servicesyouth mental health services
19th Annual RTC Conference
Presented in Tampa, February 2006
2
CASRC
Goals of the StudyGoals of the Study
To identify barriers and facilitators ofTo identify barriers and facilitators of
adoption of EBPs for organizations servingadoption of EBPs for organizations serving
youth with Mental Health disordersyouth with Mental Health disorders
Examine what various stakeholder groupsExamine what various stakeholder groups
identify as most important and mostidentify as most important and most
changeable.changeable.
CASRC
Methods IMethods I
Programs within agencies selected based on:Programs within agencies selected based on:
–– Types of Services ProvidedTypes of Services ProvidedOutpatientOutpatient
Day TreatmentDay Treatment
Case ManagementCase Management
Residential/InpatientResidential/Inpatient
–– Size of AgencySize of AgencyLarge and SmallLarge and Small
–– Size of ProgramSize of ProgramLarge and SmallLarge and Small
–– LocationLocationUrban vs. RuralUrban vs. Rural
CASRC
Participant SelectionParticipant Selection
Selected programs were either operated by theSelected programs were either operated by theCounty or provided contract services to theCounty or provided contract services to thecounty.county.
Organizational structures varied by level ofOrganizational structures varied by level ofbureaucracy and fiscal constraints on servicesbureaucracy and fiscal constraints on services(Aarons, 2004)(Aarons, 2004)
Individual participants selected by snowballIndividual participants selected by snowballsamplingsampling
CASRC
Sample SelectionSample Selection
Participants drawn from 6 organizationalParticipants drawn from 6 organizationallevels:levels:
Policy: County Mental Health Officials (n = 6)Policy: County Mental Health Officials (n = 6)
Agency: Organization/Agency directors (n = 5)Agency: Organization/Agency directors (n = 5)
Program: Program managers (n = 6)Program: Program managers (n = 6)
Clinical: Clinicians (n = 7)Clinical: Clinicians (n = 7)
Administrative: Administrative staff (n = 3)Administrative: Administrative staff (n = 3)
Consumers: Consumers of MH services (n = 5)Consumers: Consumers of MH services (n = 5)
CASRC
9.73Other
3.21Asian American
3.21African American
9.73Hispanic
74.223Caucasian
Race
27-6010.944.4Age
61.319Female
38.712Male
Gender
RangeSDMean%N
Demographics (N=31)Demographics (N=31)CASRC
7.72To a great extent
30.88To a moderate extent
30.88To a slight extent
30.88Not at all
Experience Implementing EBPs
RangeSDMean%N
Demographics Mental HealthDemographics Mental Health
19th Annual RTC Conference
Presented in Tampa, February 2006
3
CASRC
ProcedureProcedure
Concept Mapping Concept Mapping ((TrochimTrochim, Cook, & , Cook, & SetzeSetze, 1994), 1994)
–– Mixed qualitative-quantitative methodMixed qualitative-quantitative method
–– Qualitative methods used to generate dataQualitative methods used to generate data
–– Data analyzed using quantitative methodsData analyzed using quantitative methods
Begin with structured brainstormingBegin with structured brainstorming
–– Participants Participants generategenerate and then and then useuse a focus a focus
statement to guide identifying barriers andstatement to guide identifying barriers and
facilitators to implementationfacilitators to implementation
CASRC
ProcedureProcedure
Focus statementFocus statement
–– ““What are the factors that influence theWhat are the factors that influence the
acceptance and use of evidence-basedacceptance and use of evidence-based
practices in publicly funded mental healthpractices in publicly funded mental health
programs for families and children?programs for families and children?””
Independent stakeholder groupIndependent stakeholder group
brainstormingbrainstorming
Statements combined across all groupsStatements combined across all groups
CASRC
ProcedureProcedure
““UnstructuredUnstructured”” Card Sort Card Sort
–– 105 Statements105 Statements
–– All participants sort the same statementsAll participants sort the same statements
–– Sorted based on similaritySorted based on similarity
–– >1 pile>1 pile
Statement RatingsStatement Ratings
–– "Importance"Importance““
–– "Changeability""Changeability"
–– 0 to 4 point scale0 to 4 point scale(Not at all (Not at all !! A very great extent) A very great extent)
CASRC AnalysisAnalysis
Multidimensional scaling (MDS) and cluster analysisMultidimensional scaling (MDS) and cluster analysis
MDS analysis results in a MDS analysis results in a ““mapmap”” of the conceptual space of the conceptual spacewith similar issues closer togetherwith similar issues closer together
Solution represents psychological Solution represents psychological ““distancedistance”” or similarity or similaritybetween conceptsbetween concepts
Statements more similar in meaning are closer togetherStatements more similar in meaning are closer together
Statements grouped into non-overlapping categoriesStatements grouped into non-overlapping categoriescalled clusterscalled clusters
Clusters closer together are more conceptually relatedClusters closer together are more conceptually related
CASRC
ResultsResults
Fourteen overall clusters best fit dataFourteen overall clusters best fit data
One overall solution for all participantsOne overall solution for all participants
–– Participants reconvene to Participants reconvene to ““make sensemake sense”” of of
solutionsolution
–– Cluster namingCluster naming
Importance ratings overlaid on solutionImportance ratings overlaid on solution
CASRC
14 Clusters14 Clusters
"#"# Clinical PerceptionsClinical Perceptions
$#$# Staff Development & SupportStaff Development & Support
%#%# Staffing ResourcesStaffing Resources
&#&# Agency CompatibilityAgency Compatibility
'#'# EBP LimitationsEBP Limitations
(#(# Consumer ConcernsConsumer Concerns
)#)# Impact on Clinical PracticeImpact on Clinical Practice
*#*# Beneficial Features (of EBP)Beneficial Features (of EBP)
+#+# Consumer Values & MarketingConsumer Values & Marketing
",#",#System Readiness & CompatibilitySystem Readiness & Compatibility
""#""#Research & OutcomesResearch & Outcomes
"$#"$#Political DynamicsPolitical Dynamics
"%#"%#FundingFunding
"&#"&#Costs of EBPCosts of EBP
19th Annual RTC Conference
Presented in Tampa, February 2006
4
CASRC
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27 28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
Point and Cluster Map CASRC
1 Clinical Perceptions
2 Staff Development & Support
3 Staffing Resources
4 Agency Compatibility
5 EBP Limitations
6 Consumer Concerns
7 Impact on Clinical Practice8 Beneficial features (of EBP)
9 Consumer Values & Marketing
10 System Readiness & Compatibility
11 Research & Outcomes Supporting EBP
12 Political Dynamics
13 Funding14 Costs of EBP
Cluster Legend Layer Value
1 2.68 to 2.78
2 2.78 to 2.87
3 2.87 to 2.97
4 2.97 to 3.07 5 3.07 to 3.17
Figure 1: Overall Solution with Figure 1: Overall Solution with Importance Ratings
CASRC
Consumer Values & Marketing
Research & Outcomes Supporting EBP
Consumer Concerns
Impact on Clinical Practice
Clinical Perceptions
Staff Development & Support
Staffing Resources
Agency Compatibility
Costs of EBP
Funding
System Readiness
Compatibility
Beneficial Features (of EBP)
Political Dynamics
Figure 2: County Officials
EBP Limitations
CASRC
Consumer Values & Marketing
Research & Outcomes Supporting EBP
Consumer Concerns
Impact on Clinical PracticeBeneficial Features (of EBP)
Political Dynamics
Funding
Costs of EBP
Agency Compatibility
EBP Limitations
Clinical Perceptions
Staffing Resources
Staff Development & Support
System Readiness &
Compatibility
Figure 3: Agency Directors
CASRC Figure 4: Program Managers
Consumer Values & Marketing
Research & Outcomes Supporting EBP
Consumer Concerns
Impact on Clinical Practice
Clinical Perceptions
Staffing Resources
Staff Development & Support
EBP Limitations
Beneficial Features (of EBP)
Political Dynamics
Funding
Costs of EBP
Agency Compatibility
System Readiness &
Compatibility
CASRC Figure 5: Clinicians
Consumer Values & Marketing
Research & Outcomes Supporting EBP
Consumer Concerns
Beneficial Features (of EBP)
EBP Limitations
Impact on Clinical Practice
Clinical Perceptions
Staff Development & Support
Staffing Resources
System Readiness & Compatibility
Political Dynamics
Funding
Costs of EBP
Agency Compatibility
19th Annual RTC Conference
Presented in Tampa, February 2006
5
CASRCFigure 6: Administrative Staff
Consumer Values & Marketing
Research & Outcomes Supporting EBP
Consumer Concerns
Impact on Clinical Practice
Clinical Perceptions
Staff Development & Support
Staffing Resources
EBP Limitations
Agency Compatibility
Costs of EBP
Funding
Political Dynamics
System Readiness & Compatibility
Beneficial Features (of EBP)
CASRC Figure 7: Consumers
Funding
Costs of EBP
Agency Compatibility
Staffing Resources
Staff Development & Support
Clinical Perceptions
System Readiness & Compatibility
EBP Limitations
Beneficial Features
(of EBP)
Research & Outcomes Supporting EBP
Consumer Values & Marketing
Political Dynamics
Consumer Concerns
Impact on Clinical Practice
CASRC Importance Rating ScaleImportance Rating Scale
3.103.062.812.603.002.502.81System Readiness &
Compatibility
3.083.273.053.293.263.063.16Staffing Resources
3.323.473.153.093.122.963.16Staff Development & Support
3.223.152.953.213.112.913.09Research & Outcomes
Supporting EBP
2.803.222.782.953.132.672.90Political Dynamics
3.383.333.022.592.802.002.81Impact on Clinical Practice
3.333.712.943.003.253.133.17Funding
2.533.112.722.672.802.532.70EBP Limitations
3.093.562.963.083.422.913.13Costs of EBP
3.473.112.672.812.732.602.87Consumer Values & Marketing
3.033.262.872.842.672.592.85Consumer Concerns
2.953.333.082.822.532.702.88Clinical Perceptions
3.403.112.783.052.532.802.94Beneficial features (of EBP)
2.913.112.722.542.642.362.68Agency Compatibility
CnsmrAdminClincnPrgm
Mgr
Agncy
Dir
Cnty
Offcls
All
CASRC
ResultsResults
For the overall group, Funding was rated theFor the overall group, Funding was rated the
most important factor and rated the leastmost important factor and rated the least
changeable.changeable.
Staffing Resources and Staff Development andStaffing Resources and Staff Development and
Support were rated most important after funding.Support were rated most important after funding.
Clinical Perceptions and Consumer Values andClinical Perceptions and Consumer Values and
Marketing were rated most changeableMarketing were rated most changeable
Staff Development and Support ranked third inStaff Development and Support ranked third in
importance and fourth in changeabilityimportance and fourth in changeability
CASRC
ConclusionConclusion
Found a common solution that representsFound a common solution that represents
multiple stakeholder perspectivesmultiple stakeholder perspectives
There are a number of multiple stakeholderThere are a number of multiple stakeholder
concerns that may impact implementation ofconcerns that may impact implementation of
EBPs in real world service settings.EBPs in real world service settings.
Groups varied on Importance and ChangeabilityGroups varied on Importance and Changeability
ratings.ratings.
It is important to consider the concerns ofIt is important to consider the concerns of
multiple stakeholders in EBP implementation.multiple stakeholders in EBP implementation.
CASRC
ConclusionsConclusions
Processes for egalitarian multiple stakeholders input canProcesses for egalitarian multiple stakeholders input canfacilitate cultural exchangefacilitate cultural exchange
Stakeholder perspectives can inform implementationStakeholder perspectives can inform implementationprocessprocess
Examples:Examples:
–– Optimizing message content may promote more positiveOptimizing message content may promote more positiveattitudes toward implementation of change in service modelsattitudes toward implementation of change in service models
–– Staff issues need to be addressed up front to promoteStaff issues need to be addressed up front to promoteimplementation effectivenessimplementation effectiveness
Further research is needed to better understand howFurther research is needed to better understand howfactors identified in the present study impact actual EBPfactors identified in the present study impact actual EBPimplementation efforts.implementation efforts.
19th Annual RTC Conference
Presented in Tampa, February 2006
6
CASRC Reference List
Aarons, G. A. (2005). Measuring provider attitudes toward evidence-based practice: Organizational
context and individual differences. Child and Adolescent Psychiatric Clinics of North America, 14,
255-271.
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice:
The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61-
74.
Dixon, L., McFarlane, W. R., Lefley, H., Lucksted, A., Cohen, M., Falloon, I. et al. (2001). Evidence-
based practices for services to families of people with psychiatric disabilities. Psychiatric Services,
52(7), 903-910.
Haynes, B., & Haines, A. (1998). Barriers and bridges to evidence based clinical practice. Biomedical
Journal, 317, 273-276.
Henggeler, S. W., Lee, T., & Burns, J. A. (2002). What happens after the innovation is identified?
Clinical Psychology: Science and Practice, 9(2), 191-194.
Hoagwood, K., & Olin, S. (2002). The NIMH blueprint for change report: Research priorities in child
and adolescent mental health. Journal of the American Academy of Child and Adolescent
Psychiatry, 41(7), 760-767.
Jensen, P. S. (2003). Commentary: The next generation is overdue. Journal of the American Academy
of Child and Adolescent Psychiatry, 42(5), 527-530.
Morgenstern, J. (2000). Effective technology transfer in alcoholism treatment. Substance Use &
Misuse, 35(12-14), 1659-78.
Simpson, D. D. (2002). A conceptual framework for transferring research to practice . Journal of
Substance Abuse Treatment, 22(4), 171-182.
Trochim, W. M., K, Cook, J. A., & Setze, R. J. (1994). Using concept mapping to develop a conceptual
framework of staff's views of a supported employment program for individuals with severe mental
illness. 62(4), 766-775.
Top Related