Leveraging social media - DrugInfo seminar - Leveraging social media
Leveraging Implementation Practice and Research For The ... · Leveraging Implementation Practice...
Transcript of Leveraging Implementation Practice and Research For The ... · Leveraging Implementation Practice...
Leveraging Implementation Practice and Research
For The Greater Good
Sonja K. Schoenwald Professor of Psychiatry & Behavioral Sciences
Medical University of South Carolina April 8, 2014
The Fourth Annual CiMH Evidence-Based Practice Symposium
Sacramento, CA
Brought to You By •California Institute for Mental Health 2 NIMH-Funded Centers for Services and Intervention Research • Implementation Methods Research Group (IMRG); John Landsverk, PI
– NIH 1P30MH074678 (Advanced Center) • LINKS Center, Marc Atkins, PI
– NIH 1P20MH0784458 (Developing Center) 2 Philanthropies • The Annie E. Casey Foundation • The Research Network on Youth Mental Health, John Weisz, PI
– John D. and Catherine T. MacArthur Foundation Several NIMH and NIDA funded studies, including • National Institute of Mental Health, S. Schoenwald & J. Chapman Co-PIs
– 1 R21 MH097000-01, 06/11/12 – 05/31/2014 • Transportability of New Treatments… 1R01MH59138, S. Schoenwald, PI • Testing Context Effects… R01 DA018107-01, S. Schoenwald, PI • Several effectiveness and implementation studies conducted by Scott Henggeler And, I am a Board Member and Stockholder, MST Group, LLC, which has the exclusive licensing agreement through MUSC for the dissemination of MST technology
This Morning
• Why care about implementation? • What research tells us about implementation
strategies and tools; a selective review • What we can do to increase the impact of
practice and research in implementation; some ideas, some examples
Why Care About Implementation? • People cannot benefit from innovations they
do not experience (courtesy of Dean Fixsen) • How well an innovation works and how well it
is implemented are two different things
Someone left the cake out in the rain. I don’t think that I can take it. Cause it took so long to bake it. And I’ll never have that recipe again, Oh, no!
Why is Implementation Research Relevant? Dissemination Step Concept % Impacted
50% of Clinics Use Adoption 50%
50% of Practitioners Recommend Adoption 50%
50% of Patients Accept Recommendation/Attempt Change
Reach 12.5%
50% Follow Regimen Advice Correctly Implementation 6.2%
50% of Those Implementing Correctly have Substantial Benefit
Effectiveness 3.1%
50% Continue to Adhere and Benefit after 6 Months
Maintenance 1.6%
Courtesy of Russell Glasgow, Re-Aim http://www.re-aim.org/
Source: Aarons, Hurlburt & Horwitz, 2011
ClLIENT OUTCOMES
• Implementation research examines “the use of
socio-behavioral strategies to adopt, integrate, and scale-up evidence-based interventions and change practice patterns within specific settings” Chambers, Ringeisen, & Hickman, 2005
Implementation Research Leverages The
Value and Impact of Both Treatment Research and Clinical Practice
Model for Implementation Science
8
Implementation Strategies
Outcomes
Service Outcomes*
Efficiency Safety Effectiveness* Equity Patient-centeredness Timeliness
Client Outcomes
Satisfaction Function* Symptomotology*
Implementation Outcomes
Feasibility Fidelity* Penetration Acceptability Sustainability Uptake
Costs
Intervention Strategies
Evidence-Based Practices
Implementation Research Methods
Systems Environment
Organizational*
Group/Learning*
Supervision*
Individual * Providers/Consumers *IOM Standards of Care
Red bold font* = Today’s talk
Source: Proctor, Landsverk, Aarons, Chambers, Glisson, & Mittman, 2009
Quality Tools for Implementation
• Information-centered work requires quality tools that are designed and refined on the basis of relevant research (e.g., educational, cognitive, informatics) and user input (Wandersman, Chien, & Katz, 2012)
• The same guidance applies to human services (with domains of relevant research varying)
Training
• Adequate training can reduce variations in provider behavior, improve fidelity, and increase service delivery quality.
• In what settings, under what training and support conditions, can this be achieved?
• Research on training and support in EBP implementation contexts is in early stages
• But, we do know from research on learning, adult education, and professional training. . .
Evidence-Based Training ?
“Telling Ain’t Training Training Ain’t Performance”
Harold Stolovitch
Learning and Performance Solutions American Society for Training and Development and the International
Society for Performance Improvement Courtesy of the
Annapolis Coalition on the Behavioral Health Workforce http://www.annapoliscoalition.org/
Performance Based Training
• Know: Knowledge acquisition • Know How: Skill Acquisition • Do: Skill performance • Performance-based training includes
o Deliberate practice o Experiential learning o Feedback o Support
Toward Evidence-Based Training • One-time workshops can increase knowledge and
attitude toward change but not behavior • Building proficiency appears to require repetitions and
active learning o One estimate: 20 – 25 implementation attempts to achieve
consistent professional behavior change (Joyce & colleagues, 2002)
• Approaches & techniques recently reviewed o academic detailing o coaching o interprofessional learning o reminders o self-regulated learning o problem-based learning
Lyon, Wiltsey Stirman, Kerns, & Bruns (2011)
Coaching and Consultation
Common Types1 • Expertise-based • Peer-to- peer
Common Methods1 • Direct feedback following observation • Coach reflection on own practice
Emerging Type? • Peer - Expert
1Lyon, Wiltsey Stirman, Kerns, & Bruns (2011)
Coaching and Consultation
• Expertise-Based Coaching Examples o Trauma-Focused Cognitive Behavioral Treatment o Parent-Child Interaction Training o Multisystemic Therapy Consultation effects on adherence and outcomes
• Peer Coaching Example o Triple P Positive Parenting Program encourages
peer support groups One study underway to evaluate advantages and
challenges of using these groups
Lyon, Wiltsey Stirman, Kerns, & Bruns, 2011
Coaching and Consultation
• Specific functions, processes, of consultation o Engagement o Problem solving implementation barriers o Direct case application o Appropriate adaptation o Accountability o Mastery skill building o Sustainability planning
*Nadeem, Gleacher, & Beidas (2013). Consultation as an implementation strategy for evidence-
based practices across multiple contexts: Unpacking the black box. Administration & Policy in Mental Health and Mental Health Services Research
Toward Evidence-Based Consultation • Dosage, delivery-method, collaboration, &
proactive nature matter (Wandersman, Chien, & Katz, 2012)
• Less is known about the specific functions and processes that are most effective
• Study underway: o Stirman & colleagues (2013) RCT: 3 consultation conditions for clinicians treating PTSD
in military veterans with Cognitive Processing Therapy (CPT) Outcomes: CPT adherence, competence; client symptoms
Stirman et al., (2013)
Clinical Supervision What Constitutes Effective Supervision?
• Clinician behavior change • Effects on client outcomes
“The “acid test” of good supervision is client outcomes” (Ellis & Ladany, 1997)
• The “acid test” is rarely conducted • Quality of EBT coaching in clinical
supervision associated with: o Decreased emotional exhaustion & turnover intention
(Knudsen & colleagues, 2008) o Decreased turnover (Aarons & colleagues 2009)
Toward Evidence-Based Supervision
Examples from EBT Implementation Research
• “Old”News: MST Transportability Research o 9-site pilot included supervisor adherence
measurement development o 43-site study
• “New” News: • Research Network on Youth Mental Health,
System and Treatment Enhancement Projects o Prospective observational study of supervision o During RCT of treatment effectiveness o Valid, reliable measurement of supervision
Toward Evidence-Based Supervision
MST Transportability Study
• Two of four MST SAM scales predicted one-year post-treatment changes in youth behavior problems
• One scale predicted therapist adherence • Direct effects of supervisor adherence and
therapist adherence were observed
Schoenwald, Sheidow, & Chapman, 2009
Clinical Supervision, MST Transportability Study
SAM Structure & Process Predicting Post-Treatment Criminal Charges
0.86 (+1 SD)1 (Max.)
0 (Min.)
0.76 (Mean)0.66 (-1 SD)
1
1.5
2
2.5
3
3.5
4
0 (Min.) 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 (Max.)
Supervisor SAMSP
Num
ber o
f Pos
t-Tre
atm
ent C
harg
es
*MST Transportability Study, funded by NIMH and NIDA
Results reported in Schoenwald, 2008
Toward Evidence-Based Supervision
New News: “More Practice, Less Preach?”
• 57 community-based therapists trained in EBPs • 12 doctoral level supervisors with EBP expertise • 136 youths and families • Modeling and role-play higher use than discussion • Modeling practice use in next session • Greater effects for older clinicians • Discussion predicted use for male therapists (n = 18)
Bearman, Weisz, Chorpita et al & The Research Network on Youth Mental Health, 2013
Toward Evidence-Based Supervision
Newest News – Study In Progress • Randomized trial of supervision in public mental
health in Washington State (Dorsey & colleagues, in press)
o Descriptive study of supervision of those trained in TF-CBT via WA public mental health initiative
o RCT of supervision elements from efficacy trials (session review, model fidelity, outcome monitoring, skill-building – behavioral rehearsal) Both conditions: Symptom and fidelity monitoring One condition: SFM plus Behavioral Rehearsal in
supervision
Feedback To Practitioners Health Care
• Audit-and-feedback systems • Effective for:
o Relatively straightforward tasks o Physicians demonstrating low baseline adherence o When the feedback is sufficiently detailed and intensive.
(Grimshaw et al, 2001; Jamvedt et al., 2006)
Mental Health/Behavioral Health • In mental health/behavioral health, there are fewer,
and fewer rigorous, studies • Evidence from randomized trials appears promising
Feedback To Practitioners
• Feedback to school-based providers about fidelity to Coping Power as rated on observational measures improved fidelity and outcomes
(Lochman, Boxmeyer, Powel, Qu, Wells, & Windle, 2009.)
• Feedback to therapists on standardized measures of client progress and outcome
(Bickman, Kelley, Breda, De Andrade, & Riemer, 2011).
o Outcomes for children were better when practitioners had access to weekly feedback
o Positive association between practitioners’ frequency of use of outcome feedback and the extent of child improvement
From Practitioner Feedback to Implementation Feedback Systems
“Implementation feedback systems could support the reach, quality, and sustainability of evidence-based treatments”
McLeod, Southam-Gerow, Tully, Rodriguez, & Smith (2013). Making a case for
treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20, 14-32.
Implementation Feedback Systems • Distinguish closer to real time the failure of
implementation from the failure of intervention
• Refine the target of implementation support o Practitioners: Intervention- specific skills and performance o Organization: policies, procedures, leadership,
implementation climate, climate-and-culture profile o Service system: Regulations, procedures, norms, inter-
organizational interactions
• Support “learning organization” process and culture
• Contribute “shared yardstick of accountability”* amongst the multiple participants in implementation *Yeaton & Sechrest, 1981
You Get What You Measure Measurement Focuses Attention, Behavior, & Resources Some Examples
o Hospital Accreditation o No Child Left Behind o Child Welfare
Potentially Perverse Effects o No Child Left . . . Untested? o Child Welfare Paperwork
“Driving with Roadmaps & Dashboards…”
Fig. 3 Individual client summary example from the child STEPs clinical trial (Chorpita et al. 2008 ) Springer Science and Business Media
Help?
Data Dashboards
Characteristics of
Effective Measurement
Characteristics of
Efficient Measurement
Balancing characteristics of effective and efficient adherence measurement on the basis of measurement purpose(s)
Purpose(s)
Stakes (Higher/Lower)
Number (Single, Multiple)
Schoenwald, Garland, Chapman, Frazier, Southam-Gerow, & Sheidow, 2011
Quality Tools for Implementation
• Quality tools alone are not enough Wandersman, Chien, & Katz, 2012
Multicomponent Strategies
Examples • Availability, Responsiveness, Continuity (ARC;
Glisson & colleagues) • Community Development Teams (CDT; CIMH) • Interagency Collaborative Teams (ICT; Aarons
& colleagues, under evaluation) • Learning Collaboratives (see Nadeem et al., in press,
Millbank Quarterly)
Availability, Responsiveness, Continuity (ARC; Glisson & Colleagues)
Objective • “Help community based service settings improve their social contexts
by creating the capacity to identify & address contextual deficits and barriers that impede service and treatment outcomes” (Glisson, Hemmelgarn, Green, & Williams, 2013, p. 494)
Means • ARC facilitator • Team-based clinician and management focused activities • Three broad strategies
o Embed 5 principles into the organization’s processes & practices o Use 12 component tools to identify and address service barriers
(within organization and in inter-organizational relations) o Cultivate service provider attitudes and behaviors (e.g., openness
to change • Implement over an 18-month period
Rural Appalachian Project (RAP) Randomized Trial of MST & ARC
in a 2-Level EBT Implementation Strategy Glisson, Schoenwald, Hemmelgarn, Green, Dukes, Armstrong, & Chapman, 2010
Questions • Are MST model fidelity and outcomes better
in counties that receive ARC? • Are youth outcomes better in ARC counties
than non-ARC counties? • Are youth outcomes better for youth who
receive MST compared to youth who receive the usual services?
RAP Implementation Results
• MST Implementation in ARC and non-ARC o No differences in adherence (therapist or supervisor)
o Therapists in ARC counties rated progress with extra-familial systems more highly and spent fewer minutes within the family system
o But, no differences in the odds a therapist addressed a particular system in natural or service ecology
RAP Outcomes Results **
Out of home placement* 18-months after baseline
• ARC reduced the odds of placement by 44% • MST reduced the odds of placement by 53% • Together, MST and ARC reduced the odds of out of
home placement from 34% to 16% • Mechanism of action for combined effects not yet
evaluated (e.g., is effect in part accounted for by ARC-induced change in team climate/ culture?)
*Parent-reported
MTFC Implementation Trial*
• Randomized trial of Multidimensional Treatment Foster Care (MTFC)
• 40 counties in California; 11 counties in Ohio • Randomized to 2 implementation conditions Standard Implementation Community Development Teams (CDT)
• Both conditions received standard MTFC training, clinical consultation and QA procedures
• Randomized to cohorts for start time • Total of 51 sites consented
* Courtesy of Patti Chamberlain and Lisa Saldana; Research Funding from NIMH, Children’s Bureau, WT Grant Foundation
8 Stages: Involvement: 1. Engagement System Leader 2. Consideration of Feasibility System Leader, Agency 3. Readiness Planning System Leader, Agency 4. Staff Hired and Trained Agency, Practitioner 5. Adherence Monitoring Practitioner, Client Established 6. Services and Consultation Practitioner, Client 7. Ongoing Services, Consultation Practitioner, Client Fidelity Monitoring, Feedback 8. Competency (certification) System Leader, Agency,
Practitioner, Client
Stages of Implementation Completion (SIC)
Pre
Imp
Sus
Chamberlain, P., Brown, C.H., & Saldana, L. (2011).
• Meaningful prediction of implementation milestones
• Pre-implementation SIC behavior predicts successful program start-up
• Pre-implementation SIC behavior predicts discontinuing program
• Thus far, pre-implementation SIC behavior does not predict competency but does predict penetration
Emerging SIC Findings
Findings reported in Brown, Chamberlain, Saldana & Wang, submitted
Aims • Adapt the SIC for other EBPs • Evaluate SIC for other EBPs (predictive ability, which activities
are common across practices and systems, psychometrics) • Conduct cost-mapping using the COINS strategy to determine
resource allocation needed across the different practices to conduct different implementation activities
Method • 3 evidence-based treatments, 3 service sectors (JJ, CW,
Education) • Retrospective assessment of activities in 15 sites per
treatment • Prospective assessment of activities in 15 sites per treatment
Utility of SIC for Other Treatments ?
“Stages of Implementation Completion for Evidence Based Practices” L. Saldana, PI, NIH R01MH097748
Infrastructure and Implementation
. Koerth-Baker, M. (2012, October). Why your car isn't electric. The New York Times Magazine.
• The feasibility of technology use is a contextual experience •Infrastructure supporting current use of a technology is well developed and often invisible at the point of use •When use of a new technology requires adjustments to infrastructure, feasibility of use appears limited •As infrastructure to support the technology becomes commonplace, use is experienced as feasible
Implications for Larger Scale
Implementation of Effective Interventions
• “Practical” constraints on implementation may reinforced by “invisible infrastructure”
• That infrastructure can be revealed and altered
• Alternative infrastructures can be developed
Harnessing Technology Infrastructure For Implementation
Examples From Currently Ongoing Research • Broad-band enabled tablets for coach observation of
SafeCare ® fidelity during home visits (G. Aarons, PI)
• Distance digital recording and web-based feedback of group supervision sessions (Chapman & Schoenwald, Co-PIs)
• Internet-based fidelity monitoring for MTFC pilot study (Feil & colleagues, 2012).
• Computer-assisted and e-learning modalities for training (see review and discussion by Beidas, et al., 2011)
• Evidence-Based Implementation Support Systems (EBSIS; Wandersman, Chien, & Katz, 2012)
• “The D&I research field has advanced from examining barriers and facilitators to investigating the uptake of effective interventions and conducting comparative studies of systematic approaches to implementation” (Chambers & Azrin, 2013).
From Implementation Strategies to Implementation Systems
Leveraging Practice And Research for the Greater Good
• Ongoing, Collaborative Partnerships o For strategic and opportunistic purposes o Conjoint discernment of relevancy of goals (but, beware status quo) o For comparative studies of multicomponent, systematic strategies
• Synergies & Complementarities: of purpose, expertise, methods o When designing a project for one purpose, consider which data and methods
can legitimately accomplish additional purposes. o If none, are there data and methods that could be added with minimal
burden and fundraising (grant writing) to accomplish those purposes?
• Some Examples: o Development of the SIC in the context of MTFC Implementation Trial o Child STEPs, Research Network on Youth Mental Health o MST adherence instrument language mini-studies o NIMH FOA on fidelity assessment in practice contexts
References
Aarons, Hurlburt & Horwitz, 2011 Aarons, G. A., Hurlburt, M., & Horwitz, S.M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4-23. DOI 10.1007/s10488-010-0327-7. Aarons, Green, Palinkas, Self-Brown, Whitaker, Lutzker, Silovsky, Hecht, & Chaffin (2012). Dynamic adaptation process to implementation of an evidence-based child maltreatment intervention. Implementation Science, 7:32. Aarons, GA, Sommerfeld, DH, Hecht, DB, Silovsky JF, Chaffin MJ: The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology 2009, 77:270-280. Beidas, R.S., & Kendall, P.C., Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17 (1). 1 – 30. Boxmeyer, C.L., Lochman, J.E., Powell, N.R., Windle, M.,& Wells, K. (2008, Fall). School counselors’ implementation of Coping Power in a dissemination field trial: Delineating the range of flexibility with in fidelity. Emotional and Behavioral Disorders in Youth, pp. 79-84, 94-5 Chambers & Azrin (2013). Research and services partnerships: Partnerships: A fundamental component of dissemination and implementation research. Psychiatric Services, 64, 509-513. Chamberlain, P., Brown, C.H., & Saldana, L. (2011). Observational measure of implementation progress: The Stages of Implementation Completion (SIC). Implementation Science Dorsey, S., Pullmann, M.D., Deblinger, E., Berliner, L., Kerns, S.E., Thompson, K., et al. (in press). Improving practice in community-based settings: A randomized trial of supervision – study protocol, Implementation Science. Glasgow R & Steiner J. (2012). In Dissemination and Implementation Research. Brownson, R, Colditz, G, and Proctor, E (Eds.). Oxford University Press. Grimshaw, J.M., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., Grilli, R., Harvey, E., Oxman, A., & O’Brien, M.A. (2001). Changing provider behavior: An overview of systematic reviews of interventions. Medical Care, 39, 8 (Suppl 2), pp. II-2 – II-45. Knudsen HK, Ducharme LJ, Roman PM: Clinical supervision, emotional exhaustion, and turnover intention: A study of substance abuse treatment counselors in the Clinical Trials Network of the National Institute on Drug Abuse. Journal of Substance Abuse Treatment 2008, 35:387-395
References
Knudsen HK, Ducharme LJ, Roman PM: Clinical supervision, emotional exhaustion, and turnover intention: A study of substance abuse treatment counselors in the Clinical Trials Network of the National Institute on Drug Abuse. Journal of Substance Abuse Treatment 2008, 35:387-395 Lochman, J.E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77, 397-409. Lyon, Wiltsey Stirman, Kerns, & Bruns (2011). Developing the mental health workforce: Review and application of training approaches from multiple disciplines. In Becker & Stirman, Guest Editors: Special Issue, Training Clinicians in Evidence-Based Interventions, Administration and Policy in Mental Health and Mental Health Services Research, 4. McLeod, B.D., Southam_Gerow, M.A., Tully, C.B., Rodriguez, A., & Smith, M.M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology, Science and Practice, 20, 14 - 32 MST Transportability Study References Available from the Family Services Research Center, Medical University of South Carolina, at:
http://www.musc.edu/psychiatry/research/fsrc/pubs.htm Nadeem, Gleacher, & Beidas (2013). Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration & Policy in Mental Health and Mental Health Services Research Proctor, E.K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., Mittman, B. (2009). Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24-34 Saldana, L., Chamberlain, P., Wang, W., & Brown, H. (2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health Research, 39, 419-425. Schoenwald, S.K., (2011). It’s a bird, it’s a plane, it’s . . . Fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18 , 142-147. Schoenwald, S.K., Garland, A.F., Chapman, J.E., Frazier, S.L., Sheidow, A.J., & Southam-Gerow, M.A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32-43. Stirman, Shields, Deloriea, Lnady, Belus, Maslej, & Monson, (2013) A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for Posttraumatic stress disorder. Implementation Science, 8:82 Wandersman, A., Chien, V.H., & Katz, J. (2012). Toward an Evidence-Based System for Innovation Support for Implementing Innovations with Quality: Tools, Training, Technical Assistance, and Quality Assurance/Quality Improvement. American Journal of Community Psychology, 50, 445-459.