Supervision Strategies to Enhance Implementation and

Fidelity to EBP

Kelly Pitocco, LISW-S, LICDC

University of Cincinnati

Corrections Institute

kelly.pitocco@uc.edu

Putting Supervision in

Context of Implementation

Process

To Do . . . PractitionerTraining

State Policy

Funder

Supervisor

Training and Support

Supervisor

Training and Support

The Four Phases of Learning

Preparation

Practice

Presentation

Performance

OftenSkipped

Preparation

Assessing and Readying for Change

Preparation

• Get them interested

• Encourage positive feelings about new learning

• Create Learning Environment

Survey of AOD Professionals

83%

Past Year use ofCBTMETMI

12 Step Facilitation

75% Reported Currently Using EBPs

However, Many have Negative View of Curriculum-based Treatment

Attitudes About EBP

Most Clinicians eitherBelieve they are Using EBP

Or Want to Use EBP

Positive Attitudes

• Using manual help counselor evaluate and improve skills

• Treatment manual will enhance outcomes

• Counselors are ethically obliged to use EBP

Negative Attitudes

• EBP make staff more like technician than caring people

• Treatment manuals appropriate for research but not real life

Readiness Factor

Prepare for Implementation

• Determine if the organization and program are ready to adopt model

• Assess staff readiness to change

• Set up a multidisciplinary change committee

• Program leadership and change committee develop the training and follow-up plan

Areas for Readiness

• Strategic Plan

• Readiness Assessment

• Project Management

• Change Management

• Staff Training

• Staff Supervision– Rating– Coaching

• Monitoring Fidelity– CQI– Evaluation

•Staff Training

•Staff Supervision•Rating•Coaching

Goals for Preparation

• Get staff out of a passive or resistant mental state

• Remove learning barriers• Arouse interest• Give staff positive feelings about, and an

incentive to learn• Create active learners• Establish a learning group

Techniques

• Provide positive suggestion

• Discuss benefits to clinician and client

• Set clear, meaningful goals

• Raise curiosity

• Create a safe and positive environment

• Calm staff fears

• Identify and remove barriers to learning

Techniques

• Raising questions and posing problems

• Getting staff involved in implementation

Learner Barriers

NO WIIFM

Need to Save Face

If it Ain’t Broken,Don’t Fix it

This is the WayWe’ve Always

Done it

The way I do itIs Good Enough

I don’thave Time

PersonalIssuesI already do this

Learner Benefits

• Brainstorm benefits

Curiosity Arousal

• Give people problems to solve in teams

• Send staff on fact-finding missions

• Play question and answer games

• Self-discovery activities

Presentation

Broader than Training

• Need a mechanism to accomplish:

– Acceptance of change

– Means to incorporate change

– Reinforced at all levels of system

Typically underestimate the time and effort needed to:

• Train

• Implement

• Achieve fidelity to the model

Occasional Quotes from Trainees

“I was told I have to be at this training.

I have no idea why I am here.”

“I’m just here to get my 30 hours/CE’s/

mandatories done”

“That won’twork at my

Site/with ourclients/withinthe time we

have.”

Training Culture

Successful training experience wanes

with each disengaged participant

Typical Reaction of Trainees

• Excited to learn a new clinical intervention

• Enthusiastic and committed to trying it with clients

• Although there wasn't much time for skill-building during the session, they have their notes and want to try it out

Reality Returns

• Then they get back to the site– to the routine– to the caseload– to the demands by courts– to UR or required contacts– to the supervisor and co-workers who didn't

take the training– to the clients who aren't prepared for

something new or different

Not Prepared

• Notes now seem incomplete

• Can’t recall details from the training

• Enthusiasm and new knowledge begin to fade

Adopting new

practices in the context

of everyday work is

difficult and frustrating

Do One-Shot Trainings Work?

• 15 hour training on MI

• Pre-training baseline audiotape

• Helpful Responses Questionnaire

Self-Report

• Participants over-inflated their skills of using MI after the training

• Use of MI declined with time following the training (about 50%)

• Skills were about ¼ of proficient

Maximizing Gains from Classroom Training

• Use of knowledge-based pre/post tests

• Use of knowledge-based proficiency tests

• Use of skill-based rating upon completion of training

• Mechanism for use of data– Rated competent or continued development

until reach competent

Role of Trainer

• Training is a means to an end – not an end in itself

• Trainer is Performance Consultant• Trainer partners with the learner• Link to business need• Proactive and reactive • Front end assessment and evaluation of

performance

Evaluation

• Knowledge Test – pre/post

• Competency – skill-building sessions– Skill check off– Structure through policy and procedure– Program Integrity Evaluation– CQI or outcome evaluation

Supervisory Strategies for Enhancing Training Transfer

• Communicate expectations prior to training

• Demonstrate involvement of training content – integrated (clinical and staff meetings, paperwork, etc)

• Hold learner accountable for applying content in work (rewards/sanctions)

Supervisory Strategies for Enhancing Training Transfer

• Demonstrate and Model the skills

• Provide learning and practice opportunities

• Integrate learning objectives into performance appraisal

• Observe and provide feedback and coaching

• Booster training sessions

What would the Supervisor need to be able to do those items?

Small Group Discussion

Remember Two Part Process

Change Management

And

Skill Development

Areas to Address

• Skill Deficit

• Resistance

• Both

Role of Clinical Supervisor

• Safety

• Quality

• Effectiveness

• Compliance

Oh yeah, and . . . .Oversee servicesAdministrative TasksDaily Crisis ManagementPrepare ReportsManage CaseloadProvide coverageHire StaffAll other duties no one else wants to do

Evidence Supporting Supervision

• Conditions– Manual Only– 14 hour Workshop– Workshop + Feedback– Workshop + Coaching– Workshop + Feedback + Coaching

All had initial Skill Acquisition

Miller, et. al., A Randomized Trial of Methodsto Help Clinicians Learn Motivational Interviewing.Journal of Consulting and Clinical Psychology (2004)

Four Months Later

• Conditions– Manual Only– 14 hour Workshop– Workshop + Feedback– Workshop + Coaching– Workshop + Feedback + Coaching

Could not Detect Who had Training

Only condition that maintainedbenefit after 4 Months

CBT Study• Conditions

– Manual Only

– Manual + Web-based Training (40 hours)

– Manual + Training + Supervision (observation and feedback)

Sholomskas, et. al., We don’t Train in Vain: ThreeStrategies of Training Clinicians in CBT .Journal of Consulting and Clinical Psychology (2005)

CBT Study

• Conditions

– Manual Only – No Transfer

– Manual + Web-based Training (40 hours) - Modest Transfer

– Manual + Training + Supervision (observation and feedback) - Proficient

Sholomskas, et. al., 2005

“Face to face training followed by supervision may be essential for effective technology transfer and raises questions about whether practitioners should feel competent to administer an empirically-

supported treatment on the basis of reading a manual alone.”

Current State

• What is your current assessment of staff in providing evidence-based practices?

• What are the challenges in achieving staff proficiency?

• What changes could facilitate improvement in clinical supervision?

Observed Contacts

• On a scale of 1 - 10

– How important is it for the clinical supervisor to have direct observation to effectively provide supervision?

– If you rated high – why?

– If you rated low – why not?

What do youthink your staff

would say?

Behind Closed Doors

Ever Makeyou Nervous???

Types of Interventions Used

Score of 4 Considered Proficient

Types of InterventionsNever Used

Types of InterventionsRarely Used

What Were They Doing?

Those not Trained in Model

Significantly More “Chat”

What was the Informal Conversation?

• Common Experiences

• Opinions not related to Treatment

• Current Events/News

• Personal Feelings about Client

• Work-Related Problems

• Professional Background

• Other

Without Observation

• No chance to reinforce good work

• No correcting mistakes or inconsistencies

• No provision for gaps in skill

• No assurance of fidelity with model

Practice

Practice

• Integrate and incorporate the new knowledge or skill

• Use a variety of methods and scenarios

• Providing coaching and feedback

Practice - Goals

• Integrate and incorporate new skill in direct practice situations

• Encourage transfer of skill to variety of situations

Techniques

• Observations of current skill

• Together identify skill gaps

• Consider team or pair learning partnerships

• Contextual learning experiences

• Problem-solving exercises

Techniques

• Develop activities for the staff to process learning

• Hands-on trial/feedback/reflection/retrial

• Real-world simulations

• Individual reflection and articulation

Techniques

• Partner and team-based discussion• Collaborative teaching and review• Skill-building activities• Teachbacks• Interview staff member as if s/he was client• Best and worst critique• Tell a story that illustrates how skill was effective• Memory or job aids

Strategies

• Role play• Trial and error• Microskills training • Written vignettes – if client says . . . you

respond _______.• Role play and make common mistakes –

have them identify mistakes and how to replace

Strategies

• Clinician develops a pictogram or flow chart of the methods

• Imagery• Cards with vignettes – what would you

say? Or cards with skills – respond to the vignette

• Project client responses• Group brain role play

Performance

Performance

• Help apply and extend their new knowledge or skill to the job

• Create integrated skill

• Continuous improvement of performance

Performance - Goals

• Make sure the learning sticks and is applied successfully

• Make sure the skill is applied in appropriate situations

• Continuous improvement

• Is the skill having the desired impact or outcome?

Techniques

• Immediate real-world application• Creating and executing plan for clients• Follow-through reinforcement activities• On-going coaching• Performance evaluation and feedback• Peer support activities• Supportive organizational and environmental

changes

Threats to Transfer of Learning

• No immediate need to apply the skill

• No support system for reinforcing the learning on the job

• A culture or work setting that is antithetical to the new learning

• No rewards for applying the skill

• No consequences for not applying the skill

• No time to integrate the new skill

Early Attempts

• What worked well? What would have made it better?

• What were the problems with implementation and/or strategy

• How would you do it different?• What do you need to improve your

performance?

Reinforcement Strategies

• Buddy System

• Rewards and Consequences– Evaluation system – objective measure– Self-evaluation– Review to see if the benefits are realized

Current Measures

How do we determine performance?

Specific Measures - Examples

• Uses EBP language during interactions

• Goals address criminogenic needs

• Avoids power struggles with clients

• Consistently applies protocol for rewards and sanctions

• Helps client identify thinking errors and correctives in neutral manner

Fidelity

Fidelity

Implementing the intervention as closely as possible to the way it was designed and

delivered during the research stage

Drift

• Accidental adaptation can pose significant problems

• Too much adaptation might decrease an intervention's effectiveness

Evaluation

• Effective programs utilize Continuous Quality Improvement

• Assess clinical targets and outcome data

The Role of QA/QI in Community Corrections (based on UC Halfway House and CBCF study)

Every major study shows strong relationship between program

integrity and recidivism

INTEGRITY

RECIDIVISM

Program Integrity and Treatment Effect in Residential Programs

As Scores for Integrity RiseProgram Lowers Recidivism

Therapist Competency Ratings and Recidivism

Having Unskilled CliniciansWorst than Providing No Treatment

Rating Systems

Individual adherence to EBP

Fidelity to Model

Not a discussion of

RIGHT or WRONG

Identifying what is

CONSISTENT OR INCONSISTENT

Process of Supervision

(1) Direct observation of treatment sessions

(2) Structured feedback about adherence and competence

(3) Coaching to improve implementation with proficiency

(4) Evaluate

Training and Supervision Model

• Intensive training by expert for clinicians and supervisors

• Program-based supervisors proficient – based on rating

• Staff turn in tapes or observed with feedback and coaching

Training and Supervision Model

• Continued intensive supervision until achieve proficiency rating in 3 sessions

• Ongoing individual or group supervision

• Consultant monthly contact with supervisors

Anxiety with Rating

• Emphasis on learning process – don’t tie to evaluation until a preplanned learning curve period

• Emphasis on improving rather than criticism

• Utilize model in feedback

Rating Process - Example

Motivational Interviewing – MIA:STEP

Rating Items

MI Consistent Items• MI Style• Open Ended ?’s• Affirmations• Reflections• Foster Collaboration• Motivation Change• Discrepancies• Pros/Cons• Change Planning• Client-Centered FB

MI Inconsistent Items• Unsolicited Advice• Emphasize Abstinence• Direct Confrontation• Powerlessness• Asserting Authority• Closed-Ended Questions

Adherence

• Indicate when the person demonstrated the Skill

Competence

• Higher Quality – 4 through 7

• Lower Quality – 1 through 3

Agreement About Occurrence

• Supervisor and Clinician rated similarly with Clinician rating higher

• Independent rater rated lower in most cases

Feedback

• Specificity about performance strengths and weaknesses

• Routine and formal discussions

• Opportunities for counselor self-evaluation and input

• Help counselor develop discrepancy between current interventions and goal

Coaching Format

• Start with strengths• Show them the rating sheet• Listen to tape or give specific examples• Ask clinician for area they would like

coaching • Suggest role play• Summarize• Plan for next time

Development

• Establish Standards and Expectations

• Monitor Clinical Services Regularly

• Provide Feedback

• Collaborate on Professional Development Plans

• Facilitate Knowledge and Skill Acquisition

Reduce Burnout

CLINICAL SUPERVISION

EXHAUSTIONAND

TURNOVER

NIDA study reveals that clinical

supervision was negatively

associated with emotional

exhaustion and turnover

Creating Lesson Plans

• Identify a skill-based learning goal for each supervisee

• Develop a learning plan– Preparation– Presentation– Practice– Performance

Summary and Review

• Design with the Learning Cycle– What are the four P’s?

• Appeal to the Learning Style of your learner– How can you be SAVI?

• Get the learner involved• Create a Learning Environment and Community• 30/70 Rule

30% preparation and presentation70% practice and performance

• Be flexible to the needs of the learner

Closure

Be the change you want to see in the world

Ghandi