Coaching with Fidelity - APBSpresentation opportunityexample Can provide two examples from a...

Coaching with FidelityKaci Fleetwood, M.Ed, BCBA, LBAJodie Soracco, M.Ed, BCBA, LBA

APBS 2020 J 11

Lens of Implementation

State Level

District Level

Doesn’t fit in a box

Building Level

Identify a Buddy

• Make eye contact with someone close to you

• Tap elbows or give each other a thumbs up to confirm you’ve noticed each other

• Plan to talk to your buddy during think, pair, share prompts

5

“There was a moment in sports when employing a coach was unimaginable – and then came a time when not doing so was unimaginable. We care about results in sports, and if we care half as much about results in schools and in hospitals, we may reach the same conclusion.”

Atul Gawande

Why is Coaching Important?

• More accurate implementation fidelity scores (McIntosh et al., 2017)

• Greater improvements in outcomes (Artman-Meeker, Hermeter, & Snyder, 2014; Capella et al., 2012; Flynn, Lissy, Alicea, Tarzates, & Mckay, 2017)

• Sustainability and scale-up (Fixen & Blasé, 1993; Horner et al., 2014)

6

“You would never have a team without a

coach”-Shauna Bake (Pershing County MTSS Coordinator), 2019

7

Research to Practice Gap

• Research* has demonstrated that coaching is essential for translating training into effective implementation.

• However, little explicit guidance exists on how to structure the roles and activities of external coaches who support multiple implementation sites.

*(Joyce & Showers, 2002; LaPolla & Scott, 2019; McDaniel & Bloomfield, 2019; Sugai et al., 2012)

Default Approach

• Those who have been responsible for recruiting, training, and supervising external coaches at the district, regional, or state level often resort to simply hiring candidates with the most robust coach-related experience and permit a type of “free-range coaching”.

• This results in a great deal of variability in what coaching activities are performed, how school sites are supported.

Diverse Backgrounds

10

In Nevada Coaching Positions are filled by individuals with diverse professional histories• Licensed Teachers• School Psychologists• Board Certified Behavior

Analysts• Social Workers• RN/Psychiatric Nurses• School Administrators• Others

What are the Roles of a Coach?

• “External Coaches”– Coach teams– Train content– Evaluate outcomes– Coordinate implementation– Expand LEA capacity– Other duties as assigned

SOLUTION: OUTLINE PERFORMANCE EXPECTATIONS FOR DIVERSE COACHING FUNCTIONS

Leveraging a School-wide Approach

• Expectations = professional values• Locations/Contexts= job functions• Coaching Competencies= observable behaviors

Valu

e/Ex

pect

atio

nVa

lue/

Ex

pect

atio

nVa

lue/

Expe

ctat

ion

Safe Respectful

Responsible

What expectations/

values work for your coach(es)?

What professional values apply to your context?

Insert Nevada Coach Matrix

MTSSExternalCoachExpectationMatrix

Overall(O)

CoachingTeams(C)

Training(T)

Responsible

1. Attendmeetingsandcoachtrainingswithcoordinator______

2. Self-monitortasksassignedandcompletion_____

3. Engageinformaldissemination______

1. Attendsite-teammeetings(data-driven)______

2. Collectmonthlyartifactsfromteams______3. Facilitateestablishmentofdatacollectionsystems

______4. Consultationnotescompleted______

1. Askforclarificationorassistancefor

concepts/topicswithwhichyouarenotyetfluent______

2. Schedule&presenttargetedtrainings_____

Data -

Driven

4. Reportsanddataaredeliveredtothe

stateonorbeforeduedate______

5. Usedatatoallocatetimeandresources______

5. UseTFItocollectdataanddriveactionplanning

______

6. Usevariousdatasourcesforassessment,planning,andreporting______

3. Collectevaluationsonallformal

presentations______

4. Review&reflectonfeedbackfrom:participants,districtadministrators,andcoordinators______

Professional

6. Assistdistricttobuildcapacityfor

sustainedimplementation(redefineyourroleovertime)______

7. Readprofessionalliteratureata

minimumof30min/week______8. Engageinprofessionalandethical

standards(seeCoachesCode)______

7. Advocateforsmallestchangethatwillhavethe

largestimpact______8. Buildcredibility&rapport______

5. EngageinEffectiveSpokesperson

techniqueswhilepresenting______

Bonus:PresentonMTSS/PBISatdistrict,state,andnationalforums______

Professional Values/ Expectations



Overall(O)

CoachingTeams(C)

Training(T)

Responsible










Data -

Driven





______



presentations______


Professional











Locations/Contexts

delineate job functions



Overall(O)

CoachingTeams(C)

Training(T)

Responsible










Data-

Driven





______



presentations______


Professional











Continuous Improvement

• Contexts updated annually as needed• Competencies revised annually based on feedback/field test • V1 (2015); v2 (2016); v3 (2017); v4 (2018); v5 (2019)

2015 2016 2017 2018 2019

State/OverallDistrictTeamsContent/Presentations

OverallDistrictTeamsContent/Presentations

OverallDistrictTeamsContent/Presentations

OverallDistrictTeamsIndividual

OverallCoaching TeamsTraining

TEACHING COACHING COMPETENCIES

Leveraging a School-Wide Approach

• Annual Coaches “Kick-Off” event at the beginning of each school year

• Reveal the current matrix• Use a Cool Tool format to teach each competency:

– Include rationale for competency– Include examples of that competency in observable behaviors– Include non-example of that competency

23

T1: Attend site-team meetings (data-driven)

Rationale:Training alone is a death sentence for initiatives. On-site technical assistance is key to

translating what teams have learned in training into practice.

Examples: Non-Examples:

• Attending meetings to give guidance and feedback

• Allocating your time to support teams based on their need

• Proactively scheduling time to attend their meeting (pre-arranged)

• Being the leader and running meetings for teams

• Avoiding teams who are difficult to work with

• Attending meeting(s) and not providing feedback

24

O6: Assist district to build capacity for sustained implementation (redefine your role over time)

Rationale:For an initiative to be durable and sustain, it will require multiple champions/ local experts


• Coordinating trainings for PBISMTSS across departments (not a one person training)

• Continuously working on sustainability (not as a goal, but as a value)

• Relooking at organizational structures as needed

• Advocating for schools & district needs

• Leaving training to a lone soldier• Relying on others to make the change• Operating in a silo because it’s easier• Being the superhero who does everything

for everyone always

25

T5: Use TFI to collect data and drive action planning

Rationale:Using the TFI is required of all sites who are formally identified as implementing

PBIS in Nevada. Site based activities should be in pursuit of critical implementation components of the tier they are on to move forward.


• Following up with teams after TFI administrations to help support them in action planning

• Requesting copies of team action plans

• Letting teams drift and tend to topics/activities that are far removed from components of implementation

• Not prompting sites to action plan

Notes about teaching the Competencies

• Articulating the rationale is extremely important• Allow for veteran coaches to provide supplemental

examples/non-examples and reinforce or provide corrective feedback based on what they shared to enhance discrimination of appropriate ways to engage in the competencies

• A through discussion of each competency is required. The cool tools alone are insufficient

HOW TO MEASURE COMPETENCIES AND SET CRITERIA FOR HIGH-FIDELITY PERFORMANCE

How might you measure these behaviors?

Next Challenge

Needed a way to:• Quantifiably measure the activities and job duties executed by

external coaches• Capture variance within each competency

– More than yes/no

Training(T)CoachExpectations Red Yellow Green1 Askfor

clarificationorassistanceforconcepts/topicswithwhichyouarenotyetfluent

Assistancewasneededbutnotsoughtafter

Assistancewasneededbutinconsistentlysoughtafter

Assistancewasneededand

soughtafter/notapplicable

2 Schedule&presenttargeted

trainings

Trainingwasnotscheduled/deliveredordidnotaddressthedatadriven

target

Trainingwasdeliveredbutdidnotmeettheneedsofthetopic/group

Trainingwashighquality,

mettheneedsofthegroup;topicwasrelevantandbasedon

data

3 Collectevaluationsonall

formalpresentations

Evaluationswerenotcollected

Evaluationswerepartiallycollectedand/ornotturnedin

Allevaluationswerecollectedandturnedinon

time4 Review&reflect

onfeedbackfrom:participants,district

administrators,andcoordinators

Couldnotprovideaconcreteexample

Feedbackwasreceived,butdidnotdriveaction/changeinpractices

Provided1(+)concreteexampleoffeedbackthatpromptedadaptation

5 EngageinEffective

Spokespersontechniqueswhile

presenting

Couldnotprovideanexample

Canprovideoneexamplefroma

training,meeting,orother

presentationopportunity

Canprovidetwoexamplesfroma

training,meeting,orother

presentationopportunity

Bonus PresentonMTSS/PBISat

district,state,andnationalforums

Opportunitywasavailablebutdidnot

seek

Soughtopportunitybutwasdenied

PresentedonMTSS/PBIS

CoachingTeams(C)CoachExpectations Red Yellow Green1 Attendsite-team

meetings(data-driven)

Attends1meeting

Attends2

meetingsbasedondata

Attends3meetings,atleast2foremergingor

developingsites2 Collectmonthly

artifactsfromteams

0artifactscollectedper

team

1artifactcollectedperteamforlessthan80%ofschools

2relevantartifacts

collectedperteamfor80%of

schools

3 Facilitateestablishmentofdatacollection

systems

0–4contacts

5–7contacts

8–10contacts

4 Consultationnotescompleted

Documentedlessthan50%of

TA

Documented50-79%ofTA

Documented80%ormoreof

TA

5 UseTFItocollectdataanddriveactionplanning


Provided1concreteexample

Provided2concreteexamples

6 Usevariousdatasourcesforassessment,planning,andreporting




7 Advocateforsmallestchangethatwillhavethelargestimpact




8 Buildcredibility&rapport




Overall(O)CoachExpectations Red Yellow Green

1 Attendmeetingsandcoachtrainingswithcoordinator

Attendlessthan2

Attend2-3

Attendedatleast4meetings/trainings

2 Self-monitortasksassignedandcompletion

Lessthan50%ofactivitiesacrossweeklytargetsheetsarecompleted

50-79%ofactivitiesacrossweeklytargetsheetsarecompleted

80%(+)ofactivitiesacrossweeklytargetsheetsarecompleted

3 Engageinformaldissemination

Noformaldisseminationofinformation/presentations

Newsletterpublishedlate/orlackingrelevantcontent,data,orrecognition

High-qualitynewsletterwithdata,content,andrecognition

4 Reportsanddataaredeliveredtothestateonorbeforeduedate

Dataforlessthan80%ofteams

submittedontime

Dataforallteamssubmittedlate(within1week)

Dataforallteamssubmittedontimeinrelevantmonths

5 Usedatatoallocatetimeandresources

Evidencedlessthan2weeks(50%)oftimeacrosstargetsheetexpectations

Evidenced2/4weeks(50%)oftimeacrosstargetsheetexpectations

Evidencedatleast3/4weeks(75%orgreater)andactivitiesare

distributedacrossthematrix

6 Assistdistricttobuildcapacityforsustainedimplementation(redefineyourroleovertime)

Cannotprovideaconcreteexampleforthismonth

Canprovideatleast1concreteexampleforthismonth

Canprovideatleast2concreteexamplesforthis

month

7 Readprofessionalliteratureataminimumof30min/week

Participated/readlessthan1hourofprofessional

literature/webinarsthismonth

Participated/readunder2hoursofprofessional

literature/webinarsthismonth

Readatleast2hoursof

professionalliteraturethismonthor/andattendedawebinar

8 Engageinprofessionalandethicalstandards(seeCoachesCode)

Cangivenoexamplesof

adherencetotheethicalstandardsorcoachescodeorhadaviolationofethicalstandards

Cangive1exampleofadherencetotheethicalstandardsor

coachescode

Cangive2examplesof

instanceswherethecoachescodewasusedin

coaching,training,orpresentations

PROGRESS MONITORING

Feedback Rubrics

• Monthly Feedback Session with External Coach

• Provides feedback on performance with respect to:– Expectations– Roles/Functions– Overall competency distribution

How might you use these data?

Same data, different purposes

• Individual progress monitoring tool• Provides a view of districts coaching activities in action• Measurement of adherence to coaching activities by “coaches”

– Data leveraged for advocacy when “other duties as assigned” chronically interrupts coaching activities

• At times, the data reveal district barriers to effective coaching behaviors

PAYING HOMAGE: BEHAVIOR SCIENCE, OBM

Total Performance System

• “Management Without Supervision”• Management with positive reinforcement• Reduces the need for direct supervision• Managers may spend more time:

– Planning– Coordinating– Optimizing performance

39Abernathy (2000, 2011, 2014)

Alignment with Abernathy’s TPS

40

Observable, measureable

strategic objectives

Expectation Matrix

Self-Management

Tools

Feedback Rubrics

Personal scorecards with specific

measures and goals

Feedback Scorecard

Managers assist in pinpointing ongoing opportunities and

designing improvement plans

Performance Charts

Tactical Improvement

Plan

Employees share in the organization’s

success

Performance-Based

Incentives

TACTICAL IMPROVEMENT PLAN Flowchart Pinpointing Obstacles to Maximum Performance

41

Abernathy Theory

• How we typically address poor performance– Look for problems inside the person– Managers point to characteristics like cooperativeness, dedication,

commitment, work ethic

• It is easier for managers to blame their employees than themselves– “If I only had good students, I could teach!”

42

Pinpointing Obstacles to Maximum Performance

43

Target High Opportunity

Measure

Measure Validity Issue?

Performance Improvement

Issue?

Behavior Issue?

Selection or Training Behavior Prompts Behavior

FeedbackBehavior

Consequences

Process Issue

Coach Utilization Work Methods

Adapted from Abernathy, 2000

Pinpointing Obstacles to Maximum Performance

44

Target High Opportunity

Measure

Measure Validity Issue?

Performance Improvement

Issue?

Behavior Issue?

Selection or Training Behavior Prompts Behavior

FeedbackBehavior

Consequences

Process Issue

Coach Utilization Work Methods

Adapted from Abernathy, 2000

Relevant and attainable

measures and goals

Skills and/or training

Cues, signals,

instructions, job aids, or

other antecedents

to work actions

Scorecards, coaching sessions,

etc.

Praise, recognition, feedback, criticism,

etc.

Pulled away for other

projects/activities

Work process-

es, quality

of input, budget,

etc.

LESSONS LEARNED & FUTURE DIRECTIONS

Lessons Learned

• Be open to annual revisions – Keep notes from monthly feedback sessions to drive future changes

to contexts, competencies, and especially rubric criteria

• Need to depersonalize individual performance– Many activities are confounded by system barriers outside of

coaches control

• Additional lessons

Future Directions

• Desire to investigate empirical questions– How/does coaching performance on the TPS impact:

• School outcomes (fidelity, implementation, sustainability)• District outcomes (capacity, sustainability, etc)

– Is there a threshold for maximum impact?– Are coaches utilizing this system more efficacious than those who do

not have this system? (explore beyond anecdotal reports)• Analysis beyond individual progress• Additional things we want to research

What future directions would you recommend?

Questions

We appreciate you being here!

Kaci Fleetwood; [email protected]

Jodie Soracco; [email protected]

http://unr.edu

http://unr.edu

Coaching with Fidelity - APBSpresentation opportunityexample Can provide two examples from a...

Documents

Transcript of Coaching with Fidelity - APBSpresentation opportunityexample Can provide two examples from a...