Expert Panel Discussion:Measuring Fidelity as a Method for Evaluating Technical Assistance

Technical Assistance & Dissemination Program Meeting

Jill Lammert & Jill Feldman, Westat

April 28, 20151

Meet the Panelists!

Jill Lammert & Jill Feldman, Westat (Moderators)

Dean Fixsen, State Implementation and Scaling Up of Evidence Based Practices (SISEP)

Lynn Holdheide, CEEDAR Center Rob Horner, Positive Behavioral Interventions &

Supports (PBIS) Amy McCart, Schoolwide Integrated Framework for

Transformation (SWIFT) Kristin Reedy, National Center on Systemic Improvement

(NCSI)

3

Goals for the Discussion

Expose participants to a range of approaches for measuring fidelity of TA.

Pique interest among participants in creating fidelity frameworks for their TA projects.

Provide opportunities for participants to ask experts for guidance and advice.

4

Definition of Terms: Fidelity of Implementation vs. Fidelity of Intervention

Fidelity of implementation answers:– Are key components of a program being implemented as

intended?

Fidelity of intervention answers:– Did the degree of adoption of key components lead to

desired outcomes?

5

Sample Research Questions

Fidelity of Implementation– Were all elements of the technical assistance delivered by

program staff?– Were all of the mentoring services provided to related

services professionals according to the specified TA plan?

Fidelity of Intervention– To what extent did related services professionals engage

in the mentoring services with the TA team?– To what extent did the mentoring meetings cover the

specified TA topics? Were professionals satisfied with the quantity and quality of the mentoring services provided?

6

Measuring Fidelity in the Context of Evaluating TA

Measuring Fidelity

Recipients’ Use of New Knowledge

& Skills Recipients’

LearningRecipients’ Reactions

Organizational Support & Change

Ultimate Outcomes

7

Framework for Creating a Fidelity Measurement System

1. Specify the program logic model

2. Operationally define key components

3. Select data sources and measures

4. Establish fidelity thresholds

5. Calculate fidelity ratings

6. Determine how ratings will be used

8

Sample TA Logic Model

Monthly mentor/protégé meetings

Content of mentoring meetings

5 coaches, 1 for each related service profession

Length, number & frequency of coaching sessions

Content of coaching sessions

Inputs/ Strategies

Outputs/Activities/ Participation

Coaching Students’/ children’s

skills/ knowledge

increase, as measured by appropriate

assessments

Mentoring

Peer-to-Peer TA

Biweekly 1-on-1 peer meetings

Content of peer meetings

Online community of practice (CoP)

Outcomes Intermediate

Outcomes Long-term

Professionals participate in

TA in the required amounts

TA covers the specified

content

Outcomes Short-term

Professionals are satisfied with quantity & quality of

TA

Professionals apply skills & knowledge in

work with students/ children

Professionals increase

knowledge & practice skills

Consultation Length, number &

frequency of consultation meetings

Content of consultation meetings

9

Questions for Panelists

In what ways does your expert judgment guide how you think about measuring fidelity of TA? – What to measure? – What data to use? – How to approach calculating ratings? – How to decide which ratings to consider as evidence of

faithful implementation?

How do you approach measuring fidelity of TA components/activities that are intended to vary or be adapted (e.g., coaching at a range of intensity levels)?

10

Questions for Audience

Based on what you’ve heard today, what do you think will be the most challenging for your project in devising a fidelity framework?

What issues raised by the panelists seem particularly relevant for your project?

Based on what you heard today, how would you approach creating a logic model and identifying the key components for your TA project?

11

Questions?

Contact Information

Jill LammertJillLammert@westat.com

Jill Feldman

JillFeldman@westat.com ww.cippsite.org

12