PRIME - Implementation Science · PRIME was developed to help educators initiate and maintain...

Planning Realistic Implementation and Maintenance by Educators

PRIME

Lisa M. H. Sanetti

Thomas R. Kratochwill

Melissa A. Collier-Meek

Anna C. J. Long

September 2014

AcknowledgementsAuthorship

Executive Summary

Part 1: Prime OverviewChapter 1: Introduction to PRIMEChapter 2: How to Use PRIME

Part 2: Tier 1 SupportsChapter 3: Direct TrainingChapter 4: Implementation Planning

Part 3: Collect DataChapter 5: Treatment Integrity DataChapter 6: Progress MonitoringChapter 7: Implementation Beliefs Assessment Data

Part 4: Analyze ProgressChapter 8: Graphing and Interpreting

Part 5: Analyze ProgressChapter 9: Using Data to Target Implementation Supports in a Multi-Tiered Model

Part 6: Tier 2 and 3 SupportChapter 10: Role PlayChapter 11: Participant Modeling

34

6

918

2839

5064

72

79

95

108116

Contents

Chapter 12: Raising AwarenessChapter 13: Motivational ConsultingChapter 14: Performance Feedback

Appendix A: GlossaryAppendix B: Selected ReferencesAppendix C: Frequently Asked QuestionsAppendix D: PRIME Tracking FormAppendix E: General Strategy GuideAppendix F: Direct Training Strategy

Direct Training Treatment Integrity ProtocolAppendix G: Implementation Planning Strategy

Implementation Planning Treatment Integrity ProtocolAppendix H: Implementation Beliefs AssessmentAppendix I: Data-Based Decision Worksheets A–DAppendix J: Role Play Strategy

Role Play Treatment Integrity ProtocolAppendix K: Participant Modeling Strategy

Participant Modeling Treatment Integrity ProtocolAppendix L: Raising Awareness Strategy

Raising Awareness Treatment Integrity ProtocolAppendix M: Motivational Consulting Strategy

Motivational Consulting Treatment Integrity ProtocolAppendix N: Performance Feedback Strategy

Performance Feedback Treatment Integrity Protocol

124133141

149154160164166170

174177

189192195200

204207

212215

220223

227230

234

3

Acknowledgements

The development of PRIME was the result of a collective effort of a team of scholars, educators, project managers, and graduate stu-dents, that was facilitated by funding by the Institute of Education Sciences. We are grateful to the scholars and educators who contrib-uted their time and expertise to shape the development of PRIME materials and process, specifically: Sandra Chafouleas, PhD, Alan Coulter, PhD, Erin Cowell, PhD, Stephen Elliott, PhD, Susannah Everett, PhD, Elizabeth Fernandez, Nicholas Gage, PhD, Frank Gresham, PhD, Karen Loiselle, Sabina Neugebauer, PhD, Thomas Power, PhD, Caroline Racine, PhD, Ann Schulte, PhD, Dale Schunk, PhD, and Ralf Schwarzer, PhD. Further, we are indebted to grad-uate students, as well as the school districts and teachers who allowed us into their classrooms. We would like to thank Lindsey Voskowskey for her graphic design assistance. Final-ly, we are thankful that the research reported here was sup-ported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A10005 to the University of Connecticut. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

4

Authorship

Lisa M. H. Sanetti, PhD, is an Associate Professor in the Neag School of Education at the University of Connecticut. She received her doctorate in School Psychology from the University of Wiscon-sin-Madison in 2006 and joined the faculty at the University of Connecticut in 2007. She has been a licensed psychologist in Con-necticut since 2009. Her primary areas of research interest involve implementation science and translation of evidence-based strate-gies in schools. In 2012, she received the Lightner Witmer award from Division 16 of the American Psychological Association in recognition of her exceptional early career scholarship.

Thomas R. Kratochwill, PhD, is Sears-Bascom Professor at the University of Wisconsin-Madison. His research and writing interests are primarily in the area of diagnosis, assessment, and treatment of child psychopathology. Particular interests include the application of mediator-based (parent and teacher) treatments in schools for the prevention and treatment of childhood problems and training psychologists in consultation and therapy. He is the author of over 200 journal articles and book chapters and has written or edited over 30 books. He has received numerous professional awards and honors including being awarded the Lightner Witmer, Senior Scien-tist, Jack I. Bardon Distinguished Achievement, and Nadine Murphy

5

Lambert Lifetime Achievement Awards from Division 16 of the American Psychological Association (APA). He is also the recipient of the APA Distinguished Career Contributions to Education and Training of Psychologists and he received the Lifetime Achieve-ment Award from the National Register of Health Service Providers in Psychology.

Melissa A. Collier-Meek, PhD, is an Assistant Professor in the College of Education and Human Development at the University of Massachusetts, Boston. She received her Ph.D. in School Psychol-ogy from the University of Connecticut, completed her internship at the May Institute in Randolph, MA, and served as Post-Doctoral Fellow and Research Associate in the Center for Behavioral Educa-tion and Research in the Neag School of Education at the Univer-sity of Connecticut. Her research focuses on the implementation of evidence-based practices by school personnel, teams, and families.

Anna C. J. Long, PhD, is an Assistant Professor in the Depart-ment of Psychology at Louisiana State University. She completed her doctoral work in School Psychology at the University of North Carolina-Chapel Hill and worked as a Postdoctoral Fellow and Research Associate the Center for Behavioral Education and Re-search in the Department of Educational Psychology at the Uni-versity of Connecticut. Her research is focused on bridging the research-practice gap, with a primary focus in schools and a second-ary focus in mental health settings.

Several graduate students contributed to the evaluation of PRIME Implementation Supports and development of the PRIME Manual including Maggie Altschaefl, MS, Deirdre Byrne, MA, Justin Byron, MA, Lisa Dobey, MA, Lindsay Fallon, PhD, Jennifer Gallucci, PhD, Jisun (Sunny) Kim, MA, Jennifer Mills, MA, Cybeles Onuegbulem, MA, Kate Williamson, MA, and Ashley Wisniewski, MA.

AUTHORSHIP

6

Executive Summary

Evidence-based interventions are used in schools to help students meet academic and behavior goals. To be effective, educators must implement interventions as planned, a concept known as treatment integrity. The extent of implementation is linked to the student out-comes resulting from an intervention. That is, greater implementa-tion leads to improved student outcomes. However, many educators face challenges implementing interventions and treatment integrity often decreases within the first few days of implementation.

PRIME was developed to help educators initiate and maintain treatment integrity. PRIME is a system of implementation supports designed to be efficiently delivered within a multi-tiered framework during intervention implementation. School-based consultants can implement PRIME supports in an indirect service delivery approach (e.g., problem-solving consultation, coaching). PRIME supports are designed to be implemented during the intervention implementa-tion and intervention evaluation stages of indirect service delivery models. PRIME is based on research related to treatment integri-ty and an evidence-informed theory of adult behavior change from health psychology, the Health Action Process Approach (HAPA). Prevention science, behavioral theory, consultation, and coaching literatures also contribute to the PRIME model.

7

PRIME includes feasible universal implementation supports to facilitate high initial levels of treatment integrity as well as increas-ingly intense and targeted implementation supports to respond to potential decreases in treatment integrity. At Tier 1, proactive and feasible Implementation Support strategies, Direct Training and Implementation Planning, are recommended for all implementers. At Tier 2, a series of four strategies (i.e., Role Play Participant Mod-eling, Raising Awareness, and Motivating Consulting) are designed to increase intervention implementation knowledge, fluency, and/or motivation for select implementers who struggle to implement adequately after Tier 1 supports. At Tier 3, Performance Feedback is designed to increase intervention implementation of those few implementers who require ongoing, intensive implementation support.

To target implementation supports, PRIME suggests the collection of three data sources: treatment integrity data, progress monitoring data, and the Implementation Beliefs Assessment (IBA), a self-report measure of an implementer’s perceptions of the intervention and implementation. Taken together, these three data sources are used to make decisions about intervention plan modifications or imple-mentation supports.

The purpose of the PRIME model is to provide adequate support to implementers to promote the implementation of evidence-based intervention and, as such, efficiently and effectively benefit student outcomes. To maximize the impact of interventions, they must be implemented accurately and competently; toward this goal, the PRIME model offers a continuum of supports that can be feasibly and effectively delivered in schools.

EXECUTIVE SUMMARY

8

PART 1

Prime Overview

PRIME (that is, Planning Realistic Imple-mentation and Maintenance by Educators) is a system of tiered supports designed to pro-mote implementation of evidence-based interventions. To provide an overview of PRIME, this section includes two chapters. Chapter 1, Introduction to PRIME, describes the foundational basis of PRIME and ori-ents you to the PRIME Manual and other supporting materials. Chapter 2, How to Use PRIME, introduces PRIME implemen-tation supports and data sources that are described in detail in later chapters. In addition, this chapter provides an overview of the process of using PRIME to support intervention implementation within indirect service delivery (e.g., consultation, coaching).

Through reading this section, you will learn the conceptual basis of PRIME, un-derstand the organization of PRIME mate-rials, and be oriented to the major features and process of delivering PRIME.

PRIME

9

CHAPTER 1

Introduction to PRIME

What Will This Chapter Tell Me?Many educators struggle to consistently implement evidence-based

interventions that are designed to improve student outcomes. School-based consultants (e.g., school psychologists, team leaders) can help educators deliver these interventions by assessing treatment integrity and providing implementation supports when needed. To help this process, PRIME (that is, Planning Realistic Implementation and Main-tenance by Educators) was designed and evaluated as a continuum of implementation supports that can feasibly and effectively be deliv-ered in schools. This chapter introduces treatment integrity, PRIME, and the theory behind PRIME. The chapter closes with a description of the intended audience for PRIME, the skills needed to use PRIME, and the organization of the PRIME Manual. After reading this chapter, you will be able to describe treatment integrity and PRIME supports, identify the theoretical background of PRIME, and highlight who is the right person to implement PRIME in your setting.

What is Treatment Integrity?Before explaining PRIME, let’s review what treatment integrity is.

Treatment integrity can simply be described as the extent to which an intervention is implemented as planned. Researchers describe

INTRODUCTION TO PRIME

10

treatment integrity as a multi-dimensional construct including adherence, quality, and exposure. That definition means treatment integrity may incorporate different dimensions such as what inter-vention components were delivered (i.e., adherence), how interven-tion components were delivered (i.e., quality), and for how long the student received the intervention (i.e., exposure). See Chapter 5 for more detail about treatment integrity and its dimensions.

So, why is it important to support educators’ implementation of interventions? Over the past two decades, researchers and practitioners have worked to identify and adopt evidence-based interventions to support student outcomes. Though evidence-based interventions have a greater likelihood of achieving positive student outcomes (as compared to other interventions), their identification and adoption alone is not sufficient. Interventions have to be imple-mented with a high level of treatment integrity to maximize student outcomes. However, studies show that most implementers strug-gle to consistently deliver interventions as planned. In fact, most implementers don’t sustain adequate levels of treatment integrity for more than 1-10 days after an intervention begins. That means, many students are not actually receiving the evidence-based inter-ventions that educators and school teams determine are necessary for them to grow, develop, and achieve positive outcomes.

Although this relationship to student outcomes is likely the most important reason to evaluate and promote treatment integrity, there are other reasons it’s important. Evaluating treatment integ-rity is necessary to determine the functional relationship between an intervention and changes in student outcomes. That is, with treat-ment integrity data you will be able to say whether the intervention is responsible for improvement in student outcomes. This issue is particularly important within multi-tiered frameworks such as Response-to-Intervention and Positive Behavior Interventions and Supports, in which a student’s response to an evidence-based inter-vention determines the level of support he or she receives. Within


11

these frameworks, collecting treatment integrity data helps to ensure interventions are provided as designed across the tiers and, as such, decisions to increase or decrease supports are appropriate. Treatment integrity can also be important for documentation purposes. Docu-menting treatment integrity data provides a record of any adaptions to an intervention or any problems with implementation, which may inform future intervention decisions. Further, increasingly educators are expected to document treatment integrity data to demonstrate that students received interventions and supports to which they were entitled. That is, treatment integrity data help educators demonstrate accountability.

Assessing treatment integrity, making data-based decisions based on student outcome and treatment integrity data, and promoting treatment integrity levels as needed are foundational within the PRIME Model, which is discussed next.

What is PRIME?It’s clear that for interventions to optimally promote student

learning, we need strategies to support educators’ implementation of interventions. PRIME is a system of implementation supports de-signed to be efficiently delivered within a multi-tiered framework during intervention implementation (see Figure below). It includes feasible universal implementation supports to facilitate high initial levels of treatment integrity as well as increasingly intense and tar-geted implementation supports to respond to potential decreases in treatment integrity. Within PRIME, decisions to deliver implementa-tion supports are data-driven, based on treatment integrity data and progress monitoring data as well as a measure of the implementer’s perspective, the Implementation Beliefs Assessment. The specific PRIME components and process are further described in Chapter 2.


12

The development of PRIME was informed by research on treat-ment integrity and an evidence-informed theory of adult behavior change from health psychology, the Health Action Process Approach (HAPA). In addition, prevention science, behavioral theory, consulta-tion, and coaching literatures informed the organization and com-ponents of the PRIME Model. PRIME Implementation Supports have been rigorously evaluated and refined based on research results. These findings indicate that PRIME can increase educator’s delivery of evidence-based interventions and result in subsequent improve-ment in student outcomes. Further, these implementation supports are feasible within an indirect service delivery model (e.g., consulta-tion) and were described as valuable and helpful by educators.

To provide further background and context for PRIME, the follow-ing section explains the theoretical support for PRIME.

What is HAPA Theory?PRIME is based on the HAPA, a theory of adult behavior change

from the health psychology literature. Implementation of the majority of school-based interventions requires adults to commit to

STAGE 1Problem Identification

STAGE 2Problem Analysis

STAGE 4TreatmentEvaluation

STAGE 3Intervention Implementation

PERFORMANCE FEEDBACK

IMPLEMENTATIONSUPPORT STRATEGIES

DIRECT TRAINING & IMPLEMENTATION PLANNING

SUPPORTSTier 3/ Tertiary Supports

• Few implementers • Individualized, intensive, and ongoing support

Tier 2/ Secondary Supports• Select implementers• Targeted, high-intensity, and brief strategies

Tier 1/ Universal Supports• All implementers• Core, standardized intervention training and logistical planning

DATA SOURCESTreatment Integrity Data

• Evaluation of intervention adherence and quality

Progress Monitoring• Ongoing monitoring of students’ response to intervention

Implementation Beliefs Assessment• Implementer rating of self-efficacy & outcome expectations


13

behavior change. For example, to implement a behavior support plan, a teacher must remember to review behavior expectations during circle time, praise and provide a ticket when the student demonstrates appropriate behavior, and deliver a back-up reinforcer when earned. Just these three steps require the teacher to incorpo-rate a lot of new behaviors into his or her everyday routine. Thus, promoting high levels of treatment integrity can be thought of as an adult behavior change activity or process.

The HAPA model describes how adults engage in this behavior change (see Figure above). Before people change their behavior, they need to be motivated to do so. This process of developing a be-havioral intention is captured in the Motivational Phase of the HAPA model. Three variables are considered to play a role in this process: (a) perception of a problem that needs to be addressed; (b) outcome expectancies, beliefs about the positive and negative outcomes of alternative behaviors; and (c) action self-efficacy, one’s confidence in being capable of performing a difficult or novel behavior. But in-tention alone does not change behavior and the process of how


ActionSelf-Efficacy

MaitenanceSelf-Efficacy

RecoverySelf-Efficacy

OutcomeExpectancies

Perception ofa problem

BehavioralIntention

BehaviorInitiation

ActionPlanning

CopingPlanning

BehaviorMaintenance

BehaviorRecovery

MOTIVATIONAL PHASEVOLITIONAL PHASE

Action

Note: Adapted from Schwarzer (2008)

14

people actually initiate and maintain a new behavior is captured in the Volitional Phase of the HAPA model. New behaviors are pro-moted by planning, as completed through Action and Coping Plan-ning, as well as adequate maintenance and recovery self-efficacy, one’s confidence in being capable to implement over time and after disruption, respectively. Research in health psychology has demon-strated that Action and Coping Planning can facilitate initiation of a new behavior, and that individuals with higher levels of self-efficacy are more likely to sustain the behavior.

The HAPA model informed the development PRIME in several ways. First, PRIME addresses treatment integrity from the development of an intention to implement to maintenance of the implementa-tion behaviors. Second, a key hallmark of the PRIME Implementation Supports is Implementation Planning, which combines Action and Coping Planning from the HAPA model. Third, the Implementation Beliefs Assessment includes items that address the variables described in the HAPA model. Based on this assessment, treatment integrity data, and student outcome data, consultants may provide targeted implementation support to address specific areas of diffi-culty per the HAPA model.

Who Should Use the PRIME Manual?School-based consultants acting as individuals or as members of a

problem-solving team are the primary targets for delivery of PRIME Implementation Supports. We use the term “consultant” broadly to refer to any individual who uses a problem-solving model to help an implementer (e.g., teacher, parent, paraprofessional) provide evidence-based intervention or supports to a child or adolescent. A consultant could be, for example, a school psychologist, counselor, team leader, special education teacher, or instructional coach. In ad-dition, outside consultants who support educators with an indirect problem-solving approach may also find the PRIME useful.

Not every school-based consultant will be well-suited to deliver PRIME Implementation Supports. PRIME activities are designed to


15

be implemented within an indirect problem-solving framework and to facilitate the delivery of evidence-based interventions. As such, we expect that consultants who implement PRIME have expertise in both of these areas. To be more specific, a consultant who can effectively implement PRIME will be able to:

• Use an indirect service delivery approach, • Select appropriate evidence-based interventions, • Implement evidence-based interventions, and • Conduct intervention evaluation.

To gain skills and experiences in these areas, consultants will likely need didactic and applied, supervised training in consultation and evidence-based interventions. Although the PRIME Manual and PRIME Prerequisite Guides include introductions to these areas of expertise, it is not sufficient preparation for implementing PRIME. Once further training is accessed, consultants will be prepared to use the PRIME Manual to deliver Implementation Supports.

How is the PRIME Manual Organized? The PRIME Manual is divided into six parts, with each part includ-

ing specific chapters, and a section of appendices. These sections are described below.

• Part 1: PRIME Overview provides an introduction to PRIME, its key components, and how to implement PRIME.

• Part 2: Tier 1 Supports includes a description of the founda-tional PRIME Implementation Supports that can be used prior to intervention implementation to facilitate high levels of treatment integrity.

• Part 3: Collect Data explains best practices in data collection, analysis, and decision-making processes for treatment integrity, progress monitoring, and the Implementation Beliefs Assessment data.

• Part 4: Analyze Progress describes how to pull together treatment integrity, progress monitoring and Implementation Beliefs Assessment data through graphing and interpretation.


16

• Part 5: Identify Next Steps explains how to review treatment integrity and progress monitoring data to determine the current implementation situation and make data-driven decisions about next steps.

• Part 6: Tier 2 and 3 Supports describes the additional multi-tiered PRIME Implementation Supports that can be used when available data suggest treatment integrity promotion is needed.

The final section of the manual is a collection of appendices including a glossary, selected references, frequently asked questions, and the materials necessary to implement PRIME. These materials are referred to throughout the PRIME Manual and include Imple-mentation Support protocols and treatment integrity guides.

All chapters in the PRIME Manual are organized in the same for-mat. Chapters open with the section “What Will This Chapter Tell Me?” to provide a brief overview of the content and explain how it fits into the PRIME manual. Throughout the chapters, content is highlighted in boxes called “Tips for Using PRIME” and “Key PRIME Points.” These boxes describe suggestions for your consultation and points of importance and/or clarification, respectively. Chapters close with the section “What Did I Learn About PRIME?” that sum-marizes the chapter content. All subsequent chapters also include a list of key terms.

In addition to the chapters within this manual, there are several companion documents that may be useful:

• A Quick Guide provides a brief overview of the specifics of how to utilize PRIME Implementation Supports.

• Prerequisite Guides provide more background information on: -Choosing an evidence-based intervention -Problem-solving consultation • PRIME Case Examples illustrate the process and components

of PRIME through two example cases.


17

What Did I Learn About PRIME?Evidence-based interventions need to be implemented with

adequate treatment integrity to improve student outcomes. Many educators struggle to implement interventions with adequate treat-ment integrity. PRIME is a system of multi-tiered implementation supports to improve educators’ treatment integrity. The develop-ment of PRIME was informed by research on treatment integrity and a research-based theory of adult behavior change from health psychology, the Health Action Process Approach. PRIME is to be delivered by school-based consultants who have experience with indirect-service delivery and intervention selection, implementa-tion, and evaluation. The PRIME Manual includes multiple sections that introduce PRIME; describe intervention training as well as data sources and data-based decision making; describe implementation supports; and provide materials needed to deliver PRIME.

Chapter 1 Key Terms InterventionConsultantHealth Action Process ApproachImplementerMulti-tiered systems of supportPRIMETreatment integrity


18

CHAPTER 2

How to Use PRIME

What Will This Chapter Tell Me?PRIME is designed to support the process of intervention imple-

mentation. To do so, PRIME includes three levels of multi-tiered Implementation Supports that provide proactive, targeted, and ongoing treatment integrity support, respectively. You use three data sources—treatment integrity, progress monitoring, and the Implemen-tation Beliefs Assessment—to appropriately deliver these Implemen-tation Supports. The purpose of this chapter is to introduce PRIME Implementation Supports and suggested data sources. In addition, the chapter describes how these supports and data sources are ap-plied during intervention implementation. After reading this chapter, you will be able to describe PRIME Implementation Supports, identify PRIME data sources, and describe the process of using PRIME to pro-vide implementation support.

What are PRIME Implementation Supports? Within PRIME, research-based Implementation Support strat-

egies are organized into three levels. At Tier 1, proactive and feasible Implementation Support strategies, Direct Training and Im-plementation Planning, are recommended for all implementers. At Tier 2, a series of four strategies are designed to increase inter-

19

vention implementation knowledge, fluency, and/or motivation for select implementers who struggle to implement adequately after Tier 1 supports. At Tier 3, Performance Feedback is designed to in-crease intervention implementation of those few implementers who require ongoing, intensive implementation support. These PRIME Implementation Supports are described below.

Tier 1 Implementation Support PRIME Tier 1 Implementation Support includes Direct Training

and Implementation Planning. These supports can be delivered to-gether before intervention implementation begins to facilitate high levels of implementation. Direct Training and Implementation Plan-ning may also be delivered separately (e.g., only Implementation Planning) or during ongoing intervention implementation.

Both Direct Training and Implementation Planning involve a meeting between a consultant and the implementer. Direct Training aims to increase the implementer’s preparation for and confidence regarding implementation by teaching him or her how to deliver the intervention with high levels of treatment integrity (see Chapter 3). Specifically, a Direct Training session consists of didactic training on intervention steps, followed by modeling, practice, and feedback with the implementer. Implementation Planning also aims to in-crease preparation for implementation (see Chapter 4). This goal is not met through training; rather Implementation Planning involves logistical planning for each intervention step as well as identifying and problem-solving potential barriers to implementation.

Tier 2 Implementation Support PRIME Tier 2 strategies are designed to increase the implement-

er’s intervention delivery knowledge, fluency, and/or motivation. The PRIME strategies at Tier 2 are appropriate for those implementers who struggle to deliver an intervention as planned after receiving Tier 1 implementation support and may benefit from the delivery of

HOW TO USE PRIME

20

one or more of the Implementation Support strategies. PRIME Tier 2 Implementation Support strategies include Par-

ticipant Modeling, Role Play, Raising Awareness, and Motivation-al Consulting (see Chapters 10-13). Participant Modeling involves reviewing intervention steps and then modeling, guided practice, and independent practice within the implementation context (e.g., classroom). Role Play incorporates modeling and practice of several intervention implementation scenarios outside of the implementa-tion context. Raising Awareness is a talking strategy that focuses on the implementer’s perception of the intervention, its effectiveness, and implementation. Motivational Consulting involves reviewing the student intervention goals and using Motivational Interviewing techniques such as change talk and positive regard to increase the implementer’s motivation to increase their implementation across time.

Tier 3 Implementation Support PRIME Tier 3 includes an intensive and ongoing implementation

support, Performance Feedback (Chapter 14). This support is ap-propriate for the implementers who struggle to deliver an inter-vention and whose treatment integrity does not improve following less intensive implementation supports (Tiers 1 & 2). Performance Feedback can be delivered only once, but more than likely it may be deemed necessary on an ongoing basis (e.g., daily, weekly, when treatment integrity data fall below a criterion).

Performance Feedback involves a meeting between the consultant and implementer to discuss treatment integrity and progress mon-itoring. In addition, the consultant reviews difficult implementation steps and collaboratively problem solves to address challenges to implementation. In this way, the meeting provides feedback to the implementer about his or her treatment integrity as well as provides an opportunity for discussion and problem-solving.

HOW TO USE PRIME

21

What are the PRIME Data Sources? PRIME is designed to promote intervention implementation and

facilitate positive student outcomes. To do so, ongoing data streams from three sources – treatment integrity, progress monitoring, and the Implementation Beliefs Assessment – are needed. These data sources are described in Chapters 5 to 7 and are reviewed below. In addition, Chapter 8 explains how to graph and interpret data, while Chapter 9 describes how to make data-based decisions using these data sources.

Treatment IntegrityTreatment integrity data indicate the extent to which an inter-

vention is implemented as planned. Within the PRIME model, treat-ment integrity can provide two types of information. First, treatment integrity data can be used alongside progress-monitoring data to ensure the intervention is implemented as designed and, as such, it is appropriate to make decisions about the intervention. Second, treatment integrity data can be used to evaluate the effectiveness of PRIME Implementation Supports. That is, does the delivery of a PRIME Implementation Support (e.g., Implementation Planning, Participant Modeling, Performance Feedback) increase the imple-menter’s treatment integrity? Chapter 5 further describes treatment integrity as a construct and how to develop a treatment integrity assessment system.

Progress MonitoringProgress-monitoring data indicate how the student is progress-

ing toward the goals of the intervention. Progress-monitoring data should be collected prior to the intervention to establish a base-line as well as regularly collected during intervention implemen-tation. The type of progress-monitoring data collected will differ depending on the student intervention target and intervention goal. Progress-monitoring data, alongside treatment integrity data, are

HOW TO USE PRIME

22

critical to evaluate the intervention. Chapter 6 further describes progress-monitoring data and includes several internet resources to identify appropriate progress-monitoring tools.

Implementation Beliefs Assessment The Implementation Beliefs Assessment (IBA) is a self-report

measure to indicate an implementer’s perceptions of the interven-tion and his or her ability to implement the intervention. Research results suggest that implementers who have low outcome expecta-tions and/or self-efficacy have difficulty starting to implement and maintaining implementation over time. Within the PRIME model, IBA data are used, alongside treatment integrity data, to target appro-priate PRIME Implementation Supports. Chapter 7 further describes the IBA and its constructs, and how to administer the measure and evaluate the resulting data.

How Does PRIME Work? PRIME is to be delivered within an indirect problem-solving

model. The process is indirect because a consultant (e.g., a school psychologist, instructional coach) supports another implementer (e.g., a teacher, parent) who implements the intervention plan with the student. In other words, the consultant him- or herself does not intervene with the target student. Rather, the consultant helps the implementer identify and evaluate the student problem, determine an evidence-based intervention plan that addresses the problem, implement the intervention plan, and evaluate the intervention im-plementation and student progress.

Studies to evaluate PRIME Implementation Supports occurred within Problem-Solving Consultation (also called Behavioral Con-sultation). It may also be delivered within the context of other evidence-based, indirect problem-solving models, such as School Consultation, Conjoint Behavioral Consultation, and Instructional Coaching.

HOW TO USE PRIME

23

Four distinct phases are found across nearly all problem-solving models. In the first phase, Problem Identification, the consultant and implementer define the specific problem to be addressed, collect initial baseline data and develop an intervention goal. In the second phase, Problem Analysis, the consultant will identify an interven-tion to address the problem and reach the intervention goal. PRIME is relevant at the end of this phase and continues to the third phase, Intervention Implementation, in which the implementer delivers the plan. In Intervention Evaluation, the consultant and implementer review goal attainment, treatment integrity data, and plan effective-ness. These phases are detailed in the PRIME prerequisite guide Problem-Solving Consultation and described more specifically below.

Problem IdentificationDuring Problem Identification, the consultant and implementer

identify the area of primary concern through interviews and prog-ress monitoring data. For example, a consultant and teacher may identify that a student engages in off-task behavior during indepen-dent seatwork in math class. Further evaluation may include math curriculum-based measures (CBM) and behavior observations. The consultant and implementer will also identify an intervention goal for the student. That is, they will develop a shared description of the desired student improvement following intervention. This inter-vention goal should be linked to the progress monitoring measure identified during baseline. To continue the example, the consultant and teacher may decide on a specific level on a math CBM or a percentage of intervals with on-task behavior as a goal for the in-tervention. Progress monitoring data collection and reference to the intervention goal will continue throughout the following phases.

Problem AnalysisDuring Problem Analysis, the consultant will review the initial or

baseline data to develop a hypothesis for the current level of func-

HOW TO USE PRIME

24

tioning or behavior. As noted above, baseline data collection may include math CBM and behavior observations. To continue the ex-ample from above, the baseline data may indicate that the current level of math work is too easy for the student or that he likely en-gages in off-task behavior for peer attention. Based on the identified hypothesis, an evidence-based intervention that is appropriate for the student and context should be selected. Additional information about identifying an appropriate intervention can be found in the PRIME prerequisite guide, Selecting Evidence-Based Interventions.

PRIME activities begin during the Problem Analysis phase. Specifically, the consultant will need to identify or develop a treat-ment integrity measure and create a data collection and review plan. Relatedly, a plan to continue to deliver the progress-monitoring mea-sure and regularly provide the IBA will need to be established. These data collection procedures are described in Chapters 5, 6, and 7. In addition to developing a data collection and review plan, a consultant can provide Tier 1 Implementation Support(s) to facilitate high initial levels of treatment integrity. These supports include Direct Training and Implementation Planning, which target intervention knowledge and logistics, respectively. These Tier 1 Implementation Supports are described in Chapters 3 and 4. To document the regular data review and provision of PRIME Implementation Supports, use the PRIME Intervention Implementation Tracking Form (Appendix D).

Intervention ImplementationDuring Intervention Implementation, the implementer will begin

to deliver the intervention to support the student. PRIME activities occur throughout this phase. Specifically, treatment integrity, prog-ress monitoring, and IBA data collection will occur per the identified data collection plan. These data can then be graphed, interpreted, and used to make decisions (see Chapters 8 and 9). Continue to document these data on the PRIME Intervention Implementation

HOW TO USE PRIME

Tracking Form (Appendix D) and the Data-Based Decision Worksheet (Appendix I).

During this phase, data may indicate that additional PRIME Imple-mentation Supports are needed to help the implementer deliver the intervention with adequate treatment integrity. Chapter 9 describes how to make this data-based decision and identify an appropriate implementation support. In general, implementers will likely war-rant Tier 1 and Tier 2 Implementation Supports, possibly more than once, before receiving Performance Feedback (Tier 3). What strate-gies are provided and how frequently they are delivered depends on the specific intervention context as well as treatment integrity and IBA data. Again, Chapter 9 provides a detailed description of how to select a PRIME Implementation Support.

To appropriately provide an Implementation Support, the con-sultant can review the General Strategy Guide that includes sug-gestions for preparation, delivery, and follow up (see Appendix E). The consultant should also review the appropriate PRIME Manual chapter, use the PRIME Implementation Support protocol to guide the meeting, and complete the Implementation Support treatment integrity measure after the meeting. The Implementation Support protocols and treatment integrity measures are included in the ap-pendices.

Intervention EvaluationAfter the intervention has been implemented for a period of time,

during Intervention Evaluation, the consultant and implementer meet to discuss overall student progress and evaluate the interven-tion. At this time, it may be appropriate to change, modify, or fade the intervention depending on the student’s progress and interven-tion process.

What Did This Chapter Tell Me? The PRIME Model includes three tiers of treatment integrity sup-

port organized from:

• Least intensive, proactive supports to • Targeted implementation support strategies to • Intensive and ongoing implementation support strategy.

These Implementation Supports are applied through the use of information gained from three data sources – treatment integrity, progress monitoring, and the IBA. The process of PRIME occurs with-in an indirect, problem-solving approach that occurs in four phases. After Problem Identification and initial data collection in the Prob-lem Analysis phase, PRIME activities begin and proceed through In-tervention Implementation until Intervention Evaluation.

Chapter 2 Key Terms Direct TrainingImplementation Beliefs AssessmentImplementation PlanningIndirect Service DeliveryIntervention EvaluationIntervention ImplementationMotivational ConsultingParticipant ModelingPerformance FeedbackProblem AnalysisProblem IdentificationProgress monitoringRole PlayRaising AwarenessTier 1 Implementation SupportsTier 2 Implementation SupportsTier 3 Implementation SupportTreatment Integrity

27

To ensure that interventions are delivered with adequate treatment integrity from the onset of implementation, the founda-tional strategies, or Tier 1 Supports, intro-duced in this section can be provided prior to intervention implementation. Chapter 3 describes Direct Training, a strategy that focuses on increasing the implementer’s confidence and skills with implementation, while Chapter 4 describes Implementation Planning, a strategy designed to support logistical planning and preparation for im-plementation. The in-depth descriptions of how to plan for and complete these strat-egies found in these chapters are accom-panied by strategy protocols and treatment integrity guides found in Appendices F and G.

Tier 1 Implementation Supports can be delivered together, or only one strategy can be provided. Which Tier 1 Implementation Support should be delivered will depend on the intervention and implementer. Also, these strategies can be revisited after in-tervention implementation if treatment in-tegrity needs support (see Section 5).

Through reading this section, you will learn how to prepare and deliver Tier 1 Implementation Supports, Direct Training, and Implementation Planning.

PART 2

Tier 1 Supports

collect data

anal

yze

pro

gres

sidentify next s

teps

TIER

1

28

CHAPTER 3

Direct Training

What Will This Chapter Tell Me?Direct Training includes didactic training, consultant demonstra-

tion of the intervention, implementer practice, and feedback to facilitate an implementer’s treatment integrity. In the PRIME Model, Direct Training is considered a Tier 1 Implementation Support. The chapter describes the purpose of Direct Training and explains how to prepare for and deliver Direct Training. After reading this chapter, you will be able to describe and deliver the steps of Direct Training.

What is Direct Training?The overall purpose of Direct Training is to increase the imple-

menter’s confidence in delivering the intervention by teaching her or him foundational intervention implementation knowledge and skills. Direct Training is delivered before intervention implemen-tation and “booster” Direct Training sessions can be provided if an implementer is struggling to deliver the intervention with adequate treatment integrity. Direct Training consists of didactic training on intervention steps, followed by the consultant demonstrating the intervention, the implementer practicing the intervention, and the consultant providing feedback to the implementer. After an effective Direct Training session, the implementer has an increased under-

29

standing of the intervention, a positive experience with implemen-tation of the intervention steps, and more optimistic expectations about intervention effectiveness and implementation success.

How to Prepare for Direct TrainingTo deliver Direct Training, some preparation is necessary. Be sure

to review the general guidelines for a PRIME Implementation Sup-port meeting (see Appendix E). In particular, make sure to review the intervention plan and feel prepared to describe and implement it. Beyond the general guidelines, preparation for Direct Training includes 3 steps:

• Breaking down the intervention plan into teachable intervention steps;

• Deciding how to proceed through Direct Training steps based on treatment integrity data; and

• Preparing necessary materials.

Step 1: Breaking Down the Intervention PlanFirst, review the intervention plan. In doing so, ask yourself, “How

can the intervention plan be best taught to the implementer?” Divide the intervention plan into grouped intervention steps that will help the implementer understand (a) the overall intervention plan and (b) how the intervention steps fit into the larger compo-nents of the intervention. For a behavior support plan, it may be helpful to group intervention steps into antecedent (e.g., establish-ing and defining a classroom schedule, active supervision), teaching (e.g., teach behavior expectations, teach problem solving), and con-sequence (e.g., class-wide group contingencies, positive reinforce-ment) strategies. For other types of interventions, organize inter-vention steps into logical groups to teach to the implementer. For example, you may divide the intervention steps according to when the steps must be implemented (e.g., all steps delivered at once, different steps provided at separate times) or the theoretical links

DIRECT TRAINING

30

between intervention steps (e.g., if several steps are based on one principle, if intervention steps build on one another).

Step 2: Plan How to Complete Direct TrainingNext, decide how to proceed through Direct Training based on how

you grouped the intervention steps and any available treatment integrity data. You may decide to go through the didactic training, demonstration, practice, and feedback (Direct Training steps 2 to 10) for the entire intervention at once. Alternatively, you may choose to complete these Direct Training components for an individual inter-vention step or group of intervention steps before repeating them for the next intervention step or group of intervention steps. You may also find that it is not appropriate to demonstrate and prac-tice specific intervention steps (e.g., posting a sign). To decide how to proceed through Direct Training steps, consider how you broke the intervention into teachable intervention chunks, as well as the number and complexity of intervention steps. Use any available treatment integrity data to decide if you need to focus on specific intervention steps for practice.

Step 3: Gather MaterialsLast, gather the materials needed for the Direct Training session.

These materials include the intervention plan or a written list of the intervention steps, as well as any materials needed to implement the interventions (e.g., forms, student worksheets, rewards).

PRIME TipDirect Training involves teaching the implementer the content of

an intervention plan and the behaviors required to implement an intervention plan. The implementer will likely be learning some-thing new as part of his or her participation in Direct Training. As such, these activities could be considered a form of professional development. Therefore, consider the following best practices in

DIRECT TRAINING

31

professional development when planning and implementing Direct Training.

Professional development is best when it… •Links to both educator professional development goals and

student goals;•Involves collaboration and collective participation;•Includes training on the content and skills relevant to

evidence-based interventions;•Includes corrective feedback and planned follow-up;•Occurs with administrative support and leadership; and•Aligns with state and national standards for professional

development.

How To Deliver Direct Training The steps of Direct Training are described below. This chapter is a

more detailed companion to the Direct Training protocol and treat-ment integrity measure found in Appendix F.

Step 1: Explain the Purpose of the SessionA preview of the objectives for Direct Training will help the im-

plementer understand the purpose of the session and what his or her role will be during the session. It may be beneficial to give the implementer an actual written agenda that briefly lists each of the session objectives. Consider asking the implementer if there are any additional items he or she would like to add to the agenda. This step will help set a tone of collaboration and active engagement during the session.

A preview of objectives involves presenting an overview of how the Direct Training process works. Tell the implementer you will be reviewing the intervention plan, demonstrating the intervention steps, providing an opportunity for the implementer to practice the

DIRECT TRAINING

32

intervention steps, and giving feedback to ensure he or she masters the intervention.

Additionally, outline the goals of the session for the implementer. Goals for a Direct Training session may include increasing the im-plementers’ implementation skills and/or confidence (e.g., “Our goal for today is to make sure you feel confident implementing each step of the behavior support plan with the student every day during cir-cle time and small group reading.”). Ask the implementer what goals they might have for the meeting (e.g., “I wonder what other goals you have for the session today?”). When the implementer shares goals, validate his or her goals and explain how each goal can be met during the session or how and when you will address the goals after the session. Be sure to make connections between how each part of the Direct Training session will help meet the session goals (e.g., “Our goal is to increase your confidence during implementation and looking at the intervention in detail and having an opportunity for demonstration and practice should facilitate your comfort and confidence with implementation”).

Step 2: Didactic Intervention TrainingProvide didactic training to ensure the implementer understands

the purpose of each intervention step and how to implement each intervention step. The goal of this process is to provide an overview of the intervention plan, as well as how the intervention can support student outcomes.

Didactic training involves a review of each intervention step need-ed to implement the intervention and detailed instructions about how to carry out each step. When reviewing each step, be sure to reference and show any materials needed to implement the step. Emphasize (a) why each step is important and what it accomplishes, and (b) any relevant research support for the effectives of the step, as appropriate. In doing so, describe the rationale for the effective-ness of the intervention plan for this particular student and tar-

DIRECT TRAINING

33

get concern. Throughout the process, encourage the implementer’s active involvement by asking questions about implementation and use of the step as well as answering any questions.

Ideally, provide a written copy of the intervention plan to facil-itate didactic training. The written intervention plan will help the implementer follow along with your explanations of each step and provide them with a place to take notes on how to implement each step.

Step 3: Answer Implementer’s QuestionsFollowing didactic training, it is likely the implementer will have

questions about implementation of the intervention plan. Prompt for questions in a way that provides a supportive atmosphere. This step in the Direct Training sessions gives the implementer an oppor-tunity to ask questions or bring up any concerns that arose during the review of the intervention plan.

PRIME TipAsk the implementer questions to elicit their opinions, concerns,

or questions about the intervention or implementation. Go beyond generic questions such as “Do you have any questions?” or “Does that make sense?” that simply require a yes or no answer. Instead, have the implementer consider their impressions of the interven-tion, how the intervention will fit into their routines, and what po-tential questions or issues may arise during implementation. Below is a list of potential questions to include during Direct Training and your other consultation meetings. Use these and similar questions to guide your consultation.

• How do you see that intervention step happening in your classroom?

• Have you implemented interventions like this in the past? Tell me about your experience.

• What do you think will be easiest about implementing this

DIRECT TRAINING

34

intervention? What intervention steps might be challenging?• What intervention steps do you see the student being

particularly responsive to?• How will you describe this intervention to the student’s

parents? Can I help facilitate this discussion?

Address implementer questions and concerns as best you can based on intervention research and your experience. It is important to validate the thoughtfulness and perspective of the implementer’s questions and concerns (e.g., “Great question! I had not though of that,” “I see why you might have that concern about this intervention plan with this student.”). It is okay to ask the implementer in return what they might think about their own question or concern, (e.g., “In your experience, how have you handled that issue?”). Part of the consultation process is valuing the skills and knowledge of both the consultant and implementer. These types of responses will reinforce the collaboration and mutual trust critical to the consultant-imple-menter relationship.

Step 4: Demonstrate InterventionYour demonstration of the intervention will give the implementer

an example of how to deliver each step of the intervention plan. You may simply act out how to deliver the intervention steps as planned, but it is often beneficial to also describe what you are doing. Use intervention materials, as appropriate. During the demonstration, highlight intervention adherence (i.e., delivering the intervention as planned) and quality (i.e., how you are delivering intervention com-ponents). That is, beyond the intervention plan itself, describe and demonstrate how to deliver the intervention with quality. Specifi-cally, make sure your demonstration includes the following quality features:

• Appropriate interaction with the student (e.g., tone, non-

DIRECT TRAINING

35

verbal behavior);• Smooth and natural implementation (e.g., responding auto-

matically to the student, having materials accessible);• Appropriate timing (e.g., clear instructions given before new

activity);• Competent implementation (e.g., clearly responsive to

student’s needs).

Step 5: Engage the Implementer in Guided PracticeGuided practice of the intervention steps will help the implement-

er understand how he or she will implement each intervention step. It gives the implementer an opportunity to try new skills needed to implement the intervention in a supportive environment and ask new questions that arise with actual implementation practice. The guided practice step of the Direct Training session occurs before the independent practice step (Step 8). Guided practice allows for more supported, collaborative practice of the intervention steps with the consultant, while independent practice allows the implementer to demonstrate that he or she can implement each intervention step without support.

To engage in guided practice, have the implementer practice the intervention and provide supportive guidance as needed. Guidance may include additional explanations of intervention steps, prompts, hints, guiding questions, answering implementer questions, and en-couragement. Provide the implementer with materials to implement each step, as necessary.

PRIME TipSome implementers may be a bit uncomfortable with demon-

stration and practice so take care to make sure that the process is as comfortable and naturalistic as possible. To do so, set up your demonstration as an opportunity to share examples and use scenar-ios from the implementer’s setting that you have observed (e.g., “To implement this step, at the beginning of circle time you might say

DIRECT TRAINING

36

…” or “When you’re handing out papers, that would be a great time for you to …”). After providing the demonstration, ask a question to facilitate the practice (e.g., “So, that’s how it could look at circle time, what might you say during independent seatwork?” “So, in your own words, what do you think you’ll plan to say when you’re handing out papers?”). This way the demonstration and practice remain a natural part of the conversation around implementation.

Step 6: Provide Feedback about the Guided PracticeFollowing the guided practice, provide feedback to the implement-

er about their intervention implementation. Give specific feedback on the implementation of each step in a positive and constructive manner. Be sure to reinforce successes and correct any implemen-tation errors. Share with the implementer the particular steps he or she implemented according to plan and/or with high quality, and the steps he or she should focus on for improving adherence or quality. The implementer should feel more confident and positive about intervention implementation following appropriate and rein-forcing feedback.

Step 7: Repeat Guided Practice and Feedback, if NecessaryAfter the first round of guided practice in Step 6, check in with

the implementer about his or her knowledge about and confidence in implementing the intervention. Some implementers will benefit from repeated guided practice and feedback. If it appears the im-plementer needs more practice to feel confident implementing the intervention plan with the actual student after the first try at guided practice, repeat steps 5 and 6 until the implementer successfully im-plements each component of the intervention. Subsequent rounds of guided practice do not have to be identical to the initial practice. Consider fading the intensity of guidance to help the implementer transition successfully from guided to independent practices.

DIRECT TRAINING

37

Step 8: Implementer Engages in Independent PracticeNow transition to independent practice of the intervention steps

without guidance. This independent practice will give the imple-menter an additional opportunity to validate his or her confidence in how to implement each intervention step and ensure he or she is prepared to deliver the intervention. To engage the implementer in independent practice, ask the implementer to independently prac-tice the intervention or grouped intervention steps. Do not provide any guidance during the independent practice, but identify areas of strength during implementation as well as areas for improvement. In doing so, identify the implementer’s level of implementation adherence (i.e., the match between steps implemented and steps written in the intervention plan) and quality (i.e., the quality of the delivery of the intervention plan).

Step 9: Provide Feedback about the PracticeAsk the implementer to reflect on his or her own independent

practice. In doing so, help the implementer identify what steps he or she implemented with sufficient adherence and quality, and what intervention steps still need improvement. The self-evaluation will might also result in additional questions or concerns about the in-tervention plan. Provide constructive feedback regarding the imple-menter’s independent practice. Remember to keep feedback positive and constructive. In doing so, be sure to correct any implementation errors, but also reinforce successes.

Step 10: Repeat Independent Practice and Feedback, if necessary

As with the guided practice, examine the implementer’s skill and confidence during independent practice and response to feedback. If the implementer needs more practice to confidently deliver the intervention with adequate treatment integrity, repeat steps 8 and 9. Provide additional independent practice and feedback until the im-

DIRECT TRAINING

38

plementer successfully delivers each component of the intervention without your support.

Step 11: Close Direct TrainingThe implementer should leave the Direct Training session under-

standing the intervention steps and confident in their implementa-tion. Revisit the consultation goals and evaluate if those goals have been met through Direct Training. This session is a way to provide the implementer with a large amount of helpful and extensive in-formation about the intervention plan and its implementation. As a result, ask the implementer if he or she has any remaining ques-tions about the intervention plan or implementation. Validate the implementer’s active engagement in the meeting. End the session by providing positive feedback to the implementer about his or her participation in Direct Training.

What Did I Learn About PRIME?Direct Training prepares the implementer to implement an inter-

vention plan with high levels of treatment integrity. The consultant should select which steps of the Direct Training protocol will be used during the session to help the implementer learn the steps of the intervention plan. During the session, the consultant will lead the implementer through didactic training, a demonstration of the intervention, guided practice, independent practice, and feedback.

Chapter 3 Key TermsDidactic TrainingDirect TrainingIntervention Step

DIRECT TRAINING

39

CHAPTER 4

Implementation Planning

What Will This Chapter Tell Me?Implementation Planning helps an implementer define the lo-

gistics of delivering an intervention and identify potential barri-ers as well as related solutions for maintaining implementation. Within PRIME, Implementation Planning is a Tier 1 Implementa-tion Support that can be reviewed and updated throughout imple-mentation. This chapter provides an overview of how to work with the implementer to develop an Implementation Plan (i.e., how to create both an Action Plan and a Coping Plan). It explains the purpose of Implementation Planning, how to prepare for an Imple-mentation Planning session, and how to deliver steps of Implemen-tation Planning. After reading this chapter, you will understand the components of Implementation Planning.

What is Implementation Planning?The overall purpose of Implementation Planning is to prepare the

implementer to deliver all of the steps of the intervention and iden-tify potential barriers to ongoing implementation. Implementation Planning includes detailed logistical planning (Action Planning) and barrier identification and development of strategies to over-come those barriers (Coping Planning). An Implementation Planning

40

session consists of first reviewing the intervention and making mod-ifications to intervention steps, as necessary. Next, the implementer identifies the logistics of each intervention step as well as any nec-essary resources. Last, the implementer identifies potential barriers to implementation as well as potential strategies to address those barriers. The outcome of effective Implementation Planning is en-hanced implementer confidence in delivering the intervention with high levels of treatment integrity.

PRIME TipThe implementer can complete implementation Planning

independently. The electronic version of Implementation Planning (available at www.primeimplementation.com) walks the imple-menter through the Action Planning and Coping Planning items. Ini-tial research indicates that implementers can successfully complete the electronic version of Implementation Planning and it facilitates improved levels of treatment integrity. That said, more research has evaluated Implementation Planning within a consultation format and so if it is possible to deliver this strategy within this format, we suggest it is done collaboratively.

How to Prepare for Implementation PlanningWhen preparing to deliver Implementation Planning, be sure to

review the general guidelines for a PRIME Implementation Support meeting (see Appendix E). In particular, make sure to review the in-tervention plan. In addition, decide how to organize the intervention plan into teachable grouped intervention steps and gather the Im-plementation Planning forms and materials. If the intervention has already been grouped logically into steps (e.g., for Direct Training), use the same groupings. If not, spend some time with the interven-tion plan. Are there steps that fit together or can be grouped in a logical way? If so, plan to address those steps together, as a group, rather than in individual pieces. For more information about break-

IMPLEMENTATION PLANNING

41

ing an intervention down into teachable grouped steps, see Chapter 3 (How to Prepare for Direct Training).

Gather the Implementation Planning Protocol (Appendix G) and necessary materials. If you prefer paper and pencil, you will need four pages of Implementation Planning forms (Appendix G). If you would like to take notes electronically, access the electronic version of the Implementation Planning Form. Add the intervention steps, organized by group, to the Implementation Planning Forms. In ad-dition, bring a written list of the intervention steps to the meeting.

PRIME TipTo facilitate the meeting process, consider bringing a second copy

of the Implementation Planning Form so that the implementer can follow along. We have found this action helps implementers under-stand the meeting format in general, follow along with the Action Planning items, and in particular, keeps the meeting running at relatively quick pace.

How to Deliver Implementation Planning The steps of Implementation Planning are described below. This

description of the purpose of and how to deliver each step is a com-panion to the Implementation Planning protocol and treatment integrity measure found in Appendix G.

Step 1: Explain the Purpose of the SessionThe purpose of this step is to open the Implementation Planning

session in a collaborative manner by previewing the plan for the session, explaining Implementation Planning, and developing goals for the session. Start by providing a general description of why you are meeting with the implementer. Explain that you are going to look at intervention logistics and plan for implementation.


42

Explain to the implementer that Implementation Planning con-sists of two steps: Action Planning and Coping Planning. The goal of the Action Planning step is for the consultant and the imple-menter to look at the intervention steps and plan the details of the intervention. The goal of the Coping Planning step is to identify and problem solve barriers to implementation. Thus, the overall purpose of Implementation Planning is to facilitate the definition and adap-tation of the intervention so that it fits the implementer’s specific context and classroom.

Based on this description of Implementation Planning, work with the implementer to collaboratively develop goals for the session. Developing meeting goals will allow you to target the discussion and ensure there is a shared vision for the meeting. Goals for Implemen-tation Planning might include helping the implementer’s prepara-tion for implementation or making adaptions to the intervention to ensure it is contextually appropriate. Use your understanding of the Implementation Planning strategy to help target the implementer’s suggestions for the session goals. Once you’ve decided on shared goals, briefly explain how Implementation Planning will help meet these session goals.

Step 2: Review Student Issue and Goal To set up the discussion of the intervention, briefly review the tar-

get student issue, current data (if available), and the intervention goal. That is, you may highlight the major issues and address the current level of student progress compared to the intervention goal. This review of the student concern will ensure that the discussion of the intervention through the rest of the Implementation Planning steps is appropriately contextualized.

Step 3: Review Intervention StepsThis is the first step of Action Planning, and consists of reviewing

the list of intervention steps with the implementer (see Action Plan


43

Worksheet, Part A). If the intervention steps have been grouped to-gether, it is important for the consultant to go over how the steps were grouped and the logic behind the grouping, to make sure that the steps are divided in a way that makes sense to the implementer. If the implementer has any questions or suggestions about how the steps have been divided or the order of the intervention steps, make revisions to the list of intervention steps at this time.

Step 4: Modify Intervention Steps, if NeededIt is important to ensure that the intervention is feasible and

contextually appropriate. At this point, ask if the implementer has identified any intervention steps that may need modifications to be appropriate for his or her context or the target student. If modifications are requested, it is important to keep in mind the empirical and theoretical support for each intervention step and to ensure that any revisions follow the same theoretical logic. Have an active discussion that results in an evidence-based intervention that is well-suited for the implementer’s context. Any modifications that are made should be agreed upon by the consultant and the imple-menter, and noted on the Action Plan Worksheet: Part A.

Step 5: Identify Logistics of Each Intervention StepThe purpose of this step is to plan out the logistical aspects of the

intervention. Planning exactly what is needed to accomplish each step of the intervention plan facilitates sustained implementation of the plan. For each intervention step, the consultant and imple-menter should work together to answer the following questions:

• When? When will this step of the intervention plan be complet-ed? For steps occurring daily, this may mean a particular time of day (e.g., 9:00 a.m.) or a particular time period (e.g., during morning meeting, during fifth class period). For steps that occur only as needed, the type of behavior occurring prior to the step,


44

or the antecedent, can be described. Examples of this option may include, “when students are off task” or “when students are showing appropriate behavior.” Some steps may include perma-nent products (e.g., posting a schedule, arranging the classroom in a way that minimizes crowding and distraction) and may only need to be completed once. For steps such as these, examples may include “At the beginning of the year” or “By next Tuesday.”

• How Often? How often will this step of the intervention be completed? Examples may include “daily” or “as needed.” You may want to use this step to specify a goal for how often this step should be delivered. For example, behavior specific praise could be described provided “at least 10 times per period.” For steps involving simple permanent products, such as posting behavior expectations, “once at the beginning of the year” may be appropriate.

• For how long? For how long will this intervention step last? Examples may include a specific length of time for more dis-crete steps (e.g., 5 minutes, 20 minutes). Some steps may not have a proscribed length of time to complete, and “as needed” may be appropriate. Steps involving permanent products may last all year.

• Where? Where will this intervention step occur? Many (if not most) intervention steps will occur within the classroom. How-ever, it may be appropriate to describe a specific place in the classroom (e.g., on the rug area, on the calendar board, at the teacher’s desk) where each intervention step will occur.

• Resources needed? What resources or materials (if any) are needed to complete this step? Examples of resources may in-clude construction paper, materials for a specific lesson, or supplies for a reinforcement system.

All of these responses can be listed on the Action Plan Worksheet: Part B. If the implementer struggles to identify the logistics of im-


45

plementation, the Action Plan Sample Responses form can be used to provide examples. It can also be helpful to ask questions such as: “What would that step look like?” or “Talk me through the comple-tion of that step.” These types of questions may help elicit responses from the implementer about how each step of the intervention will be completed. It is important to ensure that the implementer’s re-sponses reflect his or her impressions of how the intervention will work in the context of his or her classroom.

Step 6: Discuss How Needed Resources May Be Obtained, if Applicable

If additional materials are needed for the intervention steps, it is important to identify how those materials can be accessed. Is the implementer able to access them? Can the consultant provide them or develop them, if necessary? Do others (e.g., an administrator or other professional) need to be approached to obtain the materials? In thinking about how to access materials, keep in mind that the quicker these resources are obtained, the faster the intervention can be implemented. If necessary resources cannot be obtained quickly, the implementation of the intervention may be delayed. Make sure to delineate what resources are needed, who is responsible, and by when the resources will be obtained on Action Plan Worksheet: Part C.

Step 7: Summarize the Action PlanReview and summarize any revisions that have been made to the

intervention plan and the logistical details that were determined for each intervention step. Once the Action Plan has been summarized, the consultant should praise the implementer for participating in the process. This step completes the Action Planning process.

Step 8: Identify Potential Barriers to ImplementationThis is the first step in the Coping Planning process. First, the con-

sultant should show the implementer the Coping Plan Worksheet


46

and ask for any major anticipated or current implementation bar-riers. Make sure the barriers are identified by the implementer and thus, reflect his or her issues with consistently delivering the in-tervention with high levels of treatment integrity. Have the imple-menter rank up to four barriers in order of importance. (1 = highest priority, 2 = second highest, etc.).

PRIME TipSome implementers may have difficulty coming up with barriers

on their own. In this case, it can be helpful to provide the imple-menter with an example of a barrier related to a different interven-tion. For example, if the implementer is implementing an academic intervention it would be appropriate to provide an example from a behavioral intervention. If the intervention targets one student, it may be appropriate to provide an example from an intervention that targets multiple students or an entire classroom.

Step 9: Identify Potential Strategies to Address Barriers Once barriers have been identified, problem-solve how to over-

come them. Ask the implementer to brainstorm ways that the in-tervention can be maintained in the presence of each of the top four barriers to implementation. If he or she struggles to identify strategies, provide suggestions or ideas in a collaborative manner. For example, if the implementer has identified lack of time as an im-plementation barrier, the consultant can work with the implementer to identify possible ways to make the intervention or intervention steps more efficient. Or, as is the case with the implementation of many behavioral interventions, the consultant can describe how the intervention may save the implementer time if implemented effec-tively. For example, a behavior intervention designed to reduce or prevent challenging behaviors will, if effective, reduce the amount of time the implementer will spend managing those behaviors. Once


47

an appropriate strategy has been identified, it should be written on the Coping Plan Worksheet.

Step 10: Summarize Coping PlanningThis is the last step in the Coping Planning process. Summarize

the strategies that have been developed to overcome the identified barriers to implementation. Next, praise the implementer for his or her participation in the Coping Planning process.

Step 11: Close the SessionTo complete this last step in the Implementation Planning pro-

cess, review the process of Implementation Planning and ask the implementer if he or she has any questions related to (a) the revi-sions made to the intervention plan (if applicable), (b) the logistics of implementation, (c) who is responsible for obtaining any needed resources and by when this will be accomplished, and (d) the iden-tified barriers and related strategies to maintain implementation. Once any and all questions have been answered, inform the imple-menter that you will provide a clean, typed copy of the Implemen-tation Plan (including the Action Plan and the Coping Plan; see Ap-pendix G for Implementation Planning Summary Report Template) as well as any resources that the consultant is responsible for ob-taining. Finally, thank the implementer for his or her time and work during the Implementation Planning process.

What Did I Learn About PRIME?Implementation Planning prepares the implementer to deliver an

intervention by outlining, in detail, the logistics of each individual step of the intervention as well as by identifying barriers to imple-mentation and strategies to overcome those barriers. At the con-clusion of the Implementation Planning session, the implementer should feel confident in delivering the intervention with high levels of treatment integrity.


48

Chapter 4 Key TermsImplementation PlanningAction PlanCoping Plan


49

PART 3

Collect Data

To determine if a student is benefiting from an evidence-based intervention, it is nec-essary to monitor student response to the intervention (i.e., progress monitoring) and evaluate implementation (i.e., treatment integrity, Implementation Beliefs Assess-ment). This section describes how to collect these data. Chapter 5 describes treatment integrity and explains how to develop a treatment integrity tool to feasibly and ac-curately monitor implementation. Chapter 6 provides an overview of how to identify an appropriate progress-monitoring tool based on an intervention goal and avail-able, research-based measures. Chapter 7 describes the Implementation Beliefs Assessment and how it can be adminis-

tered and evaluated. Through reading this section, you will

learn how to monitor student progress and intervention implementation so that you can make data-based decisions about stu-dent progress and implementation support.

collect data

anal

yze

pro

gres

s

MULTI-TIERED IMPLEMENTATION

SUPPORTS

identify next ste

ps

50

CHAPTER 5

Treatment Integrity Data

What Will This Chapter Tell Me?Treatment integrity data indicate how much of and how well an

intervention is being implemented. To decide if an intervention is impacting the target student(s), treatment integrity data should be reviewed alongside progress monitoring data. In the PRIME Mod-el, treatment integrity data are the primary source of information to decide when and how to support the implementer. This chapter defines treatment integrity and explains how a treatment integrity tool can be developed. After reading this chapter, you will be able to describe treatment integrity and create a treatment integrity tool to evaluate implementation.

What is Treatment Integrity?Treatment integrity is the extent to which an intervention is im-

plemented as planned. Basically, reviewing treatment integrity data will tell you how much of an intervention is being implemented and what—if any—intervention steps are being missed.

In general, interventions that are implemented with a greater level of treatment integrity have a higher likelihood of resulting in positive student outcomes. That statement makes sense; the more a student receives an evidence-based intervention, the more that stu-

51

dent can benefit from the intervention. If a student is only exposed to 50% or 70% of an intervention or if a student is exposed to 100% of an intervention but only once a week as opposed to daily, he or she may not be able to sufficiently benefit from the intervention. Higher levels of treatment integrity ensure the student is appropri-ately exposed to the intervention. For other reasons why treatment integrity is important see Chapter 1.

Treatment integrity is believed to be multidimensional. That means there may be several dimensions, or aspects, of treatment integrity important to evaluating intervention implementation. Available re-search tells us three dimensions seem especially important—adher-ence, quality, and exposure—and are explained below.

• Adherence is the degree to which the specific intervention steps are implemented as planned. Adherence data can be reviewed by session or by intervention step. For instance, adherence data can represent the proportion of intervention steps that were delivered during each intervention session. If a teacher deliv-ered 10 out of 11 steps of a reading intervention, the adherence dimension for that session would be 91% ([10/11] x 100 = 91%). Alternatively, adherence data can represent the percent-age of sessions during which specific intervention steps were delivered. If a reward was to be provided daily, but was only delivered on 2 days during one school week, the adherence of that specific intervention step would be 40% ([2/5] x100=40%).

• Quality refers to how well the intervention steps are imple-mented (e.g., high versus poor quality of implementation). Qual-ity ratings may include a Likert scale of operationalized defini-tions of quality or a checklist of relevant high-quality behaviors (see Step 3 below). It makes sense that simply walking through the steps of delivering an intervention but not doing so at the right time or with appropriate enthusiasm, may not be sufficient for promoting student outcomes. For example, a behavior sup-

TREATMENT INTEGRITY DATA

52

port plan could include a paraprofessional providing specific praise along with a token that can be traded in for later reward to reinforce the student engaging in “safe” behavior during tran-sitions. If the praise is delivered 45 minutes after a transition with a flat affect and no reference to the behavior expectation the student demonstrated, it is unlikely that the student will learn why he is earning tokens and later reinforcement. But if specific, contingent praise is consistently delivered with enthu-siasm immediately after the transition it will be more likely to change student behavior.

PRIME TipAdherence and quality are closely related, but reflect different di-

mensions of treatment integrity. A step must be implemented (i.e., adherence), before quality can be reviewed. However, simply because the step is implemented does not mean it is delivered with quality. If a step is not implemented, no quality rating can be provided.

•Exposure is the extent to which the student is exposed to the intervention. Specifically, exposure is related to the frequency and duration (i.e., length) of implementation. Exposure is often described as (a) the number of minutes an intervention is provided for or (b) the number of intervention sessions delivered to a stu-dent. For example, if a student only attended 3 out of 5 possible Tier 2 reading support sessions during the week, the exposure dimension of treatment integrity is 60% ([3/5] x 100 = 60%). If an-other student attended all 5 of the Tier 2 reading support sessions, but left each 30-minute sessions after 15 minutes, the exposure dimension of treatment integrity is 50% ([75min/ 150 min] x 100 =50%). We wouldn’t expect a student who was only exposed to 50-60% of the intervention to have the same outcomes as a student who was present for the full duration of all the sessions.

Adequate levels of adherence, quality, and exposure are essen-tial for an evidence-based intervention to result in positive student


53

outcomes. The aim of the PRIME Implementation Supports is to ef-fectively facilitate implementers’ sustained treatment integrity over time. To do so, the consultant must measure treatment integrity of the intervention plan. With treatment integrity data, the consultant will decide if implementation support is needed, determine what support is appropriate, and evaluate if that support is effective (Chapters 8 and 9 include a description of this process). To do so, a treatment integrity tool will need to be developed and used to eval-uate intervention implementation.

Developing a Treatment Integrity ToolTo assess intervention implementation, treatment integrity data

must be collected systematically. Some interventions, particular-ly manualized interventions, include treatment integrity measures with the intervention materials. When an intervention does not in-clude these measures or when an intervention is developed or indi-vidualized for a particular student, educators must create their own treatment integrity tools.

The development of a treatment integrity tool has 4 steps: • Identify intervention steps,• Choose an assessment method, • Select or create an appropriate data collection form, and • Develop a data collection plan.

These steps are described below.

Step 1: Identify intervention stepsTo measure the implementation of an intervention, the specific

intervention steps need to be defined. To do so, develop operation-al, measurable definitions of each intervention step. This process may be easier for some interventions than others. Review the evi-dence-based intervention and list the specific, behavioral steps nec-essary to complete the intervention.

For example, a behavior support plan likely includes antecedent,


54

teaching, and consequence strategies. The specific activities with-in these three types of strategies could be considered intervention steps. These items may include a list of intervention steps such as (a) review behavior expectations in the morning, (b) teach student how to request attention appropriately, and (c) provide specific praise when student meets the behavior expectation. The intervention step “teach student how to request attention appropriately” might be op-erationally defined as “Tell student to raise his hand if he needs sup-port; model hand raising; have the student practice raising his hand when he needs support; and provide praise and feedback.”

To check if your list of intervention steps and definitions is suffi-cient, consider whether it would be possible to observe and mea-sure each intervention step. Or ask someone unfamiliar with the intervention if they can clearly understand the list of intervention steps. If not, revise your definition to be sufficiently observable and measurable.

As you define an intervention, consider the dimensions of treat-ment integrity that are most relevant for each intervention step. In most cases, adherence, quality, and exposure are likely relevant. However, for some intervention steps, only a specific dimension is applicable. For example, if an intervention requires the implement-er to read a manualized script, a rating on intervention steps could be provided for both adherence to the script and the quality of its delivery. In addition, there could be an overall rating for exposure, or how long the intervention session lasted. A different student might involve the implementer providing a token sheet to a student week-ly; for this intervention step it’s possible that only adherence would be relevant.

Step 2: Choose an assessment method

The three options for treatment integrity assessment methods in-clude permanent product review, direct observation, and self-report. The following table describes how each of these methods can be used and some of the strengths and limitations of each method.


55

What is it? Strengths Limitations

Direct

Observation

The consultant sys-

tematically observes

the implementation of

the intervention plan

and then rates the

extent to which he or

she observes specific

intervention steps

•Appropriate for most

interventions

•Allows for measure-

ment of adherence,

quality, and exposure

•Most direct assess-

ment method

•Time intensive for

observer

•May not be possible

to observe across

entire intervention

implementation

•Implementers may

act differently when

observer is present

Permanent

Products

Review products cre-

ated naturally through

implementation to

determine the degree

to which the inter-

vention steps were

implemented

•Less likely to be

affected by implement-

er or student reactivity

•No need for an ob-

server to be present

during implementation

•No additional work for

implementer

•Not all intervention

steps result in a

permanent product

•In most cases, only

would be possible to

rate adherence

Self-Report Implementer rates the

extent of implementa-

tion of the intervention

steps on a checklist

or form throughout or

after an intervention

session

•Quick way to assess

treatment integrity

after an intervention

session

•Possible to evaluate

adherence, quality, and

exposure

•Self-report may act as

a prompt for interven-

tion implementation

•Implementer report

may not always be as

accurate

Implementation of a Tier 2 reading intervention, for example, could be evaluated through direct observation, permanent products, or self-report. For direct observation, the consultant would need to be present during the session to rate the extent of the implementer’s


56

delivery of specific intervention steps. In this case, adherence, qual-ity, and exposure would all be relevant and able to be evaluated through direct observation. The Tier 2 reading session could also be evaluated through permanent product review. In this case, the implementer would provide any materials produced during inter-vention implementation, such as any written examples modeled by the implementer and student written products. Then the consultant would review the permanent products for evidence of implementa-tion of each intervention step. For this example, it is likely that some intervention steps would not be able to be evaluated, as they did not result in permanent products. Also, adherence would be the only dimension assessed. For self-report, the consultant might develop a checklist of the intervention steps involved in the Tier 2 reading session. Then, immediately following the intervention session, the implementer would rate his or her adherence and quality and record student exposure.

When deciding among these assessment methods, the consultant and implementer may consider the particular strengths and limita-tions of the methods themselves, as well as their match to the sit-uation. Some of the specific considerations include the (a) match between the assessment method and the type of intervention being assessed, (b) resources available and feasibility of the assessment method, and (c) preferences of the implementer and consultant. The consultant may also consider the intensity of the intervention and what decisions are likely to be made based on these data (e.g., stu-dent intervention support, special education decisions, placement decisions). More intensive situations likely call for a more direct treatment integrity assessment method to be used.

PRIME TipIt may be possible and appropriate to use more than one method

of treatment integrity assessment. For example, a consultant and implementer may decide that they will use direct observation and self-report. That is, the implementer will report implementation


57

daily, but the consultant will come to observe implementation on a monthly basis.

Step 3: Create an Appropriate Data Collection FormBased on the treatment integrity assessment method chosen, the

consultant will need to develop a data collection form to rate and record the treatment integrity data. There are three parts of a treat-ment integrity data collection form: (a) a list of the intervention steps, (b) a space to rate each relevant treatment integrity dimen-sion for each intervention step, and (c) instructions and a space to calculate the percentage of implementation. These three parts of the data collection form are described below.

A. The list of intervention steps should include the operational definitions developed as a part of Step 1. The list should also in-clude relevant treatment integrity dimensions for each interven-tion step. For example, for a social skills intervention, adherence and quality are likely relevant for most intervention steps. But exposure might only be relevant for the overall implementation of the intervention. This information should be specified on the data collection form.

B. For each intervention step, an appropriate rating option must be determined for each relevant treatment integrity dimension. That is, if adherence and quality are applicable then a distinct rating is needed for both dimensions of treatment integrity. Possible rating options include checklists, Likert scales, multi-ple choice scales, and narrative response. The following table describes each of these ratings and their related strengths and limitations.


58

Ratings What is it? Strengths Limitations

Checklist Dichotomous rating of

whether step did or did

not occur

•Easy to develop, com-

plete, and summarize

•May not account for

the nuances of imple-

mentation or partial

implementation

Likert scale Range of ratings from

full implementation to

no implementation

•Relevant for all

dimensions of treatment

integrity

•Relatively easy to

develop, complete, and

summarize

•Decision rules need

to be developed about

what counts as “full”

versus “partial”

implementation

Multiple

choice

List of brief descriptions

that correspond with

different extents of

implementation

•Relevant for all di-

mensions of treatment

integrity

•Specific behavioral

markers may increase

consistency of ratings,

as opposed to more

general ratings

•May be time

consuming to develop

as intervention steps

may each require

unique descriptions




implementation

Fill in the

blank

Space for brief narrative

in response to specific

prompts/ questions

•Relevant for all

dimensions of treatment

integrity

•Flexible format

•Can account for nuances

in implementation

•May be time consum-

ing to develop and

complete




implementation

The following table illustrates examples of each of these rating formats for the intervention step “provide behavior-specific praise when the student demonstrates safe behavior during circle time.”


Ratings Adherence Quality Exposure

Checklist Checklist

When the student

demonstrated safe

behavior during circle

time, was behav-

ior-specific praise

provided?

•Yes

•No

Was the behavior-spe-

cific praise delivered

immediately with en-

thusiasm and reference

behavior expectations?

•Yes

•No

Was the student pres-

ent throughout circle

time?

•Yes

•No

Likert scale When the student

demonstrated safe

behavior during circle

time, was behav-

ior-specific praise

provided?

•Implemented as

planned

•Implemented, but

differently than plan

•Not implemented

When provided, what

was the quality of

the behavior-specific

praise?

•Excellent

•Good

•Fair

•Poor

When was the student

present during circle

time?

•Throughout

•More than 50%

•Less than 50%

•Never

Multiple choice When the student

demonstrated safe

behavior during

circle time, was be-

havior-specific praise

provided?

•Following 100%

of opportunities

•Following half or

more of opportuni-

ties

•Following less than

half of opportunities

•Not provided follow-

ing opportunities

•No opportunity

When provided, what

quality indicators of

praise were present?

•Behavior-specific

•Contingent

•Reference behavior

expectations

When was the

student present during

circle time?

•Throughout

•More than 50%

•Less than 50%

•Never

60

Fill in the blank Student demonstrated

safe behavior on ____

occasions. Behavior

specific praise provided

on ____ occasions.

Provide an example of

the praise provided:

______________________

Circle time lasted for

_____ minutes. The

student was present

for ____ minutes.

C. The last component of the data collection forms is developing instructions and including space to calculate the percentage of im-plementation. That is, from the form, you’ll need to be able to cal-culate a quantitative summary of treatment integrity for a session.

There are two types of quantitative summary scores. The inter-vention step treatment integrity refers the extent to which specific intervention steps are implemented across sessions. Session treat-ment integrity can be calculated as the level of treatment integrity for each session. This type of summary form will help to summarize treatment integrity data across time and will inform decisions to provide additional support to the implementers or modify the inter-vention plan. The information summarized for these review meet-ings can be graphed, a topic that is described in Chapter 8.

Step 4: Develop a data collection plan In the final step, the logistics of data collection are organized.

Though it may seem that simply developing the treatment integrity form, as completed in steps 1-3, is sufficient, it is important to make sure that the logistics of data collection are clear and feasible. The planning for data collection includes (a) training an individual re-sponsible for collecting the data; (b) determining the frequency of data collection; (c) establishing regular data review.

A. To ensure that the treatment integrity data are collected in an ac-curate and systematic manner as possible, training is necessary. That is, the data collector (e.g., implementer, school psycholo-


61

gist, consultant) will need to learn about the assessment method generally and the treatment integrity data collection form spe-cifically. Depending on the data collector, it might be useful to provide background information about treatment integrity and the intervention itself. This training might include Direct Train-ing (see Chapter 3 for a description of this process) as well as practice with another rater to ensure that both individuals are rating implementation similarly.

B. To decide how frequently to collect treatment integrity data, con-sider the situation and method of assessment. When reviewing the situation, consider the intensity of the intervention as well as the type of decisions that will be made based on the treatment integrity data. For interventions that have greater intensity (e.g., student is out of the classroom often, intervention requires sub-stantial resources) assess treatment integrity more frequently. Likewise, if high-stakes decisions will be made based on the data (e.g., student intervention support, special education decisions, placement decisions), treatment integrity should be assessed frequently. The method of assessment likely will impact the de-cision about how frequently to assess data also. For instance, it is generally more feasible for an implementer to use a self-re-port method following each intervention session, than a consul-tant to observe an intervention session each day. If the chosen treatment integrity assessment method is self-report, it could be completed more frequently, as compared to direct observation.

PRIME TipOnce the plan for how frequently treatment integrity data will be

assessed is developed, make sure the logistical planning is in place. For example, you may develop a calendar of treatment integrity assessment dates with reminders or print copies of the treatment integrity data forms. The specifics may vary by method. For direct


62

observation, the data collector must learn the intervention schedule to ensure he or she is present during implementation. Make sure these logistical steps are considered and completed to ensure that treatment integrity assessment is completed as planned.

C. For data-based decision making, treatment integrity data must be regularly reviewed alongside progress monitoring data. The specifics of this review are described in Chapter 8 and 9 and materials to support this process are in Appendix I. For the pur-poses of developing a treatment integrity data system, the focus of this chapter, it is necessary to plan when treatment integrity assessment data will be reviewed. As with the frequency of data collection, the frequency of the data review will depend on the situation and intensity of the decisions made based on the data. Once the frequency of data review is established, make sure rele-vant stakeholders can be present for the meeting and that prog-ress monitoring data are also available.

What Did I Learn About PRIME?Treatment integrity data indicate how much of and how well an

intervention is being implemented. It includes multiple dimensions, including adherence, quality, and exposure. The development of a treatment integrity tool involves four steps: identify intervention steps, choose an assessment method, select a data collection form, and develop a data collection plan. Through this process, you will create a treatment integrity tool(s) to gather implementation data. These tools can be subsequently used to make data-based decisions about student progress and the possible need for or effectiveness of PRIME Implementation Supports.


63

Chapter 5 Key TermsAdherenceExposureDirect ObservationIntervention Step Treatment IntegrityPermanent ProductsQualitySelf-reportSession Treatment Integrity


64

CHAPTER 6

Progress Monitoring

What Will This Chapter Tell Me?Progress monitoring is the regular collection and review of tar-

geted student data. In the PRIME Model, progress-monitoring data are critical to understand if the intervention being is sufficiently implemented and if the student is making progress as a result of the intervention. This chapter describes progress monitoring, provides resources for progress monitoring measures, and explains key parts of how to progress monitor. After reading this chapter, you will be able to identify progress-monitoring resources and will understand the process of progress monitoring. This chapter offers an introduc-tion only; additional resources will be necessary to be prepared to progress monitor the student intervention implemented within the PRIME Model.

What is Progress Monitoring?Progress monitoring is the frequent, ongoing assessment of a

student’s progress toward the goals of the intervention. Progress monitoring data should be collected for any intervention. The pri-mary purpose of progress monitoring is to document changes in student outcome(s) to measure goal attainment. If the intervention is being implemented as planned and the student outcome data

65

show an improvement, it is likely the intervention is effective. How-ever, a different situation arises when student outcome data are not improving or are improving too slowly for the student to meet their goal within the recommended time period. In these situations, prog-ress-monitoring tools can also be used to inform when it is neces-sary to modify the intervention plan and when the consultant should provide additional implementation support to the implementer (see Chapter 9 and Appendix I).

Progress Monitoring ResourcesSpecific progress monitoring tools vary by the student outcome tar-

geted in the intervention, supporting research, and what is feasible for the implementation context. Progress monitoring tools should be brief and feasible to administer on a regular basis with minimal disruption to classroom instruction. Common progress-monitoring tools for academic goals include curriculum-based measures while tools for behavior goals include direct observation and direct be-havior ratings.

Several national databases have been established on the Internet to help educators find and use progress-monitoring tools. A list of some of the databases is provided in the table below. These resources provide basic information (e.g., technical adequacy, implementation requirements) about numerous progress-monitoring tools. We sug-gest that consultants and other stakeholders use these and other ev-idence-based resources to identify appropriate progress-monitoring tools to monitor a student’s response to intervention.

PROGRESS MONITORING

66

Website Detail

www.aimsweb.com Information and resources for AIMSweb, a progress monitoring system that includes direct assessment measures and a data management system

http://dibels.uoregon.edu/ Information and resources for Dynamic Indicators of Basic Early Literacy Skills (DIBELS), brief literacy and reading fluency measures, and a data entry system

www.interventioncentral.org Information on available curriculum-based measures and other resources for imple-menting progress-monitoring systems, such as data collection forms and graphing

http://www.studentprogress.org/ Information on progress monitoring and links to progress monitoring methods; maintained by the National Center on Student Progress Monitoring, funded by the U.S. Department of Education, Office of Special Education Programs

http://www.rtinetwork.org/ Information and resources for implement-ing Response-to-Intervention, including ongoing student assessment areas of universal screening, progress monitoring, and data-based decision making

http://www.intensiveintervention.org/ Information and resources for implement Response-to-Intervention, including in-formation on progress monitoring and a progress monitoring methods chart

http://www.directbehaviorrating.org/ Information and resources on Direct Behavior Ratings (DBR), including sample measures

PROGRESS MONITORING

67

How to Progress MonitorThe follow section introduces key parts of how to progress mon-

itor, including (a) defining the issue of concern, (b) choosing an as-sessment measure, (c) collecting baseline data, (d) setting an inter-vention goal, and (e) administering the measure regularly.

Define Issue of ConcernBefore you can choose a particular assessment measure, it is nec-

essary to develop an initial definition of the issue of concern. That is, what is the specific academic or behavioral issue that needs to be addressed by an intervention? The issue of concern should be defined as specifically as possible. For instance, the specific issue of concern for a student might be “reading fluency” as opposed to sim-ply “reading.” In another example, the specific issue of concern for a classroom might be “following directions the first time during tran-sitions” as opposed to simply “transition difficulty.” Tentatively define the topography (i.e., what it looks like) of the issue of concern, using observable and measurable terms (e.g., frequency, rate of correct responding, duration, latency, and topography). To do so, interview the implementer, review previously collected products or data, and observe, if necessary. When interviewing the implementer, use ques-tions from problem-solving consultation to identify what the issue “looks like” (see Prerequisite Guide for additional information). That is, ask the implementer when the issues take place, what they look like, and what happens after the issues. Review previously collected products that provide data, such as academic screening measures, completed assignments, or report cards, to identify information that might further support the definition of the behavior. If interviews and data review are not sufficient for defining the issue of concern, complete brief observations of the student(s) to define the behavior appropriately.

PROGRESS MONITORING

68

Choose Assessment MeasureAssessment measures to progress monitor can be found on the

websites listed in the above table. Although we do not suggest particular progress monitoring methods, we suggest a number of factors to consider when selecting a progress monitoring measure. First, consider how the data will be used in a problem-solving pro-cess. Some types of assessments were designed for particular pur-poses and can only be administered at certain time intervals that would not allow for frequent measurement of a behavior or skill. Second, consider the match between the issue of concern and the dimension measured during progress monitoring (e.g., frequency, rate of correct responding, duration, latency, and topography). The method that is selected should be able to provide relevant infor-mation for the dimensions of interest on the target outcome. Third, make sure the progress-monitoring method has been previously validated with students who are similar in demographics and school context to the target student(s) who will be monitored. For instance, if the intervention is academic, whether the issue of concern is at instructional level or grade level, should also be taken into account. Fourth, account for the resources that will be available to collect progress-monitoring data with the student (e.g., time, training, cost). Remember, progress-monitoring measures should be feasible to use over the entire implementation period and have minimal interrup-tions to instructional time. Finally, progress-monitoring measures should be valid, reliable, and sensitive to changes that might result from implementation of the intervention. Technical adequacy is key to making appropriate data-based decisions.

Collect Baseline DataOnce an assessment measure for progress monitoring is identi-

fied, collect baseline data. The specific type of assessment measure may have a specific suggested number of times that baseline data should be collected. In general, between three to five data points are

PROGRESS MONITORING

69

an appropriate number of data points for baseline. Overall, it is most important that baseline data are stable. That is, the student’s per-formance is consistent across the baseline data. Consistent data will ensure you have an accurate understanding of the student’s perfor-mance and will be able to compare the baseline data to progress after the intervention is put in place.

Use these data to further refine the issue of the concern, as ap-propriate. That is, make the definition of the issue of concern precise using baseline data. For instance, instead of “reading fluency” the issue might be defined as “reading at a rate of X words per minute.” In another example, “following directions the first time during tran-sitions” could be defined as “following teacher directions within 15 seconds of their delivery during transitions.”

Set Intervention GoalsAn intervention goal is the level of student performance expected

at end of the progress-monitoring period. To set an intervention goal, the consultant and implementer collaboratively determine how the student should be performing after implementing the intervention for a period of time. What is the desired level of progress? Several pieces of information can be used to set an intervention goal in-cluding (a) the student’s baseline data; and (b) national norms for a specific measure, local norms, a criterion level of performance, a grade level goal, and/or peer comparison data, which inform what level of performance should be expected.

To set an intervention goal, identify (a) the student’s current level of performance, (b) a desired level of performance for the student, and (c) the date by which the student should reach or exceed that level of performance. The inclusion of a date for intervention goal attainment is necessary to determine the rate of improvement (i.e., an aim line) the student needs to maintain to achieve his/her goal. During progress monitoring, this rate can be compared to the stu-dent’s actual rate of progress to determine the most appropriate

PROGRESS MONITORING

70

intensity of intervention. How to graph and review the expected and actual rates of progress is described in Chapter 8.

Write the intervention goal as a measurable statement. That is, the goal should be worded as a way to measure improvement to-ward a goal level for the student. Indicate what the student will gain during this period, rather than describing what will decrease. A written intervention goal that is measurable will help with the next step in preparing for progressing monitoring, that is, selecting an as-sessment measure. Once an assessment measure has been chosen, it may be appropriate to revise and specify the intervention goal so that the goal and measure are directly linked.

Administer Assessment RegularlyOnce an intervention goal is written and a progress-monitoring

measure is selected, the measure should be regularly administered. To make sure this task is done consistently, identify the logistics and a schedule for regular data collection. Similar to treatment integrity assessment, planning for progress-monitoring data collection in-cludes (a) training an individual responsible for collecting the data; (b) determining the frequency of data collection; and (c) establishing regular data review. Make sure the person responsible for regular collecting progress-monitoring data has experience with the prog-ress-monitoring measure and is prepared to accurately administer the measure. Provide any training necessary. Decide how frequently data will be collected. As with treatment integrity assessment, it may be appropriate to more frequently monitor interventions that are intensive (e.g., student is out of the classroom often, intervention requires substantial resources). Last, create a plan for regular data review. The specifics of this review are described in Chapters 8 and 9. In particular, make sure to decide when progress-monitoring data will be reviewed. These steps should ensure that progress-monitor-ing data are collected and evaluated regularly.

PROGRESS MONITORING

71

What Did I Learn About PRIME?Progress monitoring is the frequent, ongoing assessment of a stu-

dent’s progress toward the goals of the intervention. In the PRIME Model, progress-monitoring data can be integrated with treatment integrity data to make decisions about intervention effectiveness, and when to provide additional support to the implementer. This chapter provided an introduction to progress monitoring, but edu-cators will need to consult other resources to be prepared to prog-ress monitor. The chapter included several resources for additional information, as well as described the general process for how to progress monitor.

Chapter 6 Key Terms Progress Monitoring

PROGRESS MONITORING

72

What Will This Chapter Tell Me?Implementers’ beliefs about interventions and their ability to

implement can impact their ability to deliver an intervention con-sistently. Within the PRIME model, the Implementation Beliefs As-sessment (IBA) can be used to evaluate implementer’s beliefs and target appropriate PRIME Implementation Supports. The purpose of this chapter is to describe the IBA. In addition to explaining the IBA generally, this chapter explains how and when to administer the assessment as well as how to score and interpret it.

What is the Implementation Beliefs Assessment (IBA)?The IBA is a questionnaire, to be completed by the implementer, to

assess his or her perception of an intervention as well as his or her ability to implement it. To complete the IBA, the implementer rates the extent to which he or she agrees or disagrees with statements about implementation. The one-page, 19-item IBA can be found in Appendix H.

The IBA includes questions about two main areas: outcome expec-tations and self-efficacy. Outcome expectations refer to how effec-tive the implementer believes the intervention will be for the target student(s). Self-efficacy refers to the implementer’s belief in his or

CHAPTER 7

Implementation Beliefs Assessment Data

73

her ability to deliver the intervention to the student in the short- and long-term. Self-efficacy is broken into three different types that are described below. Together, outcome expectations and self-ef-ficacy are key components of the HAPA model (see Chapter 1) and have been shown to influence individuals’ ability to change their behavior. That is, if you feel confident in your ability to do something and you believe that consistently doing it will lead to a positive out-come, you are more likely to do it consistently.

The Outcome Expectations subscale includes items designed to evaluate an implementer’s beliefs about how effective the in-tervention will be in helping student reach the intervention goals. An implementer with more positive outcome expectations is more likely to implement with adequate treatment integrity. The logic is that if the implementer is engaging in an intervention that he or she believes will help the student, he or she will be more engaged with the intervention and more likely to implement it consistently. Conversely, if an implementer has low outcome expectations, he or she may be less likely to consistently implement an intervention, because he or she does not see the intervention as helpful.

Sample items from the Outcome Expectations subscale of the IBA include:

• As a result of this intervention, the student will perform better in school.

• This intervention will benefit the student.

The Self-Efficacy subscale includes items related to three types of self-efficacy that are important for implementing an intervention effectively across time:

• Implementation self-efficacy is an implementer’s confidence in performing the intervention steps. This aspect of perfor-mance is considered an important component of being ready and motivated to implement an intervention. That concept

IMPLEMENTATION BELIEFS ASSESSMENT DATA

74

makes sense; believing you can do something makes it more likely that you will do it.

• Maintenance self-efficacy is an implementer’s confidence in continuing to implement the intervention steps over time. That is, a teacher may be confident in implementing the in-tervention steps immediately, but may be less confident in her ability to continue to implement the intervention consistently over the course of the school year.

• Recovery self-efficacy is an implementer’s confidence in resuming implementation of the intervention steps after an interruption. For example, a teacher may feel confident in implementing an intervention during the fall, but may be un-sure of her ability to resume the intervention right away after the winter break. This perspective makes sense because it is easier to sustain an intervention when it is part of a routine, but more difficult to resume an intervention once that routine has been interrupted.

Sample items from the Self-Efficacy subscale of the IBA include:• Implementation Self-Efficacy: I have the ability to implement

each component of this intervention.• Maintenance Self-Efficacy: I can sustain intervention imple-

mentation.• Recovery Self-Efficacy: I am capable of resuming implementa-

tion of this intervention.

Used in conjunction with treatment integrity data, the results of the IBA provide information that can be used to determine if an implementer might benefit from a support strategy and, if so, which strategies may be most effective (see Chapter 9)

How to Administer the IBAThe IBA should be administered several times throughout imple-

mentation. At least, it is recommended that the IBA be administered


75

at the following time points: • Before providing PRIME Implementation Supports to target

strategy selection (see Chapter 9);• Immediately following the delivery of any PRIME

Implementation Supports;• Two to four weeks after the delivery of any PRIME

Implementation Supports.

PRIME TipHave the implementer complete the IBA several times through-

out treatment implementation to provide useful information about how the implementer is feeling about both the effectiveness of the intervention and his or her confidence in delivering the interven-tion. The results can then guide you when determining whether an Implementation Support Strategy is necessary and in choosing the most effective strategy to use.

Multiple administrations of the IBA will ensure that you have cur-rent information about the implementer’s beliefs about implemen-tation and can therefore determine if implementation supports are necessary.

When administering the IBA for the first time, meet briefly with the implementer to describe the purpose of the IBA and how it should be completed. To describe the purpose, tell the implementer you are interested in learning about their thoughts about the intervention and their experience with its implementation. Let the implementer know that the ratings will only be used to identify the best way to support their implementation toward the larger goal of improving student outcomes, so it is best to be honest when responding to items. It is not necessary to describe the measure in detail as that may influence the implementer’s ratings. To describe how the IBA should be completed, tell the implementer to rate each of the 19 items on a 7-point Likert scale from 1= completely disagree to 7 = completely agree.


76

PRIME TipHere is a sample script can be used present the IBA:“This is called the Implementation Beliefs Assessment. It is a 19-item

survey that gives me a picture of how well you think the intervention and implementation is going at the current moment and your thoughts about it in the future. To complete the questionnaire, read each state-ment and indicate how you much you agree with it by circling the ap-propriate number. The results of the survey will help me understand the best way to support you in the implementation process.”

How to Score the IBAAs noted above, the IBA is broken down into two areas: Outcome

Expectations and Self-Efficacy, allowing for the calculation of two subscales.

To calculate the two subscale scores, average the scores of the items on each scale. The Outcome Expectations subscale includes 4 items, while the Self-Efficacy subscale includes 15 items. For each subscale, add the scores for each item together and divide by the number of items within the subscale to get the mean score for that subscale. See Appendix H for IBA Worksheet.

Once mean subscale scores have been obtained, the IBA can be used, along with the treatment integrity data, to identify if PRIME Implementation Support is needed and, if so, which support will be most useful to improve implementation. This process is described in detail in Chapter 9.

What Did I Learn About PRIME?The IBA is a tool that can be used to evaluate how confident an

implementer feels with regard to the effectiveness of an inter-vention (i.e., Outcome Expectations subscale) as well as his or her confidence in his or her ability to implement that intervention (i.e., Self-Efficacy subscale). The IBA is a brief questionnaire that can be administered multiple times throughout the implementation of an


77

intervention. The results of the IBA can then be used, in part, to determine whether the implementer would benefit from an Imple-mentation Support Strategy and which Implementation Support Strategy would be most useful.

Chapter 7 Key Terms Implementation Beliefs AssessmentImplementation Self-EfficacyMaintenance Self-EfficacyOutcome ExpectationsRecovery Self-EfficacySelf-Efficacy


78

PART 4

Analyze Progress

During intervention implementation, prog-ress monitoring and implementation data are collected to evaluate student progress and determine if implementation support is necessary. The link between collecting data and making data-based decisions is data analysis. The chapter in this section describes how to graph and evaluate treat-ment integrity, progress monitoring, and IBA data.

Through reading this section, you will learn how to graph and then interpret prog-ress monitoring and implementation data.

collect data

anal

yze

prog

ress


SUPPORTS

identify next ste

ps

79

CHAPTER 8

Graphing and Interpreting

What Will This Chapter Tell Me?Data are only useful if they are used to make decisions. Within the

PRIME Model, data are collected to (a) evaluate student progress, (b) determine intervention effectiveness, and (c) inform if and what PRIME Implementation Supports are needed. To make these deci-sions, treatment integrity, progress monitoring, and Implementation Beliefs Assessment (IBA) data should be graphed and interpreted. This chapter reviews what you will need to graph and interpret data and how to graph and interpret data. After reading this chapter, you will be prepared to graph treatment integrity, progress monitoring, and IBA data and interpret the graphed data.

Getting Ready for Graphing & Interpreting DataIt may sound obvious, but it worth stating; it is not possible to

graph and interpret data to make decisions without regularly col-lecting data. Chapters 5 to 7 describe the process of collecting treat-ment integrity, progress monitoring, and IBA data, respectively. The process of developing a data collection plan is detailed in Chapter 5. This plan ensures that data are regularly collected by a trained data collector and reviewed on a regular basis.

Beyond the data itself, it is necessary to have information about the intervention goal. That is, what level of improvement in student

80

outcomes is desired? The process of selecting an appropriate, da-ta-driven intervention goal for a student is described in Chapter 6. Make sure you have the intervention goal available when graphing and interpreting data.

How to Graph DataGraphing is an efficient and effective way to summarize and pres-

ent data. Data can be easily graphed using an electronic spread-sheet (e.g., Microsoft Excel, Google Docs) or plotting data using paper-and-pencil. Regardless of specific method, the process of graphing data includes three steps:

1. Organize data in a spreadsheet;2. Choose the graph type that is appropriate for the data and

questions being asked; 3. Add graph elements to facilitate interpretation.

Step 1: Organize data into a spreadsheet

To organize the data into a spreadsheet, you need to decide what questions you are interested in answering with the data. That is, what are the specific questions you are interested in during this pe-riod of data review. For example, are you interested in examining the percentage of adherence and quality by intervention session across time? Or are you more interested in what specific intervention steps are being implemented? The table below lists common questions of interest for treatment integrity, progress monitoring, and IBA data.

Data Source Question of Interest X-Axis Y-Axis

Treatment Integrity

What is the percentage of adherence per session across time?What is the percentage of quality per session across time?What intervention steps are implemented per session across time?To what extent is the student exposed to the intervention across time?

Time (i.e., dates, interventions sessions)Time (i.e., dates, interventions sessions) Intervention steps

Time (i.e., dates, in-terventions sessions)

Percentage of adherence

Percentage of quality


Amount of exposure (e.g., minutes, percentage of content received)

GRAPHING AND INTERPRETING

81

Progress Monitoring

How has the student performed on the progress monitoring measure across time?

Time (i.e., dates, in-terventions sessions

Student performance on progress monitoring mea-sure (e.g., number of words per minute, percentage of intervals with disruptive behavior)

IBA What is the implementer’s overall IBA rating across time?

Time (i.e., dates, intervention sessions)

Overall IBA score

How have the implement-er’s ratings of IBA items changed across time?

IBA items IBA item rating

For each of your questions of interest, define the x-axis and y-axis. The x-axis is horizontal on a graph, while the y-axis is vertical. Most often, the x-axis represents time (e.g., dates of assessment, interven-tion sessions, weeks, months) or another variable that is measured repeatedly over time (e.g., intervention steps, IBA items). The y-axis usually includes the percentage or extent to which the variable of interest is monitored. The x-axis and y-axis for the common ques-tions of interest are listed above.

Once you have defined your question of interest and x- and y- axes, you can organize your spreadsheet. This task can be done electroni-cally or with paper and pencil. First, list the data for the x-axis. Next, add one or more columns for the y-axis. Add the y-axis data to the column(s) making sure the data correspond to the x-axis data. For


81

Progress Monitoring






Overall IBA score






81

Progress Monitoring






Overall IBA score






80

outcomes is desired? The process of selecting an appropriate, da-ta-driven intervention goal for a student is described in Chapter 6. Make sure you have the intervention goal available when graphing and interpreting data.

How to Graph DataGraphing is an efficient and effective way to summarize and pres-

ent data. Data can be easily graphed using an electronic spread-sheet (e.g., Microsoft Excel, Google Docs) or plotting data using paper-and-pencil. Regardless of specific method, the process of graphing data includes three steps:

1. Organize data in a spreadsheet;2. Choose the graph type that is appropriate for the data and

questions being asked; 3. Add graph elements to facilitate interpretation.

Step 1: Organize data into a spreadsheet

To organize the data into a spreadsheet, you need to decide what questions you are interested in answering with the data. That is, what are the specific questions you are interested in during this pe-riod of data review. For example, are you interested in examining the percentage of adherence and quality by intervention session across time? Or are you more interested in what specific intervention steps are being implemented? The table below lists common questions of interest for treatment integrity, progress monitoring, and IBA data.

Data Source Question of Interest X-Axis Y-Axis

Treatment Integrity

What is the percentage of adherence per session across time?What is the percentage of quality per session across time?What intervention steps are implemented per session across time?To what extent is the student exposed to the intervention across time?

Time (i.e., dates, interventions sessions)Time (i.e., dates, interventions sessions) Intervention steps

Time (i.e., dates, in-terventions sessions)


Percentage of quality


Amount of exposure (e.g., minutes, percentage of content received)


(Table continued)

82

example, make sure the date the progress monitoring measure was administered correctly corresponds to the student’s score on that measure.

In this example, the question of interest is “How has the student performed on the progress monitoring measure across time?”. The x-axis represents time and a series of dates are listed in column A. The y-axis represents the student’s performance on progress-moni-toring measure; in this case, it’s words correct per minute. The num-ber of words the student reads per minute are listed (in the y-axis) with the corresponding dates (in the x-axis).

In this next example, the question of interest is “What interven-tion steps are implemented per session across time?”. The x-axis represents intervention steps so the specific interventions steps (by number) are listed in column A. The y-axis represents the adherence ratings. Each intervention step was rated as 1 (full adherence) or 0 (no adherence) and then an overall percentage of adherence per session was obtained. The numbers for each intervention step are listed for each intervention session (by date) in column B and C.


81

Progress Monitoring






Overall IBA score






82

example, make sure the date the progress monitoring measure was administered correctly corresponds to the student’s score on that measure.

In this example, the question of interest is “How has the student performed on the progress monitoring measure across time?”. The x-axis represents time and a series of dates are listed in column A. The y-axis represents the student’s performance on progress-moni-toring measure; in this case, it’s words correct per minute. The num-ber of words the student reads per minute are listed (in the y-axis) with the corresponding dates (in the x-axis).

In this next example, the question of interest is “What interven-tion steps are implemented per session across time?”. The x-axis represents intervention steps so the specific interventions steps (by number) are listed in column A. The y-axis represents the adherence ratings. Each intervention step was rated as 1 (full adherence) or 0 (no adherence) and then an overall percentage of adherence per session was obtained. The numbers for each intervention step are listed for each intervention session (by date) in column B and C.


83

Step 2: Select Graph Type Once your data are organized, select a graph type. Two types of

graphs often used are a line graph and a bar graph. Below these graphs are described along with their unique advantages, and direc-tions for how to electronically create them.

Line Graphs Line graphs are most often used to summarize data across time.

In these cases, time (i.e., dates, sessions) is plotted on the x-axis and the extent of the variable of interest (e.g., treatment integrity, prog-ress monitoring, IBA data) is plotted on the y-axis. In this way, in a line graph, the percentage of implementation, student progress, or IBA scores over time will be graphed.

A line graph is a useful and effective way to share data in a school setting. Its simplicity is its greatest asset. Line graphs are easy to understand and, as such, they can be valuable for communicating student progress to multiple stakeholders, such as teachers, parents, and even the student themselves (if appropriate). Line graphs also highlight trends (or patterns over time) that are very useful for data interpretation and analysis. How to consider trends and interpret graphs is more fully described later in the chapter. But, line graphs help facilitate this process because of their clean presentation and ease of interpretability.

This example line graph depicts the progress-monitoring data for Michael, a student receiving a Repeated Reading intervention to increase oral reading fluency. The x-axis depicts the dates the measure was administered. The y-axis includes the number of words read correctly. The data shows the change in Michael’s oral reading fluency, or words read correctly per minute, as measured by a curric-ulum-based measure across time.

84

Now it is time to make your graph. To create a line graph elec-tronically, highlight the relevant columns in your spreadsheet. Then select the “line graph” option in the chart tab. Your graph will appear on the page. We suggest choosing the “marked” line graph option so that each of the data points will be marked, but it is not necessary to do so. To create a line graph with paper and pencil, draw the x-axis and y-axis to make a blank graph. In doing so, make sure that the graph scale is large enough to account for the change in data across time and also small enough to notice changes in between data points. Then, plot each data point by pulling data from your spreadsheet. Last, draw a line to connect the data points to indicate the data path.

Bar GraphsBar graphs are a helpful way to illustrate a relative comparison

across a variable of interest. For PRIME purposes, variables of in-terest might include the (a) implementation of specific intervention steps, (b) amount of time a student has been exposed to the in-tervention across time, or (c) implementer’s ratings on specific IBA items. Bar graphs are a flexible way to display data. The x-axis may be intervention steps, IBA items, intervention sessions, or other vari-ables of interest. The y-axis will show the percentage or rating for each of the variables on the x-axis. When graphed, you can compare the differences between different intervention steps, intervention sessions, or IBA items.

Bar graphs can help find specific patterns that may not be clear when looking at a line graph. Data displayed on a bar graph, such as intervention steps, are easily comparable because the relationships among the bars and the number value corresponding to each bar are clear. These data can be discussed between the consultant and implementer to decide the best approach to increase treatment in-tegrity that will lead to improved student outcomes.

This bar graph shows an example of treatment integrity for an intervention plan that has seven steps. The x-axis represents inter-


85

vention steps, while the y-axis is the percentage of implementation across sessions. Each bar represents the proportion of intervention sessions in which a step was implemented according to the inter-vention plan.

Now it is time to make your graph. To create a bar graph elec-tronically, highlight the relevant columns in your spreadsheet. Then select the “bar graph” option in the chart tab. Your graph will appear on the page. Sometimes a bar graph will appear incorrectly (i.e., the x- and y-axes are mixed up); in this case, simply click “switch plot” to reorient your graph. To create a bar graph with paper and pencil, draw the x-axis and y-axis to make a blank graph. Then, plot each bar of your bar graph pulling data from your spreadsheet.

Step 3: Add graph elements to interpret data At this point, you have created a graph that corresponds to a par-

ticular question of interest. You can add graph elements to help fa-cilitate your interpretation of the data. Graph elements to consider including are (a) phase change lines, (b) trend lines, and (c) aim lines.

Phase Change LineA phase change line is a vertical line transposed on a graph to

indicate when the data are collected during different conditions, or phases. The most common phase change line is between baseline


86

(or initial) data collection and intervention implementation. Other phase change lines might include any modification or adaptions in the intervention, either to increase support or systematically fade support, or when treatment integrity support was provided. Phase change lines help us interpret the data accurately and know when a change in data might be expected.

To add a phase change line electronically, simply insert a “line” shape and drag it to the correct spot on your graph. To add a phase change line to a paper-and-pencil graph, draw a vertical line where the phase change occurred.

Trend LineA trend line is a visual representation of the actual rate of prog-

ress. A trend line will indicate how quickly a student is improving per the progress monitoring measure or the rate of treatment integ-rity across time. A trend line is different from a data path, which is the line that simply connects the data points. A data path indicates the changes from one data point to the next, while the trend line indicates the overall rate of progress. This addition can be helpful, for example, if a student or implementer has one day with a lower rate on the progress monitoring measure or a day with lower levels of implementation. As the trend line accounts for the overall rate of progress it will be only minimally impacted by one lower (or higher!) data point.

In the graph below, a trend line was added to indicate Michael’s


87

overall rate of progress. The addition of the trend line helps us to understand how Michael is doing and is expected to do across time.

To add a trend line electronically, click on your graph. Then se-lect “trendline” in the “Chart Layout” section. Insert a “linear trend-line” and it will appear on your graph. To add a trend line to a paper-and-pencil graph, use the Quickie Split Middle Technique suggested by White (1971). To do so, find the middle point and draw a vertical line. Then, find the middle point (horizontal) for each half and draw vertical line. Next, find the median point (vertical) for each half and draw a horizontal line. Mark with an X at the point where the horizontal and vertical lines meet. Last, draw a line through the Xs to find the trendline.

Aim LineIn the graph below, an aim line was added to indicate the level of

progress needed for Michael to reach his goal. The addition of the aim line helps us to better evaluate Michael’s reading performance data points and trend line. In this case, while his performance is improving (per the trend line), it is not sufficient per the aim line.

88

An aim line is the median baseline point and the desired inter-vention goal at the expected end date. Once theses two data points (i.e., median baseline and desired intervention goal) are identified, it can be added to a graph electronically or using paper and pencil. To add an aim line electronically, simply insert a “line” shape and drag it to the correct angle on your graph. To add a trend line to a paper-and-pencil graph, simply draw the line on your graph.

How to Interpret the DataAfter data have been graphed, it is ready for interpretation. For

PRIME, there are three primary questions of interest. For prog-ress-monitoring data, is the student making sufficient progress to meet his or her goal? For treatment integrity data, is the implemen-tation sufficient to support student outcomes? For IBA data, how does the IBA data enhance our understanding of implementation? Instructions on how to develop summary statements and address these questions are described further below after description of the basic rules of interpreting graphs.

Regardless of the specific question, to interpret the graphed data, use visual analysis. Visual analysis is a systematic way of examining a graph to identify patterns and evaluate change. Specifically, with visual analysis, you will review graphs for three components: level, trend, and variability.

Level. The level of data is the average value of the measured out-come within a condition. A change can be shown in the data when the level of outcome is different from baseline condition compared to an intervention condition. The dashed lines in the graph show the average level across the two phases.


across the two phases.

89

Trend. Trend is the pattern of change in the data over the course of the plan implementation stage, such as increasing, decreasing, or no trend in the data. Depending on the nature of the outcome vari-able, an increasing or decreasing trend will indicate improvement or worsening of a student’s problem. The baseline phase on the left represents a decrease and worsening trend. The intervention phase on the right represents an increasing and improving trend. Shifts in trend that show improvement for the student are helpful in deter-mining a change resulting from implementation of the intervention.

Variability. Variability is the spread of data above and below the data path. When a data path shows a relatively large amount of variability, it can be difficult to draw definitive conclusions about the effect of the intervention on the student outcome. A change in variability is sometimes the desired goal of an intervention. For example, an intervention may be designed to change a student’s homework completion from greatly variable rates of completion to consistently high rates. In this case, a significant decrease in vari-ability of the outcome may provide evidence for a successful inter-vention. The graph to the right shows a large amount of variability in the baseline phase and a small amount of variability in the inter-vention phase.


90

PRIME TipBeyond trend, level, and variability, visual analysis can incorporate

many more considerations to determine if the intervention has a functional relationship with the change and summarize the effect of that intervention quantitatively. Though a thorough discussion is beyond the scope of this chapter, if interested, we encourage readers to refer to the additional resources listed below for further informa-tion.Riley-Tillman, T. C., & Burns, M. (2009) Evaluating Educational Inter-

ventions: Single-case Designs for Measuring Response to Inter-vention. New York: Guilford Press.

Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single-case designs tech-nical documentation. In What Works Clearinghouse: Procedures and standards handbook (version 2.0). Retrieved from: http://ies.ed.gov/ncee/wwc/pdf/wwc_procedures_v2_standards_hand-book.pdf

Check your visual analysis skills:Visual analysis training module—www.singlecase.org

Develop a Summary Statement To interpret the graphs, describe the overall pattern of data high-

lighting the trend, level, and variability. Depending on the graph, it may be appropriate to describe the data all together or describe the data patterns before and after intervention. For example, a sum-mary statement of a progress-monitoring graph might be described as the following: “Before intervention, Johnny’s words per minute were consistently low, in the 20-29 range, [variability, level] with a flat trend [trend]. Following intervention, Johnny’s word per minute increased over time [trend] to 40 words per minute [level], and re-mained consistent [variability].

Though trend, level, and variability are most often associated with line graphs, these components can be applied to interpreting bar


91

graphs. If a line graph includes the steps of intervention implement-ed as planned, a consultant could interpret the trend across differ-ent steps, the overall level of implementation as well as specific steps, and variability across the steps. A summary statement might be “The implementer consistently delivers consequence steps more often then antecedent steps [trend, variability], though the level of both types of steps is high (i.e., above 80%) [level]”.

Summary statements that describe the trend, level, and variability overall or between phases are useful to indicate the pattern of the data. Record the data and/or summary statements on the PRIME In-tervention Implementation Tracking Form (Appendix D). These sum-mary statements will be used together to make data-based deci-sions in Chapter 9. Before integrating these data, let’s review specific considerations for describing progress-monitoring data, treatment integrity data, and IBA data.

For progress-monitoring data, the primary question of interest is, “is the student making sufficient progress to meet his or her goal?”. To answer this fundamental question, make sure your summary statement accounts for (a) intervention phases, (b) the trend line, and (c) the aim line. In this way, highlight changes to trend, level, and variability across intervention phases (e.g., did the trend or level change following intervention? Did the variability decrease?) and compare the trend line to the aim line (i.e., is the student on track to meet his or her goal?). This analysis will ensure your description of the student’s progress across time addresses the data and directly speaks to the primary question of interest.

For treatment integrity data, the overall question of interest is “is the implementation sufficient to support student outcomes?”. There is no magic number for implementation; interventions need to be implemented to unique levels (e.g., 80%, 90%) to result in changes for the student. Implementation data need to be reviewed carefully to evaluate if (a) critical intervention steps are being implement-ed regularly, and (b) the extent of implementation is sufficient for


92

the student to make adequate progress. To determine if intervention steps are being implemented regularly, develop a summary state-ment that accounts for the trend, level, and variability of treatment integrity across time as well as highlights specific intervention step treatment integrity. Consider adherence, quality, and exposure data. If PRIME Implementation Supports have been provided, describe treatment integrity before and after supports. Did implementation improve after delivery of implementation support?

Next, develop an interpretation of the implementation data that also accounts for the progress monitoring data. Compare across graphs. Are patterns clear? Does the student do better on the days when the intervention is implemented? What steps must be im-plemented for the student to do well? For some interventions this pattern may not be clear in simply one day of implementation, but rather, the data across time (e.g., if treatment integrity is high and consistent for three days, progress monitoring data increase). Use this understanding of the relationship between progress monitoring data and treatment integrity to inform your interpretation of the treatment integrity data.

For IBA data, the primary question of interest is “how does the IBA data enhance our understanding of implementation?”. As described in Chapter 9, consider overall scores on the two subscales, Outcome Expectations and Self-Efficacy, as well as ratings of individual items. If the IBA has been delivered on multiple occasions, evaluate the change over time in consideration of trend, level, and variability. Once you have developed an overall summary statement of the IBA data, consider whether the IBA data is consistent with treatment in-tegrity data and how it might enhance our interpretation of the im-plementer. Does the implementer consistently skip some interven-tion steps and have low-self efficacy? Does the implementer have overall high self-efficacy, but inconsistent treatment integrity and low outcome expectations? Integrate your interpretation of these two data sources to enhance your understanding of implementation.


93

What Did I Learn About PRIME?A graph can facilitate decision-making during implementation.

With a graph, the level and rate of progress is clear. In the PRIME Model, graphing progress-monitoring and treatment integrity data can help a consultant evaluate the intervention and determine ap-propriate PRIME Implementation Supports. Graphing itself includes three parts (a) developing a spreadsheet, (b) selecting a graph type, and (c) adding graph elements. Next, interpret the graph highlight-ing trend, level, and variability as well as unique considerations for progress monitoring, treatment integrity, and IBA graphs. These graphs and interpretations are necessary for making data-based de-cisions (see Chapter 9).

Chapter 8 Key Terms Aim LineBar GraphData PathData PointLevelLine GraphPhase Change LineSummary StatementTrendTrend Line VariabilityX-axisY-axis


94

After collecting data and analyzing prog-ress, student progress-monitoring and implementation data should be pulled to-gether determine if the student is making sufficient progress toward their interven-tion goal and, if not, identify the appropri-ate next step. The chapter in this section describes how to evaluate student and im-plementation data sources together, make one of four data-based decisions, and com-plete associated action steps.

Through reading this section, you will learn how to identify the next step for the intervention and implementation.

PART 5

Identify Next Steps

collect data

anal

yze

pro

gres

s


SUPPORTS

identify next ste

ps

CHAPTER 9

Using Date to Target Implementation Supports in a Multi-Tiered Model

What Will This Chapter Tell Me?Previous PRIME Manual chapters have described how to collect and

graph implementation and student data. This chapter describes how to integrate treatment integrity, student progress-monitoring, and Implementation Beliefs Assessment (IBA) data to make decisions in the PRIME Model. After reading this chapter, you will be prepared to integrate these data sources to make appropriate data-based deci-sions and, if treatment integrity levels are low, decreasing, or highly variable, identify an appropriate PRIME Implementation Support.

What is Data-Based Decision Making in PRIME?PRIME is designed to support student outcomes through a focus

on intervention implementation. As such, PRIME data-based deci-sions involve reviewing implementation data (i.e., treatment in-tegrity and IBA data) alongside progress-monitoring data to deter-mine next steps for the intervention and implementation supports. Graphing and interpreting data are critical first steps for making data-based decisions (see Chapter 8). Next, treatment integrity and progress-monitoring data are reviewed together and one of four possible data-based decisions is made. Depending on whether im-plementation is deemed sufficient and whether the student is on

95

track to meet his or her goal, the appropriate next step will involve continuing current practice, conducting further evaluation, chang-ing the intervention, or providing PRIME Implementation Supports. Specific action steps are described below. If PRIME Implementation Supports are deemed appropriate, treatment integrity and IBA data will help you identify what strategy is appropriate to support the implementer.

How to Make PRIME Data-Based Decisions PRIME data-based decisions involve incorporating treatment in-

tegrity and progress-monitoring data. In Chapter 8, you graphed these data sources based on the questions of interest and devel-oped summary statements to describe the data. The next step is to integrate these summary statements to make a data-based decision. Use Appendix I to complete this three-step process.

1. Use the progress-monitoring summary statement to deter-mine whether the student is on track to meet his or her goal. That is, you will review the trend, level, and variability of the progress monitoring data to evaluate if the progress-mon-itoring trend line (i.e., the student‚Äôs overall progress) is aligned with the aim line (i.e., the line from baseline to the intervention goal). Based on this interpretation of the prog-ress-monitoring graph, decide whether (a) yes, the student is on track to meet his or her goal, or (b) no, the student is not on track to meet his or her goal.

2. Use the treatment integrity summary statement to determine whether the implementation is sufficient or insufficient. That is, is the intervention implemented to an extent where it could be expected that the student would benefit from the intervention? There is no specific level of implementation suggested across all interventions to be sufficient. Rather, review the current level of implementation overall and by

USING DATE TO TARGET IMPLEMENTATION SUPPORTS IN A MULTI-TIERED MODEL 96

How to Make Action Plans Based on Data-Based DecisionsIn the preceding section, you integrated the progress-monitoring

and treatment integrity summary statements to identify the pres-ent scenario. With these two data sources, there are four potential scenarios. The following section describes these scenarios and their associated action steps. Use Appendix I, the PRIME Data-Based Decision Making Worksheet, to walk through these decision-mak-ing steps. Record the appropriate scenario and action steps on the PRIME Intervention Implementation Tracking Form (Appendix D).

Scenario A: On Track to Meet Student Goal & Sufficient Treatment Integrity

In Scenario A, the student is making appropriate progress toward his or her intervention goal and the intervention is being imple-mented as planned. The intervention can be considered effective, based on the student‚Äôs progress. As the student is moving toward his or her intervention goal, it is not necessary to make any changes to the intervention or its implementation at this time. The action steps that correspond to this scenario, listed below, are related to continuing the current plan to support the student and implementer.

Scenario A Action Steps Continue to deliver the intervention at its current level of implementation.

Regularly evaluate progress monitoring and treatment integrity data to ensure the current progress and level of implementation is maintained over time.

Scenario B: On Track to Meet Student Goal & Insufficient Treatment Integrity

In Scenario B, the student is making appropriate progress to-wards his or her intervention goal, but the intervention is not be-ing implemented as planned. In this case, it is unclear whether the


intervention is effective. There are three possible reasons for this pattern of progress mon-

itoring and treatment integrity data: 1. It is possible that something outside of the intervention is re-

lated to the improvement in student outcomes. For example, it is possible that a seemingly unrelated change in class-room management, family activities, or academic/behavior support may have led to improvements.

2. It may be necessary to update the intervention goal and the corresponding expected rate of student progress (i.e., aim line). Data may indicate that, with appropriate support, the student can more quickly reach higher levels of improve-ment than expected.

3. There may be issues with the treatment integrity assess-ment method. For example, the treatment integrity tool may include several steps that are not directly related to the improvements in student outcomes and, if the implementer skips these steps, it would result in a low treatment integ-rity score, but actually sufficient implementation to support student outcomes. Or, the treatment integrity observations may not have accounted for implementation across the full session (e.g., day) so may be an underestimate of treatment integrity.

The action steps that correspond to Scenario B, listed below, and provided in a flowchart on Scenario B Action Steps worksheet in Appendix I, involve identifying the reason for the pattern of prog-ress monitoring and treatment integrity data and then taking corre-sponding next steps.

Scenario B Action Steps (see also Appendix I) Talk with the implementer and other stakeholders (e.g., classroom teacher, parent, paraprofessional, student) to


identify if any changes have been made outside of interven-tion implementation.

Review research on the intervention to evaluate if the initial projection for the intervention goal was appropriate.

Review the intervention step treatment integrity data, cover-age of the treatment integrity assessment method, and the representativeness of the observation sessions to identify if the intervention steps directly related to improvements in student outcomes were appropriately captured. Note: It may be useful to review intervention research to identify what steps most impact student outcomes.

Decide which of the three possible reasons for the data pat-tern most likely explains the current scenario.

If the data pattern is explained by reason 1, systematically discontinue implementation of the intervention, as it may not be necessary for the student to reach his or her goal. Continue to regularly evaluate progress-monitoring data to ensure that the improved student outcomes are maintained.

If the data pattern is explained by reason 2, review the in-tervention goal and adjust the aim line for student progress. Reevaluate the progress-monitoring and treatment integrity data to make an appropriate, data-driven decision about next steps. Continue to regularly evaluate progress-monitor-ing and treatment integrity data.

If the data pattern is explained by reason 3, adjust the treat-ment integrity assessment method and/or observation ses-sions to appropriately account for the intervention. Reevalu-ate the progress-monitoring and treatment integrity data to make an appropriate, data-driven decision about next steps. Continue to regularly evaluate progress-monitoring and treatment integrity data.


Scenario C: Not On Track to Meet Student Goal & Sufficient Treatment Integrity

In Scenario C, the student is not making appropriate progress to meet his or her intervention goal, however, treatment integrity data show the intervention is being implemented as planned. It appears the intervention is not effective for the specific student and their intervention goal.

Before changing the intervention, consider whether the treatment integrity data accurately account for current implementation. The assessment method or sessions observed may have inadvertently suggested a higher level of implementation than actually occurred regularly. For example, during the session observed, the implement-er could have delivered the intervention with higher than usual treatment integrity or the implementer could have overestimated their implementation on the self-report measure.

If it is believed the intervention is appropriate for the problem and student, the consultant and implementer could increase the intensi-ty of the intervention. For instance, you may increase the frequency and duration of the intervention (e.g., delivering an academic inter-vention four times per week instead of two) or change the format of intervention delivery to decrease the ratio of students to teachers (e.g., delivering an academic intervention to a student one-to-one rather than in a small group). Make modifications to that interven-tion that are research supported and theoretically sound.

If changes to the treatment integrity assessment plan or modifica-tions to the intensity of the intervention do not result in changes for the student, the intervention may not be appropriate for the student. In this case, review the baseline data, hypotheses developed, and identify an alternative evidence-based intervention.

The action steps that correspond to Scenario C, listed below, and provided in a flowchart on Scenario C Action Steps worksheet in Appendix I, involve identifying the reason for the pattern of prog-ress-monitoring and treatment integrity data and then taking corre-sponding next steps.


Scenario C Action Steps (see also Appendix I) Review the intervention step treatment integrity data, treatment integrity assessment method, and the repre-sentativeness of the observations session to identify if the intervention steps directly related to improvements in student outcomes were appropriately captured. Note: It may be useful to review intervention research to identify what steps most impact student outcomes. If appropriate, revise the treatment integrity assessment method. Reevaluate the progress monitoring and treatment integrity data to make an appropriate, data-driven decision about next steps.

Consider increasing the intensity of the intervention. If such modifications are made, update the Implementation Plan to account for new implementation behaviors and poten-tial barriers. After subsequent implementation, evaluate the progress monitoring and treatment integrity data to make an appropriate, data-driven decision about next steps.

Decide to change interventions. To do so, begin the PRIME process again, beginning with Problem Identification (see Chapter 2).

Scenario D: Not On Track to Meet Student Goal & Insufficient Treatment Integrity

In Scenario D, the student is not making sufficient progress to meet his or her intervention goal and the intervention is not being implemented as planned. In this case, treatment integrity should be promoted through the delivery of a PRIME Implementation Support. To identify the appropriate PRIME Implementation Support, you need to decide (a) what Tier of support is appropriate and (b) what support is appropriate for the specific situation.

In general, provide PRIME Implementation Supports beginning at the Tier 1 level before delivering strategies at Tier 2 and, last, pro-viding ongoing intensive support at Tier 3. To decide what Tier of


PRIME Implementation Support is appropriate for the specific situa-tion, consider the following (see also flowchart on Scenario D Action Steps worksheet in Appendix I):

1. Review implementation history. What PRIME Implementation Supports have been provided to support the implementer previously? For example, if an implementer is struggling and received Tier 1 supports months ago, it might make sense to again deliver Tier 1 supports or deliver Tier 2 strategies. If an implementer has received several Tier 2 strategies, but continues to deliver the intervention with low treatment integrity, it may be appropriate to provide Tier 3 supports.

2. Consider the intensity of the intervention. At what tier is this intervention? What decisions may be made based on the student‚Äôs response to this intervention? In most cases, there should be a relationship between intensity of the intervention provided to the student and the intensity of implementation support provided to the implementer. That is, for a student intervention that is more intensive (e.g., stu-dent spends much time outside of the classroom, it‚Äôs re-source-intensive, or significant decisions will be made based on these data), more quickly deliver increasingly intensive PRIME Implementation Support.

PRIME Tip• If the student is receiving a less intense intervention, provide

Tier 1 Implementation Supports and deliver Tier 2 Implementation Support Strategies several times, before providing Tier 3 Implemen-tation Support, if needed.

• If the student is receiving an intensive intervention, the consul-tant should more quickly move from Tier 1 and Tier 2 Implemen-tation Supports to Performance Feedback at Tier 3 of the PRIME Model. For example, if an implementer experiences challenges


implementing an intensive behavior support plan for a student (i.e., Tier 3 student intervention), the consultant may provide a Tier 2 Implementation Support Strategy once before moving to Tier 3, if treatment integrity support is still warranted.

Once you have decided what Tier of PRIME Implementation Sup-port is warranted, it is time to determine what specific support is appropriate. To do so, use treatment integrity and IBA data to target what Implementation Support best addresses the challenges the implementer is experiencing.

For Tier I Implementation Supports, decide whether to provide Di-rect Training, Implementation Planning, or both supports. If these supports have not yet been provided, we suggest both are delivered to facilitate high levels of treatment integrity. At the very least, Di-rect Training should be provided due to its foundational steps. If these supports have been provided, but specific intervention steps are regularly missed, consider providing ‚Äúbooster‚Äù Direct Train-ing, updating the Implementation Plan for those steps, or both. This plan would also be an appropriate strategy if the IBA data indicate low levels of self-efficacy. If the IBA data indicate low outcome ex-pectations or overall treatment integrity is low, consider providing Direct Training again.

For Tier 2 Implementation Supports, decide which of the four strat-egies is appropriate to address the implementer‚Äôs treatment in-tegrity. To do so, review treatment integrity and IBA data and, based on these data, identify what strategy is appropriate.

• Role Play or Participant Modeling are appropriate strategies if (a) intervention step treatment integrity data indicate some steps are not implemented at all; (b) session treat-ment integrity data indicate low levels of quality, even if adherence is overall adequate; or (c) the IBA data indicate the self-efficacy subscale is low. To decide between these two PRIME Implementation Support strategies, review the


strategy protocol and consider the implementer and context. For example, some implementers may welcome a consultant into their classroom (necessary for Participant Modeling), while others would prefer to practice in a meeting format (i.e., Role Play).

• Motivational Consulting or Raising Awareness are appropri-ate strategies if (a) treatment integrity is inconsistent (e.g., fully implemented some days, little implementation the next); (b) treatment integrity has been high, but has de-creased over time; (c) treatment integrity data indicate low levels of exposure, even if adherence is overall adequate; or (d) the IBA data indicate the outcome expectations subscale is low. To decide between these two PRIME Implementation Support strategies, review the strategy protocol and consider the implementer and their feedback about the intervention and its implementation. For example, if the implementer sees the benefit of the intervention, but has low self-effica-cy about implementing consistently or frequently enough, Motivational Consulting may be more suited, while if the implementer needs support to increase their understanding of expected outcomes as well as support to implement more consistently and comprehensively, Raising Awareness may be best.

For Tier 3 Implementation Supports, provide Performance Feedback. As described in Chapter 14, use the intervention intensity, treatment integrity data, and IBA data to decide how frequently to provide Per-formance Feedback.

The action steps that correspond to Scenario D, listed below, in-volve identifying what Tier of support is appropriate, determine what specific support is appropriate for the situation, and preparing to do deliver the PRIME Implementation Support.


Scenario D Action Steps Determine what Tier of PRIME Implementation Support is appropriate to support the implementer

Review implementation support history, treatment integrity and IBA data, and contextual/implementer factors to deter-mine the most appropriate PRIME Implementation Support.

Prepare to deliver the PRIME Implementation Support by reviewing the appropriate PRIME Manual chapter (Chapter 3-4 or 10-14), the General Implementation Support Guide-line (Appendix E) and the specific PRIME Implementation Support protocol (Appendices F-G or J-N).

Continue to regularly evaluate progress-monitoring and treatment integrity data.

What Did I Learn About PRIME? In the PRIME Manual, treatment integrity data, progress-moni-

toring data, and IBA data should be collected, graphed, and used to make data-based decisions. This chapter addressed how to in-tegrate data to make appropriate decisions and next steps. One of four data patterns can be found when treatment integrity and prog-ress-monitoring data are reviewed together. Depending on whether implementation is deemed sufficient and whether the student is on track to meet his or her goal, the appropriate next step will involve continuing current practice, conducting further evaluation, changing the intervention, or providing PRIME Implementation Supports.

Chapter 9 Key Terms Data-based Decision Making


107

When progress-monitoring and implemen-tation data indicate that treatment integ-rity promotion is needed, there are several PRIME implementation supports that can be applied. To describe the Tier 2 and 3 implementation supports, this section in-cludes five chapters. Chapter 10 describes Role Play, a strategy that involves prac-ticing varied intervention scenarios, while Chapter 11 describes Participant Modeling, a strategy that includes in-vivo demonstra-tion and practice of the intervention. Chap-ter 12 describes Raising Awareness, a strat-egy focused on identifying beliefs that may influence implementation, while Chapter 13 describes Motivational Consultation, a strategy that highlights the link between

implementation and desired outcomes by promoting change talk. Chapter 14 de-scribes Performance Feedback, a strategy that involves showing the implementer with their implementation data, praising for steps consistently delivered, and prac-ticing intervention steps that are inconsis-tent. The in-depth descriptions of how to plan for and complete these strategies that is found in these chapters are accompanied by strategy protocols and treatment integ-rity guides found in Appendices J to N.

Through reading this section, you will learn how to prepare and deliver Tier 2 and 3 Implementation Supports.

PART 6

Tier 2 and 3 Support

collect data

anal

yze

pro

gres

sidentify next s

teps

TIERS

2&3

108

CHAPTER 10

Role Play

What Will This Chapter Tell Me?Role Play involves reviewing the current status of implementation

and demonstrating and practicing how to implement an interven-tion with several scenarios. In the PRIME Model, Role Play is consid-ered a Tier 2 Implementation Support. This chapter describes Role Play in detail and describes how to get ready for and deliver this strategy. After reading this chapter, you will be ready to prepare for and implement an effective Role Play session.

What is Role Play?The purpose of Role Play is to increase the implementer’s prepara-

tion and confidence to implement the intervention by (a) reviewing implementation data and discussing any concerns, (b) demonstrat-ing implementation of the intervention, (c) practicing implementa-tion of difficult intervention steps. Role Play was developed based on Social Learning Theory and Behavioral Consultation literature. In PRIME, Role Play is a Tier 2 Implementation Support. It is most appropriate to support implementation when (a) intervention step treatment integrity data indicate some steps are not implement-ed at all; (b) session treatment integrity data indicate low levels of quality, even if adherence is overall adequate; or (c) the Implemen-

109

tation Beliefs Assessment (IBA) data indicate the self-efficacy sub-scale is low (see Chapter 9 and Appendix I for additional guidance). After Role Play, the implementer will feel positive and confident to maintain intervention implementation over time.

How to Prepare for Role PlayTo deliver Role Play, some preparation is necessary. Be sure to re-

view the general guidelines for a PRIME Implementation Support meeting (see Appendix E). In particular, make sure to review the in-tervention plan, specifically the steps with low treatment integrity, and be prepared to describe and implement it. Beyond the general guidelines, preparation for Role Play includes 4 steps:

1. Planning for Role Play to take one session outside of the implementation setting;

2. Preparing intervention implementation and student outcome graphs;

3. Identifying intervention steps for practice; and4. Readying necessary materials.

Step 1: Plan How to Complete Role PlayFirst, decide when to complete Role Play with the implementer.

Plan for one session outside of the implementation setting. Review the steps of Role Play to ensure there is enough time to discuss all of the steps of the session. Make sure your meeting space is quiet and private to allow for you to demonstrate and practice the inter-vention.

Step 2: Prepare Intervention Implementation and Student Outcome GraphsSecond, prepare the intervention implementation and student

outcome information that will be discussed during the session. These graphs will help facilitate your discussion with the imple-menter about the current status of the intervention. Specific graphs

ROLE PLAY

110

to prepare include the Treatment Integrity Across Sessions Graph, Treatment Integrity Across Intervention Steps Graph, and Progress Monitoring Graph (see Chapter 8). Make sure you understand the graphs and are ready to summarize the information and answer questions in a constructive and supportive manner. See Chapter 8 for additional information on graphing and interpreting these data.

Step 3: Identify Intervention Steps for PracticeThird, identify the intervention steps to target using Role Play by

reviewing treatment integrity data. To do so, review the Treatment Integrity Across Intervention Steps Graph and find the steps with low treatment integrity. The steps that the implementer is strug-gling to deliver consistently will be the focus on the Role Play ses-sion. Brainstorm and select scenarios that may allow for practice of these intervention steps.

Step 4: Gather MaterialsLast, gather the materials needed for the Role Play session. These

materials include intervention implementation data (i.e., Treatment Integrity Across Sessions Graph and Treatment Integrity Across In-tervention Steps Graph), student outcomes data (e.g., Progress Mon-itoring Graph), the IBA data, and a written list of intervention steps. Bring a copy of the list of the intervention steps for the implementer and make sure to denote “target” steps to practice as determined by treatment integrity data.

How To Deliver Role Play The steps of Role Play are described below. This chapter is a more

detailed companion to the Role Play protocol and treatment integ-rity measure found in Appendix J.

Step 1: Explain Session Purpose Begin the Role Play session by explaining the purpose of the ses-

sion. Tell the implementer that you will discuss the intervention and

ROLE PLAY

111

any concerns they might have about implementing the intervention consistently. Let the implementer know that you will be practicing the intervention to ensure they are comfortable delivering the in-tervention.

Provide an overview of the Role Play session by briefly reviewing the steps of the strategy. That is, describe how you will review inter-vention steps linking each to treatment integrity and progress-mon-itoring data, decide on scenarios for practice, model the steps, pro-vide the implementer with an opportunity to practice, and provide support and feedback. Make sure to link the Role Play steps with the overall purpose of the session. For instance, you may discuss with the implementer how the demonstration and practice can help sup-port implementation and lead to improved student outcomes and help the student reach his or her goal

Based on this overview of Role Play, develop goals for the session with the implementer. Developing meeting goals will allow you to target the discussion and ensure there is a shared vision for the meeting. Goals for Role Play might include the implementer being more confident to implementation in challenging situations or un-derstanding specific intervention steps better. Use your understand-ing of Role Play to help target the implementer’s suggestions for the session goals. Once you’ve decided on shared goals, briefly explain how Role Play will help meet these session goals and refer to the goals throughout the session.

Step 2: Elicit Implementer Feedback about Implementation

Beliefs Assessment DataThe next step involves review of the Implementation Beliefs As-

sessment results. To do so, highlight themes from the IBA in a col-laborative and supportive manner. Ask the implementer to reflect on their responses (e.g., “Does that summarize your impressions?”). In doing so, have the implementer identify specific scenarios that relate to his or her concerns about implementing the intervention.

ROLE PLAY

112

For example, you might ask “What are the particular times when you feel less prepared to implement the intervention?” The scenarios suggested by the implementer can be used later during practice or brainstorming.

Step 3: Discuss Intervention Steps as Related to Treatment Integrity and Progress- Monitoring Data Make connections between the purpose of each intervention step

and the levels of implementation, progress-monitoring data, and IBA. Review the intervention steps in detail, explaining why that step is used and what implementing that step will serve to accom-plish. Use the graphs and summary statements from the data to talk with the implementer about how the current implementation of each step and the implementer’s beliefs about the effectiveness of the intervention and their confidence in implementation might be affecting student outcomes. Praise the implementer for consistent implementation of intervention steps. Assure the implementer that the role play will allow them to practice the intervention steps that have been more difficult to implement consistently.

Step 4: Elicit Implementer Feedback regarding Intervention StepsAsk the implementer about his or her perspective about the dis-

cussion of the intervention steps and implementation and prog-ress-monitoring data. Confirm with the implementer that the state-ments shared with them about implementation represent their perspective. Throughout, summarize and validate the implement-er’s perspective on implementation and their skills.

Step 5: Discuss Intervention Steps to be Practiced and Practice ScenariosSet up the scenario for the demonstration. Using the implementa-

tion data as well as the implementer’s responses about intervention

ROLE PLAY

113

steps and treatment integrity data, identify target intervention steps to be practiced during the demonstration. Ask the implementer to suggest some practice scenarios for the target intervention steps. This is the opportunity to demonstrate implementation in particu-larly challenging or realistic situations. Suggest the scenarios that you brainstormed before the session and decide whether or not to include them based on the implementer’s feedback. The goal is to provide a demonstration of implementation that is most helpful to the implementer and meets their specific needs.

Step 6: Demonstrate the Intervention Step(s) with the Implementer Acting as a StudentDemonstrate the target intervention steps. To do so, act as the “im-

plementer” and have the implementer act as the “student.” You may demonstrate the intervention in one of two ways. One option is to describe the implementation behaviors as they occur, making mind-ful notes of both the adherence (i.e., delivering the intervention as planned) and quality (i.e., how you are delivering intervention com-ponents) of implementation. For example, if you are demonstrating closer proximity to a student that is demonstrating problem behav-ior say “Because I am seeing the student demonstrate behaviors, I am moving torward him, while still paying attention to other students and providing behavior specific praise to them”. Another option is to simply demonstrate the intervention steps as planned without describing your behavior. Either option may be used for different in-tervention steps, depending on such factors as the complexity of the intervention step and based on the feedback from the implementer.

Step 7: Exchange Feedback about DemonstrationFollowing the demonstration, engage the implementer in a di-

alogue about the demonstration. To do so, ask the implementer to share their feedback. For example, you may ask “What did you notice about how I responded to problem behavior?” or “Did that

ROLE PLAY

114

seem like how you pictured providing additional opportunities to respond?”. Summarize the implementer’s perspective on the demon-stration and validate their feedback. Share your own thoughts about the demonstration by describing intervention steps that were easier or more challenging and strategies you used to implement the more difficult steps. Make sure to praise the implementer’s role as student during the demonstration.

Step 8: Discuss Role Play GoalsNow transition to the role play. Collaborate with the implementer

to identify general (e.g., a successful practice session) and specific (e.g., demonstrate strategies for a particularly difficult step) goals for the practice. You might say “What do you want to accomplish as you practice the intervention? Is there anything we should make sure to specifically address?”. Use the previous discussion of the IBA data, treatment integrity data, and other feedback to guide the de-velopment of role play goals.

Step 9: Implementer Role Plays with Consultant Acing as a StudentHave the implementer role play an intervention step or group of

steps while you act as a student. Listen and observe as the imple-menter practices the intervention steps. Make a note of interven-tion steps that are relatively easy or difficulty. Pay attention to both treatment integrity adherence and quality and consider both verbal and non-verbal behavior. If necessary, encourage the implementer and provide prompts for accurate implementation.

Step 10: Exchange Feedback about Implementer’s PracticeDebrief with the implementer following role play. Praise the

implementer’s efforts for implementing the intervention steps as planned. Ask the implementer what steps went well and they felt they delivered confidently. Provide praise for those steps. Also, ask

ROLE PLAY

115

the implementer what steps were more difficult to deliver. Summa-rize and validate the implementer’s perspective about the role play and implementation strategies. Share your feedback about the role play, staying positive and emphasizing intervention steps that were implemented successfully. Collaboratively with the implementer, brainstorm solutions to remaining areas of difficulty and prompt the implementer for their perspective on implementation strategies. Re-peat the role play practice until the implementer has mastered all target intervention steps without your support.

Step 11: Close the Session To close the Role Play session, thank the implementer for taking

the time to meet with you and their openness to the practice. Reiter-ate the relationship among intervention steps, progress monitoring, and consistent and sustained implementation of the intervention plan. Review the goals and the progress made during the session. Reinforce them for spending the time and effort to practice the in-tervention steps with you.

What Did I Learn About PRIME?In the PRIME Model, Role Play is a Tier 2 Implementation Support

aimed to build implementer’s implementation skill and self-effica-cy. The consultant and implementer identify difficult intervention steps that have not been implemented consistently through review of treatment integrity data. The consultant and implementer discuss the implementer’s perspective on current implementation, and the relationship between sustained implementation and student prog-ress. Through demonstration and practice of target intervention steps, the implementer’s preparation and confidence to implement the intervention is increased with the goal of maintaining consis-tent implementation.

Chapter 10 Key TermsIntervention StepsRole Play

ROLE PLAY

116

What Will This Chapter Tell Me?Participant Modeling involves demonstration and practice of the

intervention steps in the setting of implementation. In the PRIME Model, Role Play is considered a Tier 2 Implementation Support. This chapter describes the purpose of Participant Modeling as well as how to prepare for and deliver the strategy steps. After reading this chapter, you will be able to get ready for and implement an ef-fective Participant Modeling session.

What is Participant Modeling?The purpose of Participant Modeling is to improve the imple-

menter’s self-efficacy to implement the intervention and outcome expectations through a positive experience implementing the inter-vention and demonstration of the intervention benefits. The com-ponents of Participant Modeling are (a) a review of evidence sup-porting the intervention and (b) modeled practice of intervention steps within the target setting. Participant Modeling was developed based on social learning theory, which emphasizes the importance of learning from observation and imitation of role models. In PRIME, Role Play is a Tier 2 Implementation Support. It is most appropri-ate to support implementation when (a) intervention step treatment

CHAPTER 11

Participant Modeling

117

integrity data indicate some steps are not implemented at all; (b) session treatment integrity data indicate low levels of quality, even if adherence is overall adequate; or (c) the Implementation Beliefs Assessment (IBA) data indicate the self-efficacy subscale is low (see Chapter 9 or Appendix I for additional guidance). The implementer should leave Participant Modeling feeling prepared to implement and confident to implement intervention independently.

How to Prepare for Participant ModelingTo deliver Participant Modeling, some preparation is necessary. Be

sure to review the general guidelines for a PRIME Implementation Support meeting (see Appendix E). In particular, make sure to review the intervention plan and feel prepared to describe and implement it. Beyond the general guidelines, preparation for Participant Mod-eling includes 5 steps:

1. Preparing for Participant Modeling to take three sessions; the first and third sessions will occur outside of the implementa-tion setting and the second session should happen in-vivo.

2. Reviewing the intervention rationale;3. Reviewing intervention implementation data and determin-

ing target interventions steps;4. Deciding the order to model intervention steps; and5. Readying necessary materials.

Step 1: Plan How to Complete Participant ModelingFirst, plan when to complete the three sessions of Participant

Modeling. The first and third sessions should occur outside of the implementation setting, whereas the second session will happen in the implementation setting, or in-vivo. It is possible to have the three sessions in a row, or it may be necessary to hold the sessions over a few days, based on scheduling. Hold the sessions as close together as possible to ensure that the implementer is ready for the implementation practice (session 2) and able to describe the prac-tice during the debriefing (session 3).

PARTICIPANT MODELING

118

Step 2: Review Intervention RationaleAn important component of Participant Modeling is discussing the

rationale for implementing the intervention with the implementer. To be prepared, review how the selected intervention addresses the identified problem and how it will help the student reach his or her intervention goal. Make sure to include the potential benefits of the intervention when it is implemented as intended.

Step 3: Review Intervention Implementation Data and Determine Target Intervention StepsNext, review the implementation data to identify the intervention

steps to target with this strategy. Specifically, review the treatment integrity adherence and quality data to determine the steps with low treatment integrity that can be modeled in the sessions.

Step 4: Decide the Order to Model Intervention StepsOnce the target steps have been identified, decide the order to

present them to the implementer. More than one format can be used, such as ordered sequentially, by implementation scenario, or grouped by difficulty. For instance, if the intervention was straight-forward, it might make sense to simply practice the intervention steps as listed (i.e., sequentially). If the implementer is struggling with implementation during a particular time of day, then looking at a scenario may be most appropriate. If the implementer is having difficulty with particularly challenging intervention steps, then focus on steps grouped by difficulty. Prepare the list of intervention steps arranged in the order of the presentation ahead of the sessions.

Step 5: Gather MaterialsLast, gather the materials needed for the Participant Modeling

sessions. These materials include the written list of target interven-tion steps and any items needed to practice the intervention, such as tangible reinforcers or forms. Bring a copy of the intervention steps for the implementer to keep.


119

How To Deliver Participant Modeling The steps of Participant Modeling are described below. This chap-

ter is a more detailed companion to the Participant Modeling proto-col and treatment integrity measure found in Appendix K.

Note: Steps 1 to 4 occur outside of the target setting.Step 1: Explain the Session Purpose

To begin the first Participant Modeling session, explain the pur-pose of meeting with the implementer. Describe the purpose as a chance to review and practice implementation to be sure it is im-plemented as intended. Outline the steps of Participant Modeling. That is, explain (a) how you will discuss the intervention during the first session; (b) how in-vivo, the consultant will model the select intervention steps and then have the implementer practice those steps with support and then independently; and (c) how you will debrief on the practice to close Participant Modeling. Based on this description of Participant Modeling, work with the implementer to develop goals for the session. Developing meeting goals will allow you to target the discussion and ensure there is a shared vision for the meeting. Goals for Participant Modeling might include increased mastery of the intervention or improved confidence in implementa-tion. Use your understanding of the strategy to help target the im-plementer’s suggestions for the session goals. Once you’ve decided on shared goals, briefly explain how Participant Modeling will help meet these session goals.

Step 2: Describe the Intervention and Importance of ImplementationDescribe the rationale for the intervention. That is, how does the

intervention address the identified problem? Describe how the ben-efits of intervention implementation will likely help the student reach his or her intervention goal. Discuss with the implementer how high levels of treatment integrity are related to more efficient improvements in student progress monitoring data.


120

Step 3: Review the Steps of the InterventionAfter describing the intervention rationale, review the interven-

tion steps in greater detail. Describe how each step will be imple-mented, why that step is used, and what implementing that step will serve to accomplish. Ask the implementer questions throughout this step to prompt for his or her perspective of intervention implemen-tation (e.g., “Does that step rationale make sense? What’s challeng-ing about that step?”). During this review, provide the implementer with the opportunity to make minor revisions to intervention steps, as needed. If revisions occur, update the intervention plan and Im-plementation Plan.

Step 4: Identify Target Practice Steps ad Prepare for In-VivoExerciseThe last step of the first session is to decide what intervention

steps will be targeted during the in-vivo practice exercise in the implementation setting. These intervention steps should be those that have not been implemented consistently and can be identi-fied through review of the treatment integrity data. It is most help-ful to practice steps that are an appropriate fit for modeling and practice, rather than steps that are a single occurrence at the start of the implementation period (e.g., posting visual reminders in the classroom). Use the list you developed before the session to guide the discussion, but elicit suggestions from the implementer as well. Following selection of the target steps, collaborate with the imple-menter to decide the format and logistics of the in-vivo practice session, such as how you will enter and exit the target setting and the order of demonstrating the intervention steps. Suggest to the implementer that he or she may want to notify the target student(s) of your presence in advance.

Note: Steps 5 to 9 occur in the target setting.Step 5: Complete In-Vivo Demonstration

The first step of the in-vivo session is to model the delivery of the intervention steps according to plan. In doing so, make sure to


121

attend to the adherence and quality of implementation. Be aware of body position and pronunciation to ensure that implementer can see and hear you clearly.

Step 6: Facilitate In-Vivo Supported PracticeFollowing demonstration of each target intervention step, invite

the implementer to practice the intervention steps with assistance. As the implementer practices, provide specific, positive, and correc-tive suggestions and feedback related to the adherence and quality of implementation. Continue with the guided practice until the im-plementer has mastered each intervention step. Repeat the practice and feedback process with targeted intervention steps.

Step 7: Allow Implementer to Independently PracticeDirectly after guided practice, transition to an independent prac-

tice. For this step, have the implementer practice the intervention steps without your support. Take note of intervention steps imple-mented completely and with high quality, as well as those steps that might need further support.

Step 8: Provide Feedback about the PracticePrompt the implementer to self-evaluate their independent prac-

tice (e.g., “What steps went well? What steps were a little less com-fortable?”). Summarize and validate the implementer’s perspective. Provide constructive feedback based on your observation of the independent practice. In doing so, reinforce successes and correct errors in implementation.

Step 9: Repeat Independent Practice and Feedback, if necessarySome implementers may need additional practice to master the

intervention steps. If needed, repeat the guided and/or independent practice until the implementer successfully and confidently imple-ments each component of the intervention independently.


122

Note: Steps 10 to 11 occur outside of the target setting.Step 10: Discuss Skill Generalization

In the third session of Participant Modeling, review the progress made during the in-vivo session. Provide the implementer specific praise and feedback regarding the adherence and quality of imple-mentation of targeted intervention steps during independent prac-tice. Discuss with the implementer the opportunity to generalize these skills to other settings or problems (e.g., Where and when might the implementer be able to use the intervention steps and other skills mastered during in-vivo practice?).

Step 11: Close the SessionTo close Participant Modeling, review the consultation goal(s) and

determine whether those goals have been met through three ses-sions. Ask the implementer if they have remaining questions about implementation or need additional resources or support to maintain implementation of the intervention as planned. Provide positive feedback to the implementer and reinforce participation in Partici-pant Modeling.

What Did I Learn About PRIME?In the PRIME Model, Participant Modeling is a Tier 2 Implementa-

tion Support designed to increase implementer confidence and skill in implementation. To implement the strategy, the consultant and implementer discuss the rationale for implementing the selected intervention and identify target intervention steps for further prac-tice. The consultant demonstrates the target intervention steps in the target setting and facilitates guided and independent practice of the intervention steps. Finally, the consultant and implementer discuss continued implementation of the intervention and general-ization of skills acquired through Participant Modeling.


123

Chapter 11 Key TermsIn-vivoParticipant Modeling


124

CHAPTER 11

Raising Awareness

What Will This Chapter Tell Me?Raising Awareness is a PRIME Implementation Support that fo-

cuses on how events and beliefs shape interpretations of events. Implementation data and research on intervention effectiveness are presented and discussed during the session. Raising Awareness is a Tier 2 Implementation Support in the PRIME Model. This chapter describes the preparation for and implementation of Raising Aware-ness. After reading this chapter, you will be able to successfully pre-pare for and implement an effective Raising Awareness session.

What is Raising Awareness?The purpose of Raising Awareness is to reorient the implement-

er’s perception of their ability to implement the intervention. The components of Raising Awareness are (a) review of implementation data; (b) discussion of the relationship between actual events, our beliefs, and the results; and (c) examination of research supporting the intervention. This strategy was informed by the cognitive be-havioral literature. Raising Awareness is a Tier 2 Implementation Support in the PRIME model. Therefore, this strategy should be used when (a) treatment integrity data are inconsistent (e.g., fully implemented some days, little implementation the next); (b) treat-

12

125

ment integrity data have been high, but have decreased over time; (c) treatment integrity data indicate low levels of exposure, even if adherence is overall adequate; or (d) the Implementation Beliefs Assessment (IBA) data indicate the outcome expectations subscale is low (see Chapter 9 or Appendix I for additional guidance). After Raising Awareness, the implementer will have considered a new perspective on intervention implementation.

How to Prepare for Raising AwarenessTo deliver Raising Awareness, some preparation is necessary. Be

sure to review the general guidelines for a PRIME Implementation Support meeting (see Appendix E). In particular, make sure to review the intervention plan and feel prepared to describe and demonstrate it. Beyond the general guidelines, preparation for Raising Awareness includes 5 steps:

1. Planning for the session;2. Reviewing implementation data;3. Collecting and reviewing evidence of intervention effective-

ness; and4. Readying necessary materials.

Step 1: Plan How to Complete Raising AwarenessFirst, determine when and where to complete the Raising Aware-

ness session with the implementer. Plan for one session outside of the implementation setting. Review the steps of Raising Awareness to adequately plan for time to discuss all of the talking points during the session. Make sure your meeting place is quiet and private to fa-cilitate honest discussion.

Step 2: Review Implementation DataNext, review the intervention implementation data found in the

Treatment Integrity Across Sessions Graph and Treatment Integrity Across Intervention Steps Graph. Be prepared to present these fig-

RAISING AWARENESS

126

ures to the implementer, summarize the salient themes from the data, and answer questions in a constructive and supportive manner.

Step 3: Collect and Review Evidence of Intervention EffectivenessThird, collect information relevant to demonstrating the effective-

ness of the intervention when implemented as planned across time. Be sure to use high-quality research, case studies, and/or literature reviews, which can be found in libraries, credible websites, literature databases and/or your own resources.

Step 4: Gather MaterialsLast, gather the materials needed for the Raising Awareness ses-

sion. These materials include the intervention implementation data (i.e., Treatment Integrity Across Sessions Graph and Treatment Integ-rity Across Intervention Steps Graph), the written intervention plan, high-quality research related to importance of implementation in intervention effectiveness, and the Raising Awareness Worksheet depicting the 3-step sequence.

How To Deliver Raising Awareness The steps of Raising Awareness are described below. This chapter

is a more detailed companion to the Raising Awareness protocol and treatment integrity measure found in Appendix L.

Step 1: Explain Session PurposeBegin the Raising Awareness session by explaining that you are

meeting to review treatment integrity data and talk about the im-plementer’s perspective on implementation. Provide an overview of Raising Awareness by describing steps including looking at the current status of implementation, discussing how the implementer feels about intervention progress and delivery, and reviewing inter-vention research. Having the implementer understand the plan for the session will help them to be comfortable. In addition, collabo-

RAISING AWARENESS

127

ratively develop goals for the session. For Raising Awareness, these might include increasing the implementer’s skills, confidence, and motivation related to intervention implementation. Use your knowl-edge of this strategy to help the implementer identify appropriate session goals. Describe how the steps of Raising Awareness will meet the session goals.

Step 2: Review Implementation DataReview the treatment integrity data with the implementer. To do

so, present the Treatment Integrity Across Sessions Graph and the Treatment Integrity Across Intervention Steps Graph and describe the current levels of overall and component treatment integri-ty, respectively. Highlight intervention steps that were completed consistently and praise the implementer for delivering these steps with high levels of treatment integrity. Next, review steps that have not been implemented consistently and offer support to the im-plementer to deliver those challenging steps as planned. Ask the implementer for their impressions and feedback on the treatment integrity data. Summarize and validate the implementer’s response and confirm you provided an accurate summary.

Step 3: Ask the Implementer About ImplementationAsk the implementer for his or her perspective on intervention

implementation. In doing so, ask the implementer what has been going well and what challenges to implementation have arisen. Be supportive and non-judgmental in your response to their perspec-tive. Ask probing questions as appropriate so that you can identi-fy their impressions and beliefs about implementation. Summarize and confirm the implementer’s perspective, and make links between the implementation data and the implementer’s perspective, as ap-propriate. Confirm that your summary of the implementer’s perspec-tive is accurate.

RAISING AWARENESS

128

Step 4: Describe the 3-Step SequenceDescribe that how we understand and think about events impacts

the results of these events. Suggest the relationship between (a) an actual event, (b) our beliefs, and (c) the results. Use the Rais-ing Awareness Worksheet (see Appendix L) and real life scenarios, such as the ones described in the PRIME Tip below, to illustrate and explain the 3-step sequence (i.e., actual events-our beliefs-the re-sults). For intervention implementation, describe how results might include insufficient student progress, discarding the current inter-vention, and/or looking for a new, more intensive intervention. These results are impacted not only by intervention implementation (i.e., the actual event), but also by our beliefs about the student, inter-vention, and other factors. Make sure to note that it’s not “bad” that our beliefs impact our interpretation of results, but simply a part of being human and important for us to consider and reflect upon.

PRIME TipsIn our experience, it is helpful to show how the 3-step sequence

relates to non-implementation oriented activities before the im-plementer explains how the sequence fits into the context of their implementation. To help facilitate this dialogue, there are two ex-amples below.

• A person interested in losing weight might go to the gym twice a week for a light workout (i.e., actual event). The person might believe that once they workout they will quickly lose weight and gain strength (i.e., belief). The result of their brief workouts and this belief might result in their decision that the workout is ineffective and make plans to try a different, more expensive plan (i.e., results). If they consider the actual event, this person might realize they don’t need a different, more expensive plan, but they need to reorient their beliefs (e.g., “I am going to work on getting healthier over time”) and/or increase their time at the gym (i.e., the actual event). This

RAISING AWARENESS

129

example is illustrated on the Raising Awareness Worksheet.• A person might be trying a diet, but be cheating on a diet

by eating on unhealthy snacks in the evenings (i.e., actual event). Because this person has not been sticking to the diet, they will likely not see desired results. If a person thinks that the diet is simply not working (i.e., beliefs), but has not been sticking to it, then they might try a more restrictive diet or give up on losing weight (i.e., results). If they consider the actual event, this person might realize that they don’t need a more challenging diet or to give up on their hard work, but they need to reorient their beliefs (e.g., “I need to pay more attention to sticking to my diet to make sure it works”) and/or change their behavior by choosing healthy snack options (i.e., the actual event).

Step 5: Ask the Implementer to Describe the 3-Step SequenceAsk the implementer about how the treatment integrity data and

his or her perspective might fit into the (a) actual event, (b) beliefs, and (c) results sequence. Some implementers will be able to easily describe this 3-step sequence. For others, probe each step of the sequence specifically to build an understanding of the sequence. In this case, it might be best to start by identifying the actual event, as you have recently reviewed the treatment integrity data. Then, ask the implementer to frame their perspective of the intervention that was shared in Step 3 as the beliefs and results. If the implement-er continues to struggle to describe the 3-step sequence, provide some tentative suggestions based on your perspective. Summarize the implementer’s perspective on the 3-step sequence and make sure to confirm with the implementer that your summary is correct and update, if needed.

Step 6: Present Research About Intervention ImplementationPresent the prepared research on effectiveness of the intervention

when implemented as planned. Explain how the research can be

RAISING AWARENESS

130

looked at in terms of the 3-step sequence. In particular, highlight the actual event (i.e., the intervention was implemented as planned) and the result (i.e., student outcomes improved). Emphasize the im-portance of properly implementing all components of an interven-tion to positively impact student outcomes and intervention goal attainment. Reflect on what the participants who implemented the intervention in the study might have held as beliefs. Target this dis-cussion based on what beliefs the implementer is struggling with. Ask the implementer about their impression of the research on im-plementation and how it relates to the 3-step sequence. Ask how this information makes the implementer think about their 3-step sequences and reasons to sustain implementation. Summarize and validate their perspective.

Step 7: Brainstorm Strategies to Improve ImplementationTransition the discussion to identifying strategies to support con-

sistent implementation of the intervention in the target setting. For instance you may say, “As we’ve seen that implementing the in-tervention can really improve student outcomes, let’s look at what strategies we can identify to support your implementation.” Ask the implementer to identify strategies to help implement the interven-tion as planned consistently in the future, even when it may be diffi-cult to implement. Target this discussion based on the treatment in-tegrity data and the implementer’s beliefs. Support the implementer to develop feasible strategies to deliver the intervention as planned and suggest strategies as needed. Update the Implementation Plan, if the strategies identified impact the logistics of implementation or related barriers.

Integrate the topics of intervention effectiveness research, current implementation, the 3-step sequence, and implementation strate-gies. To do so, summarize and affirm the implementer’s perspective of (a) the intervention implementation research, (b) how these rea-sons fit into the 3-step sequence, and (c) strategies to assist with implementation. Confirm that your summary is accurate.

RAISING AWARENESS

131

Step 8: Summarize the 3-Step SequenceAsk the implementer to summarize the current and future inter-

vention implementation in relation to the 3-step sequence. The im-plementer previously described the current 3-step sequence during Step 5. The future 3-step sequence should include the new per-spective the implementer has considered by reviewing the research on intervention effectiveness and identifying new strategies to help implement the intervention as planned consistently. Support the implementer to summarize these 3-step sequences by asking prob-ing questions and framing their statements, as needed. Summarize and validate the implementer’s response.

Step 9: Close the SessionTo close the Raising Awareness session, summarize the discussion

of the session and make statements about how the implementation strategies brainstormed in the session can be used in the target set-ting. Summarize the importance of implementing the intervention consistently as planned and how the interpretation of implementa-tion can affect implementation, highlighting the 3-step sequence. Finally, thank the implementer for taking the time to meet with you, being open to new ideas, and engaging in critical examination of the data and research.

What Did I Learn About PRIME?Raising Awareness is a Tier 2 PRIME Implementation Support that

involves discussion of how events and beliefs shape interpretations of events. The purpose of Raising Awareness is to reorient the im-plementer’s perception of their ability to implement the interven-tion. The components of Raising Awareness are (a) review of imple-mentation data, (b) discussion of the relationship between actual events, our beliefs, and the results, and (c) examination of research supporting the intervention. The implementer should leave the ses-sion considering a new perspective on intervention implementation.

RAISING AWARENESS

132

Chapter 12 Key TermsRaising Awareness

RAISING AWARENESS

133

CHAPTER 13

Motivational Consulting

What Will This Chapter Tell Me?Motivational Consulting involves reviewing the intervention and

treatment integrity data with the implementer in a supportive man-ner to highlight the relationship between improvement in student outcomes and consistent implementation. In the PRIME Model, Mo-tivational Consulting is considered a Tier 2 Implementation Sup-port. This chapter describes Motivational Consulting and how to get ready for and deliver this strategy. After reading this chapter, you will be able to successfully prepare for and implement an effective Motivational Consulting session.

What is Motivational Consulting?The purpose of Motivational Consulting is to increase the imple-

menter’s self efficacy by (a) providing the implementer with infor-mation about intervention implementation and (b) using empathy and positive regard as a consultant. Motivational Consulting was de-veloped based on the Motivational Interviewing literature. In PRIME, Motivational Consulting is considered a Tier 2 Implementation Sup-port. It is most appropriate when (a) treatment integrity is inconsis-tent (e.g., fully implemented some days, little implementation the next); (b) treatment integrity has been high, but has decreased over

134

time; (c) treatment integrity data indicate low levels of exposure, even if adherence is overall adequate; or (d) the Implementation Be-liefs Assessment (IBA) data indicate the outcome expectations sub-scale is low (see Chapter 9 or Appendix I for additional guidance). Following Motivational Consulting, the implementer will know the link between treatment integrity and student outcomes and thus, be ready to maintain intervention implementation over time.

How to Prepare for Motivational ConsultingTo deliver Motivational Consulting, some preparation is necessary.

Be sure to review the general guidelines for a PRIME Implementa-tion Support meeting (see Appendix E). In particular, make sure to review the intervention plan and Implementation Plan and feel pre-pared to describe and implement them. Beyond the general guide-lines, preparation for Motivational Consulting includes 4 steps:

1. Breaking down the intervention plan into teachable interven-tion steps;

2. Planning for Motivational Consulting to take one session outside of the Implementation Setting;

3. Reviewing the implementer’s Implementation Plan, IBA data, and treatment integrity data; and

4. Readying necessary materials.

Step 1: Breaking Down the Intervention PlanFirst, review the intervention plan. In doing so, consider how to

best review the intervention plan with the implementer. Divide the intervention plan into grouped intervention steps that will help the implementer understand (a) the overall intervention plan and (b) how the intervention steps fit into the larger components of the intervention. For example, with a behavior support plan, it may be helpful to group intervention steps into antecedent (e.g., establish-ing and defining a classroom schedule, active supervision), teaching (e.g., teach behavior expectations, teach problem solving), and con-

MOTIVATIONAL CONSULTING

135

sequence (e.g., classwide group contingencies, positive reinforce-ment) strategies. For other types of interventions, organize inter-vention steps into logical groups to teach to the implementer. For example, you may divide the intervention steps according to when the steps must be implemented (e.g., all steps delivered at once, different steps provided at separate times) or the theoretical links between intervention steps (e.g., if several steps are based on one principle, if intervention steps build on one another). Note that this step may have already been completed when Direct Training was delivered. In this case, it will likely be appropriate to use those de-lineated intervention steps.

Step 2: Plan How to Complete Motivational ConsultingNext, plan when to complete the Motivational Consulting session

with the implementer. Delivering this strategy will take one session outside of the implementation setting. Review the steps of Motiva-tional Consulting to adequately plan for time to discuss all of the talking points during the session. Make sure your meeting place is quiet and private to facilitate honest discussion.

Step 3: Review the Implementation Plan, Implementation Beliefs Assessment Data, and Treatment Integrity Data Third, review the data that you will present to and discuss with

the implementer. Look over the step revisions, logistical planning, and barrier problem-solving that happened as a part of completing the Implementation Plan. Review treatment integrity data. In do-ing so, calculate the levels of adherence, quality, and exposure (as available) and identify areas for improvement. Review the results of the Implementation Beliefs Assessment and identify the important themes and responses to items that indicate low Outcome Expecta-tions and/or Self-Efficacy.

Step 4: Gather MaterialsLast, gather the materials needed for the Motivational Consulting

session. These materials include the intervention plan or a written


136

list of the intervention steps, the Implementation Plan, a summary of the Implementation Beliefs Assessment, and treatment integrity data. Bring copies of these materials for the implementer.

How To Deliver Motivational Consulting The steps of Motivational Consulting are described below. This

chapter is a more detailed companion to the Motivational Consult-ing protocol and treatment integrity measure found in Appendix M.

Step 1: Explain Session PurposeDescribe the overall purpose of the Motivational Consulting ses-

sion so the implementer understands what the meeting will entail and their role in the session. For instance, you may say that you will be discussing implementation and getting the implementer’s per-spective on the intervention and implementation. Note the impor-tance of implementation, as well as the importance of supporting implementation of the intervention plan. Provide an overview of Mo-tivational Consulting by briefly describing steps including looking at Implementation Beliefs Assessment and treatment integrity data and discussing the implementer’s impression of implementation. Work with the implementer to collaboratively develop goals for the session, such as increasing the implementer’s skills, confidence, and motivation related to intervention implementation. Use your under-standing of the Motivational Consulting strategy to help target the implementer’s suggestions for the session goals. Describe how the steps of Motivational Consulting will meet the session goals.

Step 2: Elicit Implementer’s Goals for and Perception of Intervention ImplementationAsk the implementer about their goals for intervention implemen-

tation, specifically. Prompt goals through the use of open-ended questions. For example, you may say “Tell me about the goals you have for the student or yourself in implementing this intervention.” Then, ask the implementer to explain their perspective on how im-


137

plementation has been going so far. Again, use open-ended ques-tions to understand the implementer’s beliefs related to implemen-tation (e.g., “How do you think implementation of the plan has been going?”).

Step 3: Summarize and Validate the Implementer’s Goals and Reinforce Change TalkSummarize the implementer’s perception of current implementa-

tion and goals for supporting the student in the continuing imple-mentation period. As part of this step, validate the implementer’s perception and feelings through empathic responses (e.g., “I under-stand that you want the students to be able to pay attention more in math class and I think that is a great goal for the student.”). Ad-ditionally, reinforce any change talk in the implementer’s responses (e.g., “I really appreciate your desire to implement the plan more fre-quently during the week to help the student reach his goals.”) Sum-marize the implementer’s responses and confirm that the summary was accurate. If it was not accurate, repeat the summary process.

Step 4: Highlight IBA Data and Elicit Implementer FeedbackAsk if the implementer is willing to review IBA data together.

Then, discuss with the implementer the items or themes related to reservations or concerns about implementing the intervention. Use open-ended questions, such as “What more can you say about stick-ing to the implementation of all steps of the intervention plan?”, to obtain more of the implementer’s concerns about implementation and the intervention plan.

Step 5: Summarize and Validate the Implementer’s FeedbackFollowing discussion of the IBA, summarize the implementer’s

positive beliefs and concerns about implementation. Validate feed-back about implementation and the results of the IBA. Make a note of “good things,” or endorsements of positive outcome expectations and high self-efficacy, and “less good things,” or statements of neg-


138

ative outcome expectations and low self-efficacy. For example, you may say “It’s wonderful that you agree that this intervention will work, but I see that sustained implementation of the intervention seems challenging” or “I see that you really want to support these student outcomes and you think the intervention will work, but it’s challenging to feel confident about implementing components of this intervention. Is that right?” Reinforce any change talk men-tioned by the implementer and confirm that your summary of the IBA results was correct.

Step 6: Provide Information Regarding Intervention ImplementationAsk if the implementer is wiling to review information about in-

tervention implementation. In doing so, frame this discussion as an important way to meet implementer goals and improve implement-er motivation to maintain implementation. Use case examples and/or research to explain the effectiveness of the intervention steps when they are implemented as planned. This step will transition into a review of the implementer’s current implementation of the intervention plan.

Step 7: Review Implementation Plan and Treatment

Integrity DataDiscuss the current status of implementation by reviewing the Im-

plementation Plan and treatment integrity data. Identify interven-tion steps that the implementer is implementing as planned and provide praise. Help the implementer identify the intervention steps that are of concern. If appropriate, offer to update the Implementa-tion Plan to increase contextual fit, if appropriate, toward the goal of improving the implementation of these steps.

Step 8: Explore Implementer’s Thoughts about Intervention ImplementationEngage the implementer in a discussion of their thoughts about

intervention effectiveness and current implementation by asking


139

open-ended questions. These questions might include “I am won-dering what you think about the current treatment integrity data?” or “What changes would you make to the Implementation Plan to make progress toward implementation goals?”

Step 9: Summarize, Validate, and Respond to Implementer’s PerspectiveSummarize the implementer’s responses and validate their per-

spective. Respond to questions about the intervention plan with de-tailed information. Validate implementer concerns about the inter-vention plan and implementation and address those concerns by (a) providing further information about sustained implementation of the intervention, (b) reviewing the intervention plan, or (c) encour-aging the implementer to continue implementation to see if results are similar to the research and case studies can be achieved.

Step 10: Ask Implementer about the Relationship Between Implementation and Intervention GoalsHelp the implementer articulate the potential impact of improved

implementation on student outcomes based on your discussion. Prompt the implementer to make conclusions about the relationship between implementation and intervention goals through open-end-ed questions. For instance, you may say “So, I think you’ve identified that there is a link between implementation and student interven-tion goals, how would you describe that?”.

Step 11: Summarize and Validate the Implementer’s PerspectiveSummarize and validate the implementer’s perspective on the im-

plementation-intervention goal relationship. Validate any change talk from the implementer and confirm that your summary is cor-rect. If the implementer brings up continued reservations about the intervention, validate the implementer’s concerns and encourage them to attempt to continue sustained implementation of the in-tervention.


140

Step 12: Close the SessionTo close Motivational Consulting, review the consultation goals

and determine whether those goals have been met through the ses-sion. Strongly affirm the implementer’s participation in Motivational Consulting and their willingness and readiness for change.

What Did I Learn About PRIME?In the PRIME Model, Motivational Consulting is a Tier 2 Implemen-

tation Support that aims to build the implementer’s understanding of the intervention, the importance of its consistent implementa-tion, and the relationship between consistent implementation and improved student outcomes, thus increasing their self-efficacy and outcome expectations. To deliver Motivational Consulting, the con-sultant and implementer review the intervention plan, Implemen-tation Plan, and the results of the IBA to elicit the implementer’s perspective on implementation and the intervention plan. The con-sultant provides validation, empathic responses, and positive regard to the implementer’s perspective on current implementation and reinforces implementer change talk to improve implementation.

Chapter 13 Key TermsMotivational Consulting


141

What Will This Chapter Tell Me?Performance Feedback is an implementation support that involves

sharing treatment integrity and progress-monitoring data with the implementer. It is a widely researched and effective strategy for in-creasing treatment integrity. In the PRIME Model, Performance Feed-back is suggested when the implementer does not respond (i.e., his or her treatment integrity does not increase) after delivery of Tier 1 and/or Tier 2 implementation supports. This chapter describes the preparation for and implementation of Performance Feedback. After reading this chapter, you will be able to successfully prepare for and implement an effective Performance Feedback session.

What is Performance Feedback?Performance Feedback is the more intensive Implementation

Support described in PRIME. In the multi-tiered system of supports available to implementers in the PRIME Model, Performance Feed-back is considered a Tier 3 Implementation Support. More specif-ically, Performance Feedback is used when an implementer con-tinues to demonstrate low levels of treatment integrity after other Implementation Supports have been provided to the implementer. The delineation as a Tier 3 strategy is primarily because it needs to

CHAPTER 14

Performance Feedback

142

be delivered on an on-going basis. That is, research has found Per-formance Feedback is effective at supporting implementation when delivered on a daily, weekly, or regularly as needed basis (i.e., when implementation drops below a certain criterion).

The purpose of Performance Feedback is to improve the imple-menter’s treatment integrity. During Performance Feedback, the con-sultant will engage the implementer in a discussion of treatment integrity and progress-monitoring data, review difficult implementa-tion steps, and problem-solve collaboratively with the implementer to address challenges to implementation. The treatment integrity of all intervention steps can be addressed during the session, but the primary focus is on the steps that treatment integrity data indicate have not been implemented or have been challenging to implement consistently. A successful Performance Feedback session will result in the implementer having increased confidence and preparation for implementing the intervention steps.

How to Prepare for Performance FeedbackTo deliver Performance Feedback, some preparation is necessary.

Be sure to review the general guidelines for a PRIME Implementa-tion Support meeting (see Appendix E). Beyond the general guide-lines, prepare the treatment integrity and student outcome data, re-view the intervention plan, and gather necessary materials.

During Performance Feedback, the consultant will review the treatment integrity data and progress-monitoring data with the im-plementer. Therefore, this information – Treatment Integrity Data Across Sessions, Treatment Integrity Data Across Intervention Steps and Progress-Monitoring Data– needs to graphed in a clear manner. Make sure each graph has a title, the axes are labeled, and the infor-mation is readable (see Chapters 5 and 6 for information on collect-ing treatment integrity and progress monitoring data; see Chapter 8 for graphing). Carefully review these graphs prior to the session. Be prepared to be able to summarize the overall findings of each of


143

the graphs, as well as how they related to each other. For example, you may point out a particular week when treatment integrity was low and the corresponding data points on the progress-monitoring graph were below the average level or demonstrated a lower level or declining trend. The implementer may ask questions about the graphs and data, so it is essential that you are able to explain the data. Be prepared to answer questions constructively and support-ively in the session.

In addition, prior to the Performance Feedback session, review the intervention plan. Part of the session will be spent discussing the intervention steps and helping the implementer problem-solve around challenges to implementing specific steps. Have a thorough understanding of the intervention so that you are ready to engage in a meaningful discussion with the implementer.

Last, assemble and organize the materials that will be needed to conduct Performance Feedback. Materials to bring to the meeting include (a) the intervention implementation data, including graphs of treatment integrity across both sessions and intervention steps; (b) the progress-monitoring graph; and (c) the written intervention plan.

How to Deliver Performance Feedback The steps of Performance Feedback are described below. This de-

scription of the purpose of and how to deliver each step is a com-panion to the Performance Feedback protocol and treatment integ-rity measure found in Appendix N.

Step 1: Explain Session PurposeTo begin the Performance Feedback session in a collaborative

manner, describe the purpose of the meeting. In doing so, explain that you are meeting to discus the intervention and its implementa-tion and evaluation of student progress. Then, provide an overview of Performance Feedback. Tell the implementer you will look at data,


144

identify areas that are going well, steps that seem to be challenging, and set goals for the next meeting.

Step 2: Elicit Implementer Feedback About The Intervention Open the Performance Feedback session by asking the implement-

er how intervention implementation is going so far. This discussion presents an opportunity for the implementer to ask questions or raise concerns about the intervention or its implementation. This step will set the stage for an open and collaborative dialogue about the current status of intervention plan implementation and student outcomes. The goal of this step is to shape the session into a con-structive conversation, rather than a one-way sharing of data by the consultant to the implementer.

Step 3: Ask the Implementer About Student ResponsivenessAsk the implementer to share their perspective on and concerns

about student progress and the intervention. Keep the discussion linked to the intervention goals. Address any questions or concerns the implementer has regarding student progress. Similar to the first step, the goal of this step is to understand the implementer’s per-spective and support a collaborative dialogue between the consul-tant and implementer.

Step 4: Review Implementation DataReview two types of intervention implementation data using the

Treatment Integrity Across Sessions and Treatment Integrity Across Intervention Steps Graphs. The goal of this step is to show the im-plementer how implementation is going, both at an overall session level and at the individual intervention step level. In your review of the graphs, identify the steps that the implementer has been con-sistently delivering according to the intervention plan. How often has the implementer been implementing the intervention plan with high levels of treatment integrity? When did low levels of treatment


145

integrity begin and how long has this trend been present? Which intervention steps have been implemented with high levels of treat-ment integrity? Offer specific praise about these consistently imple-mented steps. Next, note the steps that have not been consistently implemented. Use the graphs of these data to help explain the level, trend, and variability of implementation. Transition this conversation into Step 4 by highlighting the connection between implementation of the intervention plan and student progress monitoring. Keep out both treatment integrity graphs to use in the following step.

Step 5: Review Progress-Monitoring DataReview student progress-monitoring data relative to the inter-

vention goal to evaluate the student’s response to the intervention. To do so, show the Progress-Monitoring Graph to the implementer. Note the trend line and aim line. Importantly, make connections be-tween the Progress-Monitoring Graph and the implementation data discussed in Step 3. What is the relationship between implementa-tion data and student outcome data? Do student outcomes improve when implementation is high? Are lower student outcomes present on days when implementation is low? Spend some time reviewing all three graphs with the implementer to think critically about the data and identify important relationships between the implementa-tion data and student outcome data.

Step 6: Review Intervention Steps and Confirm Implementer UnderstandingReview with the implementer the intervention steps he or she

struggles to deliver consistently. Review the steps from the written intervention plan. Describe to the implementer (a) the procedures for implementation, (b) when the steps should be implemented, and (c) the materials needed to implement the steps. Following this re-view, ask the implementer how they feel about the logistical im-plementation of the steps. Make sure the implementer understands


146

how to implement the entire intervention plan with adequate treat-ment integrity. The purpose of this step is to confirm the implement-er understands the logistics of implementing each of these steps.

Step 7: Problem-Solve Strategies for Implementation ImprovementEngage the implementer in a dialogue to identify strategies to

overcome the challenges he or she may be experiencing with in-tervention implementation. What strategies can be used to improve the implementation of steps that have not been implemented con-sistently? Ask the implementer about the barriers to the implemen-tation of each of these steps (e.g., “From your perspective, what challenges are your experiencing completing the behavior chart at the end of each period?” I wonder what might be difficult about giv-ing the student a break in the morning. Tell me more about how that goes for you and the student.”). Actively listen to what the imple-menter is sharing with you. Validate their challenges and concerns with the intervention steps. Collaboratively devise feasible strate-gies to address the challenges to implementation that have been offered by the implementer. After a discussion with the implementer, it may be appropriate to update the Implementation Plan.

Step 8: Confirm Implementer Commitment to Increasing ImplementationEnsure that the implementer is feeling confident to resume im-

plementation of the intervention. Confirm that he or she is feeling prepared to perform the logistical implementation of each step and is ready to increase his or her implementation of the intervention.

Step 9: Close the Session Close Performance Feedback with a positive summary of the dia-

logue during the session. Describe important relationship between high treatment integrity and progress monitoring data. Remind the


147

implementer of the specific steps of the intervention plan that were reviewed in the session and the strategies that were devised to prob-lem-solve challenges encountered in the implementation of these steps. Give the implementer an opportunity to ask questions about the intervention plan, implementation, or other topics discussed at the session. End the meeting by validating the implementer’s ef-forts thus far, reiterating your continued support as the consultant in this process, and confirming the implementer’s commitment to implementation.

PRIME TipThe Performance Feedback session may seem to be a challenging

component of the PRIME model. Here are some important and prac-tical tips from consultants who have implemented the Performance Feedback strategy.

• As much as possible, make Performance Feedback a conver-sation. Engage back and forth to help encourage your under-standing of the implementation challenges and so that your suggestions will be more targeted and useful.

• Focus on support and collaboration – as opposed to providing feedback. Understand that implementation is hard – be there to help, not judge the implementer.

• Use questions to make your points. For example, ask why was implementation challenging on these days? How were oth-er days different? How could this intervention step be more realistic for your classroom?

• Set goals with the implementer. If he or she is really strug-gling with implementation, ask what specific intervention steps he or she could focus on in the week ahead? Focus on these steps and praise the implementer’s success.

• End on a positive note. Summarize the gains and successes the implementer has had during the implementation period.


148

What Did I Learn About PRIME?Performance Feedback is a strategy in the PRIME Model used when

implementer treatment integrity data does not adequately improve with the delivery of Tier 1 and Tier 2 Implementation Supports. During Performance Feedback, the consultant shares treatment in-tegrity and progress monitoring data, reviews particular steps of the intervention plan, and confirms implementer commitment to con-tinued implementation of the intervention plan. The goal of Perfor-mance Feedback is to improve the implementation of the interven-tion steps and help the implementer feel supported and ready to deliver the intervention.

Chapter 14 Key TermsPerformance Feedback


149

Action Plan: The first section of an Implementation Plan, which consists of a struc-tured format for completing detailed logistical planning related to implemen-tation of school-based interventions. In completion of the Action Plan, the consultant and consultee define intervention steps and the plan the logistics of implementing the intervention (i.e., the “who,” “what,” “when,” “where,” “how often,” and “with what” about each intervention step).

Adherence: A dimension of treatment integrity referring to the degree to which specific intervention steps are implemented as planned.

Aim Line: A visual representation on a graph of rate of progress needed for the student to reach the intervention goal by the date set as the end of the prog-ress-monitoring period.

Bar Graph: A type of graph used to illustrate relative comparisons across a variable of interest; data are displayed on bars and a value is assigned to each bar. An example of a bar graph that might be used in PRIME is a graph with the x-axis representing intervention step (i.e., one bar for each intervention step) and the y-axis representing the proportion of intervention sessions in which a step was implemented according to the intervention plan.

Consultant: We use the term “consultant” broadly to refer to any individual who uses a problem-solving consultation model to interact with a consultee (e.g., teacher, parent) to help the consultee provide evidence-based intervention services to a child in a school setting.

Coping Plan: The second section of an Implementation Plan, which consists of a structured format for identifying barriers to intervention implementation and developing strategies to address those barriers. In completion of the Cop-ing Plan, the consultant and consultee proactively identify major barriers to intervention implementation and brainstorm coping strategies to address the identified barriers.

Data-based Decision-making: A process of integrating multiple sources of data (e.g., progress monitoring data, treatment integrity data) to inform decisions

Glossary

Appendix A

APPENDIX A: GLOSSARY

150

about modifying the intervention plan or changing implementation supportsData Path: A line that connects the data points on a graph indicating the changes

from one data point to the next.Data Point: A marked point on a line graph indicating the level of the variable of

interest for a particular session. Direct Observation: A treatment integrity assessment method in which the consul-

tant systematically observes the implementation of the intervention plan and then rates the extent to which he or she observes specific intervention steps.

Direct Training: A Tier 1 Implementation Support intended to increase the im-plementer’s implementation self-efficacy by teaching him or her foundational intervention implementation knowledge and skills. Direct Training consists of didactic training on intervention steps, followed by the consultant demon-strating the intervention, the implementer practicing the intervention, and the consultant providing feedback to the implementer.

Didactic Training: A component of Direct Training in which the implementer reviews each intervention step needed to implement the intervention and gives detailed instructions about how to carry out each step.

Exposure: A dimension of treatment integrity referring to the extent (i.e., frequency or duration) the student receives (i.e., is exposed to) the intervention.

Health Action Process Approach (HAPA): A theory of adult behavior change from the health psychology literature that provides the theoretical and empirical basis for PRIME.

Implementation Beliefs Assessment: A self-report measure to indicate an imple-menter’s perceptions of the intervention (i.e., outcome expectations) and his or her ability to implement (i.e., self-efficacy)

Implementation Planning: Implementation Planning is a Tier 1 Implementation Support for increasing preparation for implementation and the Implementa-tion Plan is a tool for completing detailed logistical planning of intervention implementation and identifying and problem-solving significant barriers to implementation.

Implementation Self-efficacy: One’s confidence in being capable of performing a dif-ficult or novel behavior. Implementation Self-Efficacy is one variable measured by the Implementation Beliefs Assessment.

Implementer: The person responsible for delivering an intervention. Indirect Service Delivery: A service delivery model in which a consultant (e.g., a

school psychologist) supports another implementer (e.g., a teacher, a parent) who implements the intervention plan with the student; PRIME is designed to be delivered within an indirect problem-solving model.

Intervention: A prevention, treatment, educational, and/or service curriculum, practice, or program that is typically implemented in school settings. An evi-dence-based intervention is one which data from research studies generally


151

support the efficacy or effectiveness of the intervention; also commonly referred to as research-based interventions or empirically supported interventions.

Intervention Step Treatment Integrity: Intervention Step Treatment Integrity is cal-culated separately for each intervention step. It is a measure of the percentage of intervention sessions that the consultee delivered a particular intervention step.

Intervention Evaluation: The fourth stage of a problem-solving model during which the consultant and implementer meet to discuss overall student progress and evaluate the intervention after implementation.

Intervention Implementation: The third stage of a problem-solving model during which the implementer begins to deliver the intervention to support the stu-dent.

Intervention Step: The discrete behaviors the implementer performs to deliver the intervention. An intervention plan usually includes several intervention steps.

In-vivo: Occurring during typical intervention implementation (i.e., in setting with target students)

Level: On a graph of progress monitoring or treatment integrity data, level is the average value of the measured outcome within a condition summed across all the data point within that condition.

Line Graph: A type of graph used to summarize data across time with time plotted along the x-axis (i.e., dates, sessions) and the extent of the variable of interested plotted on the y-axis (e.g., treatment integrity, progress monitoring, IBA data). An example of a bar graph that might be used in PRIME is a graph with the x-axis representing intervention step (i.e., one bar for each intervention step) and the y-axis representing the proportion of intervention sessions in which a step was implemented according to the intervention plan.

Maintenance Self-efficacy: A concept from the Volitional Phase of the Health Action Process approach, maintenance self-efficacy is one’s confidence in being capable of keeping up a difficult behavior. Maintenance self-efficacy is one variable measured by the Implementation Beliefs Assessment.

Motivational Consulting: A Tier 2 Implementation Support strategy that involves reviewing the student intervention goals and using Motivational Interviewing techniques such as change talk and positive regard to increase the implement-er’s motivation to increase his or her implementation across time.

Multi-tiered Systems of Supports: A service delivery framework, such as Re-sponse-to-Intervention and Positive Behavior Intervention and Supports, in which students are provided instructional and intervention supports along a continuum of intensity based on results from universal screening and prog-ress-monitoring data.

Outcome Expectations: A concept from the Motivational Phase of the Health Action Process Approach, outcome expectations are one’s beliefs about the positive


.

152

and negative outcomes of alternative behaviors. Outcome Expectations is one variable measured by the Implementation Beliefs Assessment.

Participant Modeling: A Tier 2 Implementation Support strategy that involves reviewing intervention steps and then modeling, guided practice, and indepen-dent practice within the implementation context (e.g., classroom).

Performance Feedback: A Tier 3 Implementation Support Strategy that involves a meeting between the consultant and implementer to discuss treatment integ-rity and progress monitoring data, review difficult implementation steps, and collaboratively problem solve to address challenges to implementation.

Permanent Products: Permanent products are products naturally created through intervention implementation. Permanent product review is a treatment integrity assessment method that involves the review of permanent products to deter-mine the degree to which the intervention steps were implemented.

Phase Change Line: A vertical line transposed on a graph to indicate when the data are collected during different conditions, or phases; the most common phase change line is between baseline data collection and intervention imple-mentation.

PRIME (Planning Realistic Intervention Implementation and Maintenance by Educators): A multi-tiered framework of implementation supports to plan for the implementa-tion of any school-based intervention.

Problem Analysis: The second stage of a problem-solving model during which the consultant reviews the baseline data to develop a hypothesis for the current level of functioning or behavior, develops an intervention plan, and establishes plans for progress monitoring and treatment integrity data collection and review with the implementer.

Problem Identification: The first stage of a problem-solving model during which the consultant and implementer identify the area of primary concern through interviews and progress monitoring data.

Progress Monitoring: Frequent collection of data on the target skills that indicate how the student is progressing toward the goals of the intervention.

Raising Awareness: A Tier 2 Implementation Support talking strategy that focus-es on the implementer’s perception of the intervention, its effectiveness and implementation.

Recovery Self-efficacy: A concept from the Volitional Phase of the Health Action Process approach, recovery self-efficacy is one’s confidence in being capable of resuming a difficult behavior after an interruption. Recovery self-efficacy is one variable measured by the Implementation Beliefs Assessment.

Role Play: A Tier 2 Implementation Support Strategy that incorporates modeling and practice of several intervention implementation scenarios outside of the implementation context.

Self-efficacy: A concept in the Health Action Process approach, self-efficacy is one’s


PRIME (Planning Realistic Intervention Implementation and Maintenance by Educators): A multi-tiered framework of implementation supports to plan for the implementation of any school-based intervention.

Problem Analysis: The second stage of a problem-solving model during which the consultant reviews the baseline data to develop a hypothesis for the current level of functioning or behavior, develops an intervention plan, and establishes plans for progress monitoring and treatment integrity data collection and review with the implementer.

Problem Identification: The first stage of a problem-solving model during which the consultant and implementer identify the area of primary concern through interviews and progress monitoring data.

Progress Monitoring: Frequent collection of data on the target skills that indicate how the student is progressing toward the goals of the intervention.

153

confidence in their ability to act in a certain way to achieve certain goals. Self-report: A treatment integrity assessment method in which the implementer

rates the extent of implementation of the intervention steps on a checklist or form throughout or after an intervention session.

Session Treatment Integrity: The proportion of intervention steps delivered during each intervention session.

Summary Statement: A statement used to help interpret graphs describing the overall pattern of data highlighting the trend, level, and variability.

Tier 1 Implementation Supports: Tier 1 Implementation Supports are proactive and feasible Implementation Support strategies that can be delivered together be-fore intervention implementation begins to facilitate high levels of implemen-tation; Tier 1 Implementation Supports are Direct Training and Implementation Planning.

Tier 2 Implementation Supports: Tier 2 Implementation Supports are a series of four strategies designed to increase intervention implementation knowledge, fluency, and/or motivation for select implementers who struggle to implement adequately after Tier 1 supports. Tier 2 Implementation Supports are Partici-pant Modeling, Role Play, Raising Awareness, and Motivational Consulting.

Tier 3 Implementation Supports: Tier 3 Implementation Supports are designed to increase intervention implementation of those few implementers who require ongoing, intensive implementation support. The Tier 3 Implementation Support is Performance Feedback.

Treatment Integrity: The extent to which an intervention is implemented as planned. Treatment integrity data are collected throughout the plan implemen-tation period.

Trend: On a graph of progress-monitoring or treatment integrity data, trend is the pattern of change in the data over a period of time (e.g., before intervention, after intervention implementation), such as increasing, decreasing, or stable trends in the data.

Trend Line: A visual representation on a graph of the actual rate of progress, indicating how quickly the student is improving per the progress monitoring measure or rate of treatment integrity across time.

Variability: On a graph of progress monitoring or treatment integrity data, variabili-ty is the spread of data above or below the data path.

Quality: A dimension of treatment integrity referring to how well the intervention steps are delivered as planned.

X-axis: The horizontal axis on a graph representing time (e.g., dates of assessment, intervention sessions, weeks, months) or another variable that is measured repeatedly over time (e.g., intervention steps, IBA items).

Y-axis: The vertical axis on a graph representing the percentage or extent to which the outcome variable of interest is evident.


154

Chapter 1: Introduction to PRIME Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treat-

ment integrity: Introduction to the special series. School Psychology Review, 38, 445-459.

Sanetti, L. M. H., & Kratochwill, T. R. (2013a). Treatment Integrity: A foundation for evidence-based practice in applied psychology. Washington DC: American Psycho-logical Association.

Schwarzer, R. (1992). Self-efficacy in the adoption and maintenance of health be-haviors: Theoretical approaches and a new model. In R. Schwarzer (Ed.), Self-ef-ficacy: Thought control of action (pp. 217-243). Washington, DC: Hemisphere.

Schwarzer, R. (2008). Modeling health behavior change: How to predict and modify the adoption and maintenance of health behaviors. Applied Psychology: An International Review, 57(1), 1-29.

Chapter 2: How to Use PRIMEBergan, J. R., & Kratochwill, T. R. (1990). Behavioral consultation in applied settings.

New York: Plenum.Erchul, W. P., & Martens, B. K. (Eds.). (2010). School consultation: Conceptual and

empirical bases of practice. New York: Springer.Kratochwill, T.R. (2008). Best practices in school-based problem-solving consul-

tation: Applications in prevention and intervention systems. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 5, pp. 1673-1688). Bethseda, MD: National Association of School Psychologists.

Sanetti, L. M. H., Fallon, L. M., & Collier-Meek, M. A. (2011). Treatment integrity assessment and intervention by school-based personnel: Practical applications based on a preliminary study. School Psychology Forum, 5, 87-102.

Sanetti, L. M. H., & Kratochwill, T. R. (2013b). Treatment integrity assessment within the problem-solving model. In R. Brown-Chidsey (Ed.), Assessment for interven-tion: A problem-solving approach (2nd ed.) (pp. 297-320). New York: Guilford

Selected References

Appendix B

155

Publications, Inc.Sanetti, L. M. H., Kratochwill, T. R., & Long, A. C. J. (2013). Applying adult behavior

change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47-62.

Chapter 3: Direct TrainingOwens, S. H., Fredrick, L. D., & Shippen, M. E. (2004). Training a paraprofessional to

implement spelling mastery and examining its effectiveness for students with learning disabilities. Journal of Direct Instruction, 4, 153-172.

Sterling-Turner, H. E., Watson, T. S., & Moore, J. W. (2002). The effects of direct train-ing and treatment integrity on treatment outcomes in school consultation. School Psychology Quarterly, 17, 47-77.

Zins, J. E., & Ponti, C. R. (1996). The influence of direct training in problem solving on consultee problem clarification skills and attributions. Remedial and Special Education, 17, 370-376.

Chapter 4: Implementation PlanningSanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Kim, J. S., & Kratochwill, T. R. (in

press). Using implementation planning to increase teachers’ adherence and quality to behavior support plans. Psychology in the Schools.

Sanetti, L. M. H., Collier-Meek, M. A., Long, A. C. J., Byron, J. R., & Kratochwill, T. R. (2014). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Manuscript submitted for publica-tion.

Sanetti, L. M. H., Kratochwill, T. R., & Long, A. C. J. (2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47-62.

Chapter 5: Treatment Integrity Collier-Meek, M. A., Fallon, L. M., Sanetti, L. M. H., & Maggin, D. M. (2013). Focus on

implementation: Strategies for problem-solving teams to assess and promote treatment fidelity. Teaching Exceptional Children, 45, 52-59. Retrieved from: http://cec.metapress.com/content/p72233487l54536g/

DiGennaro Reed, F. D., & Codding, R. S. (2014). Advancements in procedural fidelity assessment and intervention: Introduction to the special issue. Journal of Behavioral Education, 23, 1-18.

Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., et al. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87-106.

Roach, A. T., & Elliot, S. N. (2008). Best practices in facilitating and evaluating

APPENDIX B: SELECTED REFERENCES

156

intervention integrity. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 2, pp. 195-208). Bethseda, MD: National Association of School Psychologists.

Sanetti, L. M. H., & Kratochwill, T. R. (2009a). Toward developing a science of treat-ment integrity: Introduction to the special series. School Psychology Review, 38, 445-459.

Sanetti, L. M. H., & Kratochwill, T. R. (2013a). Treatment Integrity: A foundation for evidence-based practice in applied psychology. Washington DC: American Psy-chological Association.

Chapter 8: Progress MonitoringBriesch, A.M., & Volpe, R.J. (2007). Important considerations in the selection of

progress-monitoring measures for classroom behaviors. School Psychology Forum: Research in Practice, 1, 59-74.

Chafouleas, S. M., Riley-Tillman, T. C., & Sugai, G. M. (2007). Introduction to school-based behavioral assessment. In School-based Behavioral Assessment: Informing intervention and instruction (pp. 1–10). New York, NY: The Guilford Press.

Hixson, M., Christ, T.J., & Bradley-Johnson, S. (2008). Best practices in the analysis of progress-monitoring data and decision making. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 6, pp. 2133-2146). Bethseda, MD: National Association of School Psychologists.

Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 2, pp. 141-157). Bethseda, MD: National Association of School Psychologists.

Chapter 7: Implementation Beliefs AssessmentSanetti, L. M. H., Kratochwill, T. R., & Long, A. C. J. (2013). Applying adult behavior

change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47-62.

Long, A. C. J., Sanetti, L. M. H., Neugebauer, S., & Kratochwill, T. R. (2014) Imple-mentation beliefs assessment: Measuring implementers’ outcome expectan-cies and self-efficacy. Manuscript in preparation.

Chapter 8: Graphing and Interpreting DataHood, C., & Dorman, D. (2008). Best practices in the display of data. In A. Thom-

as, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 6, pp. 2117-2132). Bethseda, MD: National Association of School Psychologists.

McDougal, J.L., LeBlanc, M., & Hintze, J. (2010). Graphing student academic data for universal screening and progress monitoring. Helping Children at Home and


157

School III. Bethseda, MD: National Association of School Psychologists.

Chapter 9: Using Data to Target Implementation Supports in a Multi-Tiered ModelChafouleas, S. M., Riley-Tillman, T. C., & Sugai, G. M. (2007). Introduction to

school-based behavioral assessment. In School-based Behavioral Assessment: Informing intervention and instruction (pp. 1–10). New York, NY: The Guilford Press.

Hintze, J. M., & Marcotte, A. M. (2010). Student assessment and data-based de-cision making. In T. A. Glover, & S. Vaughn (Eds.), The promise of response to intervention: Evaluating current science and practice (pp. 57-77). New York, NY: Guilford Press.

Hixson, M., Christ, T.J., & Bradley-Johnson, S. (2008). Best practices in the analysis of progress-monitoring data and decision making. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 6, pp. 2133-2146). Bethseda, MD: National Association of School Psychologists.

Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill improvement. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology V (Vol. 2, pp. 141-157). Bethseda, MD: National Association of School Psychologists.

Chapter 10: Role PlayBylund, C. L., Brown, R. F., di Ciccone, B. L., Levin, T. T., Gueguen, J. A., Hill, C., & Kis-

sane, D. W. (2008). Training faculty to facilitate communication skills training: Development and evaluation of a workshop. Patient Education and Counseling, 70, 430-436.

Kratochwill, T. R., Sheridan, S. M., Rotto, P. C., & Salmon, D. (1991). Preparation of school psychologists to serve as consultants for teachers of emotionally disturbed children. School Psychology Review, 20, 530-550.

Lerman, D. C., Vorndran, C. M., Addison, L., & Contrucci Kuhn, S. (2004). Preparing teachers in evidence-based practices for young children with autism. School Psychology Review, 33, 510-526.

McNaughton, D., Hamlin, D., McCarthy, J., Head-Reeves, D., & Schreiner, M. (2008). Learning to listen: teaching an active listening strategy to preservice educa-tion professionals. Topics in Early Childhood Special Education, 27, 223-231.

Trevisan, M. S. (2004). Practical training in evaluation: A review of the literature. American Journal of Evaluation, 25, 255-272.

Chapter 11: Participant ModelingBandura, A., Blanchard, E.B., & Ritter, B. (1969). Relative efficacy of desensitization

and modeling approaches for inducing behavioral, affective, and attitudinal


158

change. Journal of Personality and Social Psychology, 13, 173-199. Hughes, D. (2002). Participant modeling as a classroom assessment. In R.A. Griggs

(Ed.), Handbook for teaching introductory psychology: Vol. 3. With an emphasis on assessment (pp. 239-241). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Minor, S. W., & Leone, C., & Baldwin, R. T. (1984). A comparison of in-vivo and ima-ginal participant modeling. Journal of Clinical Psychology, 40, 717-720.

Pridham, K.F., Limbo, R., Schroeder, M., Thoyre, S., & Van Riper, M. (1998). Guided participation and development of care-giving competencies for families of low birth-weight infants. Journal of Advanced Nursing, 28, 948-958.

Romi, S. & Teichman, M. (1995). Participant and symbolic modelling training programmes: changes in self-efficacy of youth counsellors. British Journal of Guidance and Counseling, 23, 83-94.

Romi, S., & Teichman, M. (1998). Participant modelling training programme: Tutoring the paraprofessionals. British Journal of Guidance and Counselling, 26, 297-301.

Chapter 12: Raising AwarenessBernard, M.E., & DiGiuseppe, R. (2000). Advances in the theory and practice of

rational-emotive behavioral consultation. Journal of Educational and Psycholog-ical Consultation, 11, 333-355.

Bowman, P., & Goldberg, M. (1983). Reframing: A tool for the school psychologist. Psychology in the Schools, 20, 210-214.

Forman, S.G., & Forman, B.D. (1978). A rational-emotive therapy approach to con-sultation. Psychology in the Schools, 15, 400-406.

Maag, J.W. (2008). Rational-emotive therapy to help teachers control their emo-tions and behavior when dealing with disagreeable students. Intervention in School and Clinic, 44, 52-57.

Nucci, C. (2002). The rational teacher: Rational emotive behavior therapy in teacher education. Journal of Rational-Emotive & Cognitive-Behavior Therapy, 20, 15-32.

Chapter 13: Motivational ConsultingGaume, J., Gmel, G., Faouzi, M., & Daeppen, J. (2009). Counselor skill influences

outcomes of brief motivational interventions. Journal of Substance Abuse Treat-ment, 37, 151-159.

Herman, K. C., Reinke, W. M., Frey, A. J., & Shepard, S. A. (2014). Motivational inter-viewing in schools: Strategies for engaging parents, teachers, and students. New York, NY: Springer Publishing Company.

Miller, W. R., Moyers, T. B., Ernst, D., & Amrhein, P. (2008). Motivational Interviewing Skills Code Manual (2.1). Retrieved September 30, 2010 from casaa.unm.edu/download/misc.pdf.


159

Miller, W. R., & Rollnick, S. (2002). Motivational Interviewing: Preparing people for change (2nd ed.). New York, NY: The Guilford Press.

Moyers, T. B., Martin, T., Manuel, J. K., Hendrickson, S. M. L., & Miller, W. R. (2005). Assessing competence in the use of motivational interviewing. Journal of Sub-stance Abuse Treatment, 28, 19-26.

Rosengren, D. B. (2009). Building motivational interviewing skills: A practitioner workbook. New York, NY: The Guilford Press.

Chapter 14: Performance FeedbackFallon, L. M., Collier-Meek, M. A., Maggin, D. M., Sanetti, L. M. H., & Johnson, A. H. (in

press). Is performance feedback an evidence-based intervention? A systematic review and evaluation of single-case research. Exceptional Children.

Noell, G. H., Witt, J. C., Gilbertson, D. N., Ranier, D. D., & Freeland, J. T. (1997). In-creasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77-88.

Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., … Duhon, G. J. (2005). Treatment implementation following behavioral consultation in schools: A comparison of three follow-up strategies. School Psychology Review, 34, 87-106.

Sanetti, L. M. H., Fallon, L. M., & Collier-Meek, M. A. (2013). Performance feedback provided by school personnel to increase teachers’ intervention implemen-tation: An evaluation of effectiveness, procedural integrity, and feasibility. Psychology in the Schools, 50, 134-150.

Solomon, B. G., Klein, S. A., & Politylo, B. C. (2012). The effect of performance feedback on teachers’ treatment integrity: A meta-analysis of the single-case literature. School Psychology Review, 41, 160-175.


160

Treatment IntegrityWhat is treatment integrity? In the most general sense, treatment integrity refers to the degree to

which a treatment is implemented as planned. Many contemporary concep-tual models of treatment integrity represent it as a multidimensional con-struct. Commonly, these models consider the adherence (what intervention steps were implemented), quality (how well were steps implemented), and exposure (how much of each intervention step was delivered) of intervention implementation.

Why should I be concerned about treatment integrity? The importance of knowing exactly how school-based interventions are

implemented cannot be overstated. When it is assumed that an intervention was implemented as planned, and it was not, the likelihood of making an incorrect conclusion about the effectiveness of the intervention is high. De-viations from the intervention plan may negatively impact student outcomes. The concept of treatment integrity is central to the daily implementation of interventions by school personnel.

How is treatment integrity related to student outcomes?Treatment integrity can be directly related to student outcomes. When in-

terventions are implemented as planned, or with a high level of treatment integrity, there is a better chance that the intervention help the student reach their intervention goal. When interventions are not implemented as planned, or with a low level of treatment integrity, there is a lower chance that the intervention will produce the intended positive student outcomes. Addition-ally, if interventions are being implemented as planned but the student is not

Frequently Asked Questions

Appendix C

161

making progress, teams are able to make sound decisions regarding chang-ing interventions or the intensity of interventions.

The implementer is uncomfortable with assessing her treatment integrity. What can I do?

Discomfort with implementation assessment can be a common situation when the concept of treatment integrity is not openly discussed or not dis-cussed in a supportive manner. Talking about treatment integrity openly from the beginning of consultation or discussing the concept at the beginning of the year with all teachers, such as at professional development meetings, can be a good first step toward making implementers more comfortable with treatment integrity. Additionally, explaining that evaluating treatment integ-rity is another way to make sure interventions are working the way they are supposed to and linking it to student outcomes can be helpful.

PRIMEHow can I use PRIME to support my school’s RTI process?In addition to monitoring student outcomes through the Response-to-In-

tervention (RTI) process, monitoring treatment integrity is another piece to making appropriate data based decisions about the appropriate level of in-tervention supports. Students should not be moved between tiers unless there are treatment integrity data to support that interventions are indeed being implemented the way they were intended. If the intervention is be-ing implemented with adequate treatment integrity and students are still not making adequate progress, then they should be considered for receiving more intensive supports and “move up the triangle.” However, if treatment integrity is low and a student is not making adequate progress, then appro-priate decisions cannot be made about changing interventions. In this case, implementers should be supported with their implementation through the PRIME Model. The intervention may produce positive results for the student if it was fully implemented. See Chapter 9 for further discussion on integrat-ing treatment integrity and progress-monitoring data.

How much time will PRIME take? Are there efficient ways to use PRIME?

Like any new system, PRIME will take time to set up at the beginning. Fa-miliarizing stakeholders with treatment integrity, setting up systems of data

APPENDIX C: FAQ

162

collection, deciding how to measure treatment integrity, and familiarizing yourself with the different PRIME supports available to implementers may seem like a daunting task. But once in place, measuring treatment integri-ty and using PRIME will become more efficient and routine. As you begin, remember that not all teachers require intensive implementation supports and data on treatment integrity can be collected in a variety of ways, and not all by the same person.

How can I describe PRIME to my administrators to support their buy-in?

There are a variety of ways to approach describing PRIME to administra-tors. PRIME will promote efficiency within multi-tiered systems of supports. Looking at student outcomes as well as treatment integrity data will aid in making appropriate data-based decisions. Further, taking and using treat-ment integrity data is not only best practice, but is also either implicitly or explicitly required by law in several states. In addition, PRIME can be de-scribed as a way to support student outcomes, as well as improve imple-menter confidence and skills in delivering interventions.

How can I describe PRIME to parents?PRIME can be explained as a system of supports that aims to train imple-

menters in appropriately delivering evidence-based interventions. Delivering interventions as they were intended promotes better student outcomes in academics and behavior. However, if students are not making gains despite the implementer delivering the intervention appropriately, the data collect-ed through PRIME will allow the team to make data-based decisions about what are appropriate next steps for supporting the student. The bottom line is that implementation data is designed to provide support to implementers and ensure students receive interventions as designed.

How can I explain the PRIME findings to a team? A discussion of treatment integrity, what it means for student outcomes,

and how to support implementation to achieve sufficient treatment integrity should take place. Treatment integrity graphs can be used to illustrate imple-mentation alongside student outcomes (see Chapter 8 regarding graphing and analyzing data). Graphs can be especially powerful when first explaining the importance of treatment integrity when student outcomes are low and implementation is also low, but after PRIME Implementation Supports are

APPENDIX C: FAQ

163

offered, both treatment integrity and student outcomes increase. The com-bination of treatment integrity and student outcome graphs are valuable in making data-based decisions within a team and treatment integrity should always be considered when changes in interventions are made.

How do I decide between the different PRIME Implementation Supports?

The use of the different PRIME Implementation Supports depends on the level of support a teacher needs. Tier 1 supports, such as Direct Training and/or Implementation Planning, should be offered when first introducing an intervention to an implementer. These supports should provide most im-plementers with the skills and confidence needed to sufficiently implement the intervention as planned. If treatment integrity is low after Tier 1 sup-ports are delivered, more intensive supports may be needed. Decisions about what supports are most appropriate to use with a particular implementer are based on a variety of implementer characteristics. To learn more about how to choose supports, see Chapter 9 of the PRIME Manual, which discusses data-based decision making in great detail.

What are the major advantages and disadvantages to using PRIME?There are always advantages and disadvantages to implementing a new

system. PRIME can be used to support implementers in their delivery of ev-idence-based interventions, which will facilitate better student outcomes. It also helps teams to make appropriate decisions about levels of intervention support. To start, PRIME could be considered time consuming as there will be more data to collect, and more graphs to make and interpret. This can seem like a pretty resource- and time-intensive activity. However, these data are essential to making appropriate decisions for students, and collecting treat-ment integrity data in schools is required by many states. Additionally, the focus on supporting and promoting implementation will likely lead to a de-crease in developing more intensive interventions that may not be necessary.

Where can I learn more about PRIME? You can find out more about PRIME including the research, PRIME mate-

rials, and people involved at the website: http://www.primeimplementation.com/ There you may also email PRIME Manual authors for additional infor-mation.

APPENDIX C: FAQ

164

PRIME Tracking Form

Appendix D

PRIME Intervention Implementation Tracking Form

Student(s) Name: _______________________________ Grade(s): ____________________

Implementer Name: ____________________ Consultant Name: ____________________

Target Student Problem & Goal: _________________________________________________

Evidence-based Intervention: _______________Initial Implementation Date: ________________

Data Review Date:

Team Members Present (Initials):

Summary Statements:Applicable Scenario*:

Action Steps –Action/Supports Provided, Person Responsible, Date

Comments:Progress Monitoring

Treatment Integrity

On track to meet goal

Not on track to meet goal

Sufficient implementation

Insufficient implementation













* Refer to PRIME Data-Based Decision Making Worksheet for scenario & related action steps

Planning Realistic Implementation and Maintenance by Educators

PRIME

166

General Strategy Guide

Appendix E

GENERAL PRIME IMPLEMENTATION SUPPORT GUIDEAlthough each PRIME Implementation Support meeting requires

individualized preparation, delivery, and follow up, there are some general guidelines that can be followed for any PRIME Implemen-tation Support meeting. To increase collaboration and increase the productivity of meetings, consider the items /complete the tasks outlined below.

Before the Meeting:Identify the Meeting Logistics

•Decide who will be involved in the strategy meeting (e.g., primary implementer, potential collaborators). In some cases, there is more than one person involved in implementing the intervention (e.g., paraprofessionals, school support profes-sionals) or other stakeholders (e.g., case worker, parent) may be interested in attending.

•Determine when to complete the meeting, for how long the meeting will last, and how many sessions are needed to com-plete the strategy.

•Arrange where you will meet. Consider what setting is most appropriate for the strategy and the skills to be practiced. For instance, do you need a quiet space or will you use the target setting? Will you be able to complete didactic training and/or in-vivo practice when necessary?

167

Prepare and Review Implementation Support, Intervention, and Data•Read the specific implementation support chapters and protocol

to (a) familiarize yourself with steps and talking points and (b) identify materials to be created or information to be gathered. Confirm understanding of the purpose of the strategy and each of the steps. Prepare any necessary information or mate-rials (e.g., graphs, reports, written intervention plan, research).

•Preplan your dialogue with the implementer based on treat-ment integrity, progress- monitoring, and IBA data. This preparation will help you to maximize the strategy session to address and support the individual implementer’s concerns and needs. When applicable, (a) break the intervention into a written list of teachable steps, and arrange them into groups for the purpose of demonstration and practice; (b) identify target steps for practice based on treatment integrity and/or IBA; or (c) brainstorm potential scenarios to use for practicing target steps.

•Review the intervention plan and goals for intervention. Make sure you feel fluent in the intervention plan before preparing for the strategy meeting.

•Check progress-monitoring, treatment integrity, and IBA data. Use the graphs and summary statements (see Chapter 10) as well as the PRIME Intervention Implementation Tracking Form (see Appendix X).

•Reach out to the implementer. If you think it would be help-ful for the implementer to review any materials prior to the strategy meeting, provide those in advance (e.g., written list of intervention steps, sample intervention materials, research, data).

If the specific strategy requires an updated Implementation Be-liefs Assessment, ask the implementer to fill this out ahead of time or allow extra time at the beginning of the meeting session to ad-minister and score.

APPENDIX E: GENERAL STRATEGY GUIDE

168

Develop a goal •Identify a goal for the PRIME Implementation Support meet-

ing. To do so, review the treatment integrity data, IBA data, and intervention goal as well as the purpose of each PRIME Implementation Support. For example, you may identify a goal related to increasing implementation of a particular interven-tion step or related to a subscale of the IBA data. Target your efforts to support or increase this specific goal. If appropriate, share this goal with the implementer.

During the Meeting:•Be ready. Come to the meeting on time and bring all needed

materials.•Follow the PRIME Implementation Support protocol. To do

so, complete each intervention step and use talking points to guide the session.

•Throughout, work collaboratively with implementer to help fa-cilitate their treatment integrity. Be sure to answer questions and address any concerns. If you are not sure of an answer, promise to follow up with the implementer as soon as possi-ble after the meeting.

•Keep the goal for the PRIME Implementation Support meeting in mind. Work to target and address the goal throughout.

After the Meeting:•Complete the PRIME Implementation Support treatment

integrity measure to evaluate your delivery of the implemen-tation support (see Appendices F-L). Keep the completed Implementation Support treatment integrity measure with the PRIME Intervention Implementation Tracking Form to contin-ue to document your support efforts.

•Follow up with the implementer to ensure that he or she feels


169

that the goals of the meeting have been met and his or her questions have been answered. Confirm that implementer knows how to contact you with any questions about the inter-vention or its implementation.

•If any changes to the Implementation Plan were made during the PRIME Implementation Support, update the Implementa-tion Plan to reflect those changes. Share the updated Imple-mentation Plan with the implementer as soon as possible.


170

Direct Training Strategy

Appendix F

DIRECT TRAININGDirect Training aims to increase implementer’s implementation

self-efficacy by teaching her/him foundational intervention imple-mentation skills. Through detailed training, modeling, practice, and feedback, the implementer will build knowledge and positive expe-riences with the intervention. Completion of Direct Training should improve the implementer’s positive expectations about intervention effectiveness and implementation success.

Preparing for Direct Training •Review general guidelines for preparing for an Implementa-

tion Support meeting and complete necessary planning steps (see Appendix E).

•Prepare for Direct Training to take one session outside of the implementation setting.

•Break down the intervention into teachable intervention steps. •Decide how to proceed through the Direct Training steps based

on the specific intervention and treatment integrity data (if available). You may go through the didactic training, modeling, practice, and feedback (steps 2 to 10) for the entire interven-tion or only for individual or chunked intervention steps and then go back through this sequence for the next intervention step(s). You may also find that it is not appropriate to model and practice specific intervention steps (e.g., posting a sign).

171

Making the decision among these options may depend on the number and complexity of intervention steps, or the theoretical links between intervention steps (e.g., if several steps are based on one principle, if intervention steps build on one another). Also con-sider how the intervention is implemented (e.g., all steps delivered at once, different steps provided at separate times) or if the teacher struggles with particular intervention steps (based on treatment in-tegrity data).

MATERIALS: •A written list of intervention steps broken out into teachable

steps. •Any items needed to practice the intervention (e.g., forms,

training manual).

Steps1. Explain session purpose

2. Didactic intervention training

Talking Points•Explain that you are meeting to look at the intervention and practice its implementation.

•Provide an overview of Direct Training by briefly describing steps including review of the intervention, modeling, prac-tice and feedback.

•Discuss and collaboratively develop goals for Direct Train-ing. These might include increasing the implementers’ implementation skills and confidence. Highlight how the steps of Direct Training will help meet the session goals.

•Provide an overview of the intervention, its purpose in supporting student outcomes and a rationale for its effec-tiveness.

•Review each skill/step needed to implement the interven-tion, providing detailed instructions on how to carry out each skill/step, including any intervention materials need-ed. Make sure to emphasize (a) why each step is important and what it accomplishes, and (b) any relevant research support, as appropriate.

APPENDIX F: DIRECT STRATEGY TRAINING

172

3. Answer implementer’s questions

4. Demonstrate intervention

5. Engage the implementer in guided practice

6. Provide feedback about the practice

7. Repeat guided practice and feedback, if necessary

•Throughout, encourage the implementers’ active involve-ment by asking questions about implementation, use of the step, and answering any questions.

•Ask the implementer if he/she has any questions or concerns about the intervention or its implementation.

•Address these questions and concerns the best as you can based on intervention research and your experience.

•Demonstrate intervention components. During your demonstration, you may simply demonstrate delivering the intervention as planned or you may describe what you are doing. If you describe your actions, be sure to note aspects of implementation related to adherence (i.e., de-livering the intervention as planned) and quality (i.e., how you are delivering intervention components).

•Have the implementer practice the intervention and provide supportive guidance as needed. Guidance may include additional explanations of intervention steps, prompts, hints, guiding questions, answering implementer questions, and encouragement.

•Provide feedback about the guided practice. Give specific (e.g., detailed) feedback in a positive and constructive manner. Be sure to reinforce successes and correct any implementation errors.

•If needed, repeat steps 5 and 6 until the implementer successfully and confidently implements each component of the intervention.


173

8. Implementer engages in independent practice


10. Repeat independent practice and feedback, if necessary

11. Close the session

•Have the implementer independently practice all of the intervention or grouped intervention steps.

•Do not provide any guidance during the independent practice, but note areas of strength during implementa-tion as well as areas for improvement.

•Ask the implementer to self-evaluate their independent practice.

•Provide constructive feedback regarding the implement-er’s independent practice. Be sure to reinforce successes and correct any implementation errors.

•If needed, repeat steps 8 and 9 until the implementer successfully and confidently implements each component of the intervention independently.

•Revisit the consultation goals and evaluate if those goals have been met through Direct Training.

•Ask if the implementer has any questions. •Provide positive feedback to the implementer about her/his participation in Direct Training.


174

Direct Training Treatment Integrity Protocol

Appendix F

To ensure PRIME Implementation Supports are delivered as planned, consultants should evaluate the treatment integrity of their delivery. Use the key below to rate the (a) adherence, (b) quality, and (c) implementer responsiveness on the Treatment Integrity Data Sheet (below) during or immediately following provision of a strategy.

Adherence is the degree to which the strategy steps are imple-mented as planned. To rate adherence, circle the descriptor that best describes how completely each step was delivered.

Complete All aspects completed (100%) Substantial More than half of aspects completed (99-51%) Limited Less than half of aspects completed (50-1%)None No aspects completed (0%)

Quality refers to how well the strategy steps are implemented. Quality can be evaluated only if the step was implemented; rate on those steps for which adherence was rated as complete, sub-stantial, or limited. To rate quality, circle the descriptor that best describes how well each step was delivered.

Excellent Step was implemented skillfully as indicated by: • Appropriate interaction and specificity, • Step smooth, • Appropriately paced, • Competently implemented (e.g., clearly responsive to teacher’s unique needs)APPENDIX F: PROTOCOL

175

Good Step implemented adequately, but in a less skillful manner; step somewhat flawed in at least 1 of the i Indicators under “excellent”Fair Step implemented poorly in a manner that is inadequate or seriously flawed in at least 1 OR somewhat flawed in at least 2 of the indicators under “excellent”Poor Step implemented poorly, with none of the indicators under “excellent”

Implementer Responsiveness refers to how actively engaged and cooperative the implementer was during the PRIME Implementa-tion Support session. Two items related to implementer respon-siveness are rated at the end of the session based on the percent-age of time the implementer demonstrated these characteristics per the definitions below.

Actively Engaged The implementer is purposefully participating in the intervention process. Examples include: Note taking, reading materials, intently listening, asking questions, nodding head, vocalizing understanding/interest (e.g., “okay”), making affirmative statements (e.g., “I will…”) Non-examples include: Looking out the window, distracted by things unrelated to the current task, checking the clockCooperated The implementer willingly and agreeably working jointly with the consultant during the intervention process. Examples include: Reviewed presented data, actively participated in role plays, followed through with tasks asked of them Non-examples include: Refusal to partici- pate in intervention step(s), lacked elabora- tion when asked questions

APPENDIX F: PROTOCOL

DIRECT TRAINING: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

DIRECT TRAINING: Treatment Integrity Data Sheet

*Only complete if adherence step is rated complete, substantial, or limited

Implementer Consultant Date Start Time End Time

Strategy Steps Adherence Quality* Complete Substantial Limited None NA Excellent Good Fair Poor

1. Explain session purpose 3 2 1 0 NA 3 2 1 0

2. Didactic intervention training 3 2 1 0 NA 3 2 1 0

3. Answer implementer’s questions 3 2 1 0 NA 3 2 1 0

4. Demonstrate intervention 3 2 1 0 NA 3 2 1 0 5. Engage the implementer in

guided practice 3 2 1 0 NA 3 2 1 0

6. Provide feedback about the practice 3 2 1 0 NA 3 2 1 0

7. Repeat guided practice, providing feedback, if necessary 3 2 1 0 NA 3 2 1 0

8. Implementer engages in independent practice 3 2 1 0 NA 3 2 1 0


10. Repeat independent practice and feedback, if necessary 3 2 1 0 NA 3 2 1 0

11. Close the session 3 2 1 0 NA 3 2 1 0

Sum Columns

Sum Adherence Columns

A

Sum Quality columns

A

Number of Applicable Steps x 3 B

Number of Rated Quality

Steps x 3 B Divide A / B Divide A / B Adherence % Quality %

Implementer Responsiveness Always

100% Mostly

>51% Rarely ≤50%

Never 0%

Implementer was actively engaged. 3 2 1 0

Implementer cooperated with the intervention. 3 2 1 0

177

Implementation Planning Strategy

Appendix G

IMPLEMENTATION PLANNINGImplementation Planning aims to prepare the implementer to im-

plement all of the intervention steps and face potential barriers to ongoing implementation. Implementation Planning includes Action Planning, detailed logistical planning, and Coping Planning, barrier identification and remediation. Completion of Implementation Plan-ning should improve the implementer’s confidence in delivering the intervention with high levels of treatment integrity.

Preparing for Implementation Planning •Review general guidelines for preparing for an Implementa-

tion Support meeting and complete necessary planning steps (see Appendix E).

•Review general guidelines for preparing for an Implementa-tion Support Strategy meeting and complete necessary planning steps (see Appendix E).

•Prepare for Implementation Planning to take one session outside of the implementation setting.

•Break down the intervention into teachable intervention steps (see Action Plan Worksheet: Part A).

•Gather the Implementation Planning forms in the format that you prefer. That is, have the paper and pencil forms (i.e., 4 pages) or the electronic version of the Implementation Planning form.

178

MATERIALS: •A written list of intervention steps•Implementation Planning forms

APPENDIX G: IMPLEMENTATION PLANNING STRATEGY

Step-by-Step1. Explain session purpose

2. Review student issue and goal

3. Review intervention steps

4. Modify intervention steps, if needed

Talking Points•Explain that you are meeting to look at the logistics of the intervention to plan for implementation.

•Provide an overview of Implementation Planning. Explain that Implementation Planning has two steps: Action Plan-ning, where you’ll look at the intervention steps and plan the details of implementation, and Coping Planning, where you’ll identify and problem-solve barriers to implementa-tion.

•Discuss and collaboratively develop goals for Implementa-tion Planning. These might include helping the implement-er’s preparation for implementation or making adaptions to the intervention to ensure it is contextually appropriate. Explain how Implementation Planning will help meet these session goals.

•Review the target student(s) issue, current data, and in-tervention goal. In doing so, describe generally how the intervention is designed to address the student issue and support the student to meet his or her goal.

• To begin Action Planning, show the implementer the list of intervention steps (see Action Plan Worksheet: Part A).

• Ask if the list of steps, divided in this way, makes sense. If the implementer has any suggestions about (a) how the steps are broken out or (b) the order of the intervention steps, revise the list.

•Ask the implementer if any modification to intervention steps might increase feasibility or help make the

179

5. Identify logistics of each intervention step

6. Discuss how needed resources may be obtained, if applicable

7. Summarize the action plan

intervention more contextually appropriate. • When making modifications to specific intervention steps, keep in mind the empirical and theoretical support for any revisions.

• Make notes about any agreed upon revisions to the intervention (see Action Plan Worksheet: Part A).

• Describe how thinking about the specific logistical aspects of an intervention plan can support sustained implementation.

• For each intervention step, ask the implementer to identify the logistics of implementation (i.e., What? How often? For how long? Where?). Also, note if any materials are needed (i.e., yes, needed or no, not needed).

• Use the Action Plan Worksheet: Part B or the elec-tronic form to record the implementer’s responses.

• If the implementer is struggling to identify logis-tics of implementation, provide helpful questions to facilitate their identification or use the Action Plan Sample Responses form to provide examples. Make sure the implementer’s responses reflect the impres-sions of implementation for his/her context.

• If additional materials are needed for interventions steps, determine if (a) the implementer can access them, (b) you can provide or develop them, or (c) someone needs to be approached to obtain them. Consider that resources should be obtained as quick-ly as possible as their absence might delay imple-mentation. Use the Action Plan Worksheet: Part C or the electronic form to record the resource plan.

• Summarize the revisions made (if applicable), and the logistical details that were defined. Praise the


180

implementer for his/her participation in the process.• Identify potential barriers to implementation• Show the implementer the Coping Plan Worksheet and ask the implementer for major anticipated/current im-plementation barriers.

• Barriers should be identified by the implementer and not offered by the consultant. If the implementer cannot identify any barriers, provide examples of a barrier relat-ed to a different intervention to prompt brainstorming.

• Ask the implementer to prioritize up to 4 and put the priority numbers in the left-hand column.

• Ask the implementer to brainstorm ways to maintain intervention implementation in the presence of each of the top 4 barriers.

• If he or she struggles to identify strategies, provide sug-gestions or ideas in a collaborative manner.

• Once an appropriate strategy is identified, write it on the Coping Plan Worksheet.

• Summarize Coping Plan• Summarize the strategies to overcome these barriers. Praise the implementer for his/her participation in the process.

• Review the process of completing Implementation Plan-ning. Ask if the implementer has any questions about (a) the revisions made to the intervention plan, (b) the logis-tics of implementation, (c) who is responsible for obtain-ing what resources by when, and (d) the identified barriers and related strategies to maintain implementation.

• Tell the implementer when you will provide a clean ver-sion of the Implementation Plan (see Appendix G- Imple-mentation Planning Summary Report Template) and any resources you are responsible for obtaining.

•Thank the implementer for working with you.

8. Identify potential strategies to address barriers

9. Close session


Action Plan Worksheet: Part A

INTERVENTION STEP # INTERVENTION STEP REVISION TO INTERVENTION STEP (if applicable)

INTERVENTION TO BE IMPLEMENTED STEP #

When? How often? For how long? Where?

Action Plan Worksheet: Part B

INTERVENTION STEP # RESOURCES NEEDED?

What? Who is responsible? By when?

Action Plan Worksheet: Part C

Action Plan Sample Responses

•GenEd-Homeroom•GenEd-reading/languagearts/English•GenEd-math•GenEd-science•GenEd-socialstudies/history•GenEd-foreignlanguage•GenEd-other•Music•Physicaleducation•Technology•Chorus•Orchestra•Band

•Specialeducation-inclusion•Specialeducation-resourceroom•Teacheraidepresent•Specialeducationaidepresent•Duringlunch•Beforeschool•Afterschool•Duringrecess•Inplaceofinstructiontime(specify)

•Duringstudyhall/freeperiod•Duringtransitions•Duringbusride

•Duringassemblies•Beforeschool•Afterschool•Duringprepperiod•Whenteacherwiththestudent(s)•Allday•Allmorning•Allafternoon•Whenstudentexhibits____behavior/skill(specify)

•Whenstudentdoesn’texhibit____behavior/skill

When?

How often?

•Once•Every__min•___times/period•___times/activity•Atthebeginningof_______•Attheendof___

•Hourly•__times/day•Daily•__days/week•Weekly•___days/month

•___weeks/month•Monthly•___days/markingperiod•___weeks/markingperiod•Asneeded•Other(specify)

For how long?

•Throughout______instruction(specify)

•Throughoutperiod

•For___minutes(specify)•Aslongasneeded•Throughoutactivity

•Other(specify)•Untilstepcompleted•Notapplicable

Where? General locations

•GenEdclassroom-homeroom•GenEdclassroom-reading/ELA/English

•GenEdclassroom-math•GenEdclassroom-science•GenEdclassroom-socialstudies/history

•GenEdclassroom-foreignlan-guage

•GenEdclassroom-whenclassnotinsession

•GenEdclassroom-other(specify)

•Specialeducation-resourceroom

•Specialeducation-resourceroom-whenclassnotinsession

•Musicclassroom•Technologyclassroom•Chorusclassroom•Cafeteria

•Generalpurposeroom•Theatre•Band•Library•Bathroom•Hallway•Schooloffice•Schoolpsychologist’soffice•Schoolcounselor’soffice•Schoolsocialworker’soffice•Gym

•Playground•Bus•Emptyclassroom•Emptyconferenceroom•Emptyoffice•Home•Other(specify)•Atteacher’sdesk•Atstudent’sdesk•Atstation(s)/center(s)•Attable•At/nearcubbies•Atlabtable•Whereverthestudentis•Atcomputer•Onrug•Onfloor

•Ingirl’sbathroom-atsinks•Inboy’sbathroom-atsinks•Ingirl’sbathroom-install•Inboy’sbathroom-install•Inboy’sbathroom-aturinal•At/onchalk/whiteboard•Inlockerroom•Onstage•Onfield•Ontrack•Inlibrarystacks•Immediatelyoutsideclass-room

•Inhallwayleadingtonextclass/activity

•Studyhall•Principals’office

•Viceprincipal’soffice•Nearadministrativeassis-tant’sdesk

•Nurse’soffice•Inchair/seat•Onplayequipment•Onplayground•Inbedroom•Inlivingroom•Inkitchen•Indiningroom•Inparents’bedroom•Insiblings’bedroom•Inbackyard•Infrontyard•Insideyard•Other(specify)

Where? General locations (continued)

PRIORITY BARRIER TO INTERVENTION IMPLEMENTATION STRATEGY TO CONTINUE IMPLEMENTATION

Coping Plan Worksheet

ACTION & COPING PLAN ReportStudent: Birth Date:

Date Written: School:

Implementer: Consultant:

This Action and Coping Plan is based on collaborative decisions made about how implementation of the intervention can best fit within the context of the current classroom and potential barriers to implementation in this context. The purpose of this Action and Coping Plan is to (a) define intervention implementation steps, (b) complete detailed logistical planning regarding how the intervention will be implemented in the specific context (e.g., integration with classroom routines), (c) identify barriers to initiating and sustaining implemen-tation, and (d) develop a plan to maintain implementation when barriers are encountered. The first section outlines the action plan for each intervention step and the second section outlines potential barriers and strategies to maintain implementation.

TO BE IMPLEMENTED

When? How often? For how long? Where?

I. Action Plan

INTERVENTION STEP #

Potential Major Barrier toIntervention Implementation

II. Coping Plan

NOTE: If the intervention is adjusted (e.g., new components added or removed, implemented in new context) or new

barriers are identified during implementation, the Action and Coping Plan may be updated as needed.

Strategy to Implement the Intervention Nevertheless

189

Appendix GImplementation Planning Treatment Integrity Protocol

174


Appendix F






190APPENDIX G: IMPLEMENTATION PLANNING PROTOCOL 175





IMPLEMENTATION PLANNING: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

IMPLEMENTATION PLANNING: Treatment Integrity Data Sheet



Strategy Steps Adherence Quality* Complete Substantial Limited None NA Excellent Good Fair Poor 1. Explain session purpose

3 2 1 0 NA 3 2 1 0

2. Review student issue and goal 3 2 1 0 NA 3 2 1 0

3. Review intervention steps 3 2 1 0 NA 3 2 1 0

4. Modify intervention steps, if needed 3 2 1 0 NA 3 2 1 0

5. Identify logistics of each intervention step 3 2 1 0 NA 3 2 1 0

6. Discuss how needed resources may be obtained, if applicable 3 2 1 0 NA 3 2 1 0

7. Summarize the Action Plan 3 2 1 0 NA 3 2 1 0

8. Identify potential barriers to implementation 3 2 1 0 NA 3 2 1 0

9. Identify potential strategies to address barriers 3 2 1 0 NA 3 2 1 0

10. Summarize the Coping Plan 3 2 1 0 NA 3 2 1 0


Sum Columns


A

Sum Quality columns

A





100% Mostly >51%

Rarely ≤50%

Never 0%



192

Implementation Beliefs Assessment

Appendix H

Thank you for taking the time to complete the Implementation Beliefs Assessment. The Implementation Beliefs Assessment consists of 19 statements about implementation of school-based interventions. Please consider each statement with regard to your own implementation and rate the extent to which you agree or disagree by circling a number on a scale of 1 to 7. The qualification associated with each number is written above each number on the next page. For example, circling “1” means that you completely disagree with the statement as it applies to you, circling “7” means you completely agree with the statement as it applies to you, and circling 4 means you neither agree nor disagree with the statement as it applies to you.

Implementation Beliefs AssessmentStudent name: Implementer name: Date:

Implementation Beliefs Assessment

Com

plet

ely

disa

gree

Mos

tly

di

sagr

ee

Mod

erat

ely

disa

gree

Nei

ther

agr

ee

nor d

isag

ree

Mod

erat

ely

agre

e

Mos

tly

agre

e

Com

plet

ely

agre

e

1. As a result of this intervention, the student(s) will perform better in school. 1 2 3 4 5 6 7

2. I believe I can maintain this intervention for an extended period of time. 1 2 3 4 5 6 7

3. I can implement this intervention as long as needed. 1 2 3 4 5 6 7

4. The student(s)’ performance will improve as a result of using this intervention. 1 2 3 4 5 6 7

5. I can figure out how to restart this intervention if implementation is disrupted. 1 2 3 4 5 6 7

6. I have the ability to implement each component of this intervention. 1 2 3 4 5 6 7

7. I can resume this intervention after a scheduled break. 1 2 3 4 5 6 7

8. I have the ability to start implementing this intervention after a break. 1 2 3 4 5 6 7

9. I can stick to implementing all steps of this intervention. 1 2 3 4 5 6 7

10. I am prepared to implement this intervention. 1 2 3 4 5 6 7

11. I believe I can implement this intervention. 1 2 3 4 5 6 7

12. I can sustain intervention implementation. 1 2 3 4 5 6 7

13. I can continue implementing this intervention regardless of obstacles/chal-lenges. 1 2 3 4 5 6 7

14. I can restart implementation of this intervention efficiently after an interrup-tion. 1 2 3 4 5 6 7

15. I have the necessary skills to carry out this intervention. 1 2 3 4 5 6 7

16. This intervention will work. 1 2 3 4 5 6 7

17. This intervention will benefit the student(s). 1 2 3 4 5 6 7

18. I am capable of resuming implementation of this intervention. 1 2 3 4 5 6 7

19. I know the steps that will allow me to implement this intervention effectively. 1 2 3 4 5 6 7

Outcome Expectations: one’s perceptions of the outcomes or likely consequences of changing their behavior

4. The student’s performance will improve as a result of using this intervention.

17. This intervention will benefit the student.

16. This intervention will work.

1. As a result of this intervention, the student will perform better in school.

OUTCOME EXPECTATIONS AVERAGE [(SUM of above items / 4) x 100]

Self Efficacy: one’s belief in his or her ability to deliver the intervention to the student in the short- and long-term

Implementation Self-Efficacy: one’s confidence in being capable of performing a difficult or novel behavior.

11. I believe I can implement this intervention.

6. I have the ability to implement each component of this intervention.

19. I know the steps that will allow me to implement this intervention effectively.

15. I have the necessary skills to carry out this intervention.

10. I am prepared to implement this intervention.

Maintenance Self-Efficacy: one’s confidence in being capable of keeping up a difficult behavior over time

3. I can implement this intervention as long as needed.

9. I can stick to implementing all steps of this intervention.

12. I can sustain intervention implementation.

2. I believe I can maintain this intervention for an extended period of time.

13. I can continue implementing this intervention regardless of obstacles/challenges.

Recovery Self-efficacy: one’s confidence in being capable of resuming a difficult behavior after an interruption

14. I can restart implementation of this intervention efficiently after an interruption.

7. I can resume this intervention after a scheduled break.

5. I can figure out how to restart this intervention if implementation is disrupted.

18. I am capable of resuming implementation of this intervention.

8. I have the ability to start implementing this intervention after a break.

SELF-EFFICACY AVERAGE [(SUM of above self-efficacy items / 15) x 100]

IMPLEMENTATION BELIEFS ASSESSMENT TOTAL SCORE [(SUM of all items / 19) x 100]

Student name: Implementer name: Date:Implementation Beliefs Assessment (con’t)

195

Data-Based Decision Worksheets A–D

Appendix I

Next, Describe the Treatment Integrity

Summary Statement:

_________________________________

_________________________________

_________________________________

_________________________________

_________________________________

_________________________________

Last, determine if (a) the student is on

track or not on track to meet his or her

goal and (b) whether sufficient or

insufficient implementation has occurred.

Identify the appropriate scenario and

complete related action steps. Record the

decision on the PRIME Intervention

Implementation Tracking Form.

First, Describe the Progress-Monitoring Summary Statement:

____________________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

____________________________________________________________________________________

□ Student is on track to meet his or her goal

□ Student is not on track to meet his or her goal

□ Insufficient Implementation

□ Sufficient Implementation

Scenario A The student is making appropriate progress and the intervention is

effective.

Continue to deliver intervention at current level

Regularly evaluate progress monitoring and TI data

Date of next meeting: ________

Scenario B The student is making appropriate progress, but it is unclear whether

the intervention is effective.

See Scenario B Action Steps

Scenario C The student is not making

appropriate progress and the intervention may not be effective.

See Scenario C Action Steps

Scenario D The student is not making

appropriate progress as the intervention has not been implemented as planned.

See Scenario D Action Steps

PRIME Data-Based Decision Making Worksheet: Integrating the Data

Student name: _________________________________ Date of meeting: ___________________

Team members present: ____________________________________________________________

□ Do stakeholders report any changes outside of intervention implementation that might influence student progress?

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________



Scenario B Action Steps: On Track to Meet Student Goal & Insufficient Treatment Integrity

First, Identify the Reason for Progress-Monitoring & Treatment Integrity Data Pattern

Discontinue implementation of the intervention

□ Was the intervention goal appropriate based on review of research?

________________________________________________

________________________________________________

________________________________________________

________________________________________________

__

Adjust the aim line for student progress

Adjust treatment integrity measure

and/or data collection plan

□ Does the treatment integrity measure and data sufficiently capture intervention steps directly related to student outcomes?

_________________________________________________

_________________________________________________

_________________________________________________

_________________________________________________

Reevaluate progress -monitoring and TI data to

make an appropriate, data-driven decision

about next steps.

See Data-Based Decision Worksheet A

Then, Based on the Most Likely Reason, Take Appropriate Action Steps

Collect additional treatment integrity and progress-monitoring

data

If yes

If no

If no

If yes

If no

If yes

Review questions again, identify most likely reason and take appropriate action step

Continue to collect progress-monitoring data

to ensure appropriate reason selected and student continues

improve

□ Does the treatment integrity measure and data sufficiently capture intervention steps directly related to student outcomes?

__________________________________________

__________________________________________

__________________________________________

__________________________________________

__________________________________________

__________________________________________



Scenario C Action Steps: Not On Track to Meet Student Goal & Sufficient Treatment Integrity

First, Identify the Reason for Progress Monitoring & Treatment Integrity Data Pattern

Discontinue implementation of the intervention

□ Is the intervention appropriate for the problem behavior and student?

_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________

_________________________________________

Increase the intensity of

the intervention

Adjust treatment integrity measure and/or data

collection plan

Reevaluate progress -

monitoring and TI data to make an

appropriate, data-driven

decision about next steps.

See Data-Based

Decision Worksheet A

Then, Based on the Most Likely Reason, Take Appropriate Action Steps

Collect additional treatment integrity

and progress-

monitoring data

If no

If no

If yes

If yes

Begin PRIME process again beginning with Problem

Identification

See Chapter 2

Update Implementation Plan

See Chapter 4

□ What PRIME Implementation Supports have been provided to support the implementer recently?

_______________

_______________

_______________

_________



Scenario D Action Steps: Not On Track to Meet Student Goal & Insufficient Treatment Integrity

First, Consider Provided Supports & Intervention Intensity to Identify the Appropriate Support Tier

Then, Choose a PRIME Implementation Support

If no support has been provided (or not recently) or low intensity intervention

If Tier 1 support has been provided recently and moderate intensity intervention

If Tier 1/2 support has been provided recently and/or intensive intervention

Tier 1 Implementation

Support


Support


Support

Direct Training

Implementation Planning

Performance Feedback

Role Play or Participant Modeling

Motivational Consulting or

Raising Awareness

□ Review the General

Strategy Guide Appendix E

□ Review the specific PRIME Implementation

Support Chapter and

Appendix

Collect additional TI and progress-monitoring

data

Reevaluate progress-monitoring and TI data to make an appropriate, data-driven decision about next steps.

See Data-Based Decision Worksheet A

If overall TI is low or IBA indicates low OE

If specific steps are missed or IBA indicates low self-‐efficacy

If some steps are never implemented, quality is low, or IBA indicates low self-‐efficacy

If TI is inconsistent, TI has decreased over time, exposure is low, or IBA indicates low OE

Last, Prepare for PRIME Implementation Support

□ What is the intensity of the intervention? What decisions will be made based on the intervention?

______________

______________

______________

______________

____________

200

Role Play Strategy

Appendix J

ROLE PLAYRole Play aims to increase the implementer’s preparation for and

confidence in implementation through discussion, intervention demonstration, practice and problem-solving. After this session, the implementer should feel ready to deliver the intervention to the tar-get student and maintain implementation over time.

Preparing for Role Play •Review general guidelines for preparing for an Implementa-

tion Support Strategy meeting and complete necessary plan-ning steps (see Appendix E).

•Prepare for Role Play to take one session outside of the imple-mentation setting.

•Prepare the intervention implementation and student outcome graphs and be prepared to share these figures, summarize the data, and answer questions in a constructive and supportive manner.

•Use treatment integrity data to identify “target” intervention steps for practice using Role Play. Brainstorm and identify sce-narios that may be good practice scenarios for role play.

MATERIALS: •Intervention implementation data (e.g., Treatment Integrity

Across Sessions Graph and Treatment Integrity Across Inter-

201APPENDIX J: ROLE PLAY STRATEGY

vention Steps Graph).•Student outcome data (e.g., Progress-Monitoring Graph). •Written list of intervention steps denoting “target” steps as de-

termined by treatment integrity data.

STEPS1. Explain session

purpose

2. Elicit implementer feedback about Implementation Beliefs Assessment (IBA) data

3. Discuss intervention steps as related to treatment integrity and progress- monitoring data

TALKING POINTS• Explain that you are meeting to discuss the intervention,

any concerns about being able to implement consistently, and practice the intervention.

• Provide an overview of Role Play by briefly describing steps including review of the intervention and modeling and practice of specific scenarios.

• Discuss and collaboratively develop goals for Role Play. These might include the implementer being more confi-dent to implementation in challenging situations or under-standing specific intervention steps better. Highlight how the steps Role Play will help meet the session goals.

• In a collaborative and supportive manner, ask the imple-menter to describe his or her responses on the IBA.

• Ask the implementer to identify specific scenarios related to his or her concerns about implementing the intervention.

• Review the intervention steps in detail, explaining why that step is used, what implementing that step will serve to accomplish, and how it relates to treatment integrity, progress-monitoring, and IBA data. To do so, share the graphs and describe the summary statements and links between the data sources.

• Praise implementer for those intervention steps imple-mented consistently. Assure the implementer that the role play will provide an opportunity to practice some of the steps/skills that might be more difficult.


• Ask for the implementer’s perspective of his or her ability to consistently implement the intervention steps, after having reviewed the data in step 3.

• Summarize and validate the implementer’s perspective.

• Identify target intervention steps to be practiced based on your review of the treatment integrity data and fit for the activity.

• Ask the implementer to suggest some practice scenarios that he or she could use to practice the target interven-tion steps. There may be one or more than one practice scenario depending on the number of target steps and the order in which they are intended to be implemented.

• If the implementer has difficulty identifying scenarios, suggest these ideas for practice based on treatment integrity data.

• Demonstrate intervention components. During your demonstration, you may simply demonstrate delivering the intervention as planned or you may describe what you are doing. If you describe your actions, be sure to note aspects of implementation related to adherence (i.e., delivering the intervention as planned) and quality (i.e., how you are delivering intervention components).

• Ask the implementer for feedback regarding the demon-stration.

• Summarize the implementer’s perspective of the demon-stration and validate his/her thoughts and feelings.

• Share your feedback about the demonstration, highlight-ing areas of relative ease and describing how you navi-gated through difficult steps.

• Praise the implementer’s role as the student.

4. Elicit implementer feedback regarding intervention steps

5. Discuss intervention steps to be practiced and practice scenarios to be used

6. Demonstrate the intervention step(s) with the implementer acting as a student

7. Exchange feedback about demonstration


• With the implementer, use treatment integrity and IBA data to develop and agree on goals for the role play.

• Include both general (e.g., a successful practice session) and specific (e.g., demonstrate strategies for a particular-ly difficult step) goals.

• Have the implementer role play the intervention step (or group of steps) while you act as a student.

• Observe the implementer closely, paying attention to ver-bal and non-verbal behavior. Look for moments of ease, difficulty, and when interaction shifts.

• Encourage the implementer and provide prompts as nec-essary.

• Praise the implementer’s efforts and ask for feedback from the implementer about his/her performance during the role play. Ask about intervention steps that went well and those that were more difficult. Ask about areas where he or she would like more help.

• Share your feedback about the role play in a positive way, emphasizing areas of success, and collaboratively brain-storming solutions for areas of difficulty.

• Summarize and validate implementer’s suggestions, and repeat Role Play as needed until implementer has mas-tered all target intervention steps.

• Thank the implementer for meeting with you, being open to new ideas, and praise him or her for practicing the intervention.

• Describe overall relationship among intervention steps, progress monitoring, and sustained implementation.

• Recap the demonstration and role play(s) and highlight the implementer’s improvements.

8. Discuss role play goals

9. Implementer role plays with consultant acting as a student

10. Exchange feedback about implementer’s practice


204

Appendix JRole Play Treatment Integrity Protocol

174


Appendix F






205APPENDIX J: ROLE PLAY PROTOCOL 175





ROLE PLAY: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

ROLE PLAY: Treatment Integrity Data Sheet




1. Explain session purpose 3 2 1 0 NA 3 2 1 0

2. Elicit implementer feedback about implementation Beliefs Assessments (IBA) data

3 2 1 0 NA 3 2 1 0

3. Discuss intervention steps as they relate to treatment integrity and progress monitoring data

3 2 1 0 NA 3 2 1 0

4. Elicit implementer feedback regarding the intervention steps 3 2 1 0 NA 3 2 1 0

5. Discuss intervention steps to be practiced and practice scenarios to be used

3 2 1 0 NA 3 2 1 0

6. Demonstrate the intervention step(s) with the implementer acting as the student

3 2 1 0 NA 3 2 1 0

7. Exchange feedback about demonstration 3 2 1 0 NA 3 2 1 0

8. Discuss role play goals 3 2 1 0 NA 3 2 1 0 9. Implementer role plays with the

consultant acting as the student 3 2 1 0 NA 3 2 1 0

10. Exchange feedback about implementer’s practice 3 2 1 0 NA 3 2 1 0


Sum Columns


A

Sum Quality columns

A




Implementer Responsiveness

Always 100%

Mostly >51%

Rarely ≤50%

Never 0%



207

Appendix KParticipant Modeling Strategy

Participant ModelingParticipant Modeling aims to increase both an implementer’s con-

fidence to carry out an intervention, as well as his or her outcome expectations, through positive experience with the intervention and demonstration of its benefits. Through a brief review of evidence supporting the intervention and modeled practice of intervention steps within the target setting, the implementer should be prepared to implement and feel confident approaching the intervention inde-pendently.

Preparing for Participant Modeling• Review general guidelines for preparing for an Implementa-


• Prepare for Participant Modeling to take three sessions; the first and third sessions will occur outside of the implementa-tion setting and the second session should happen in-vivo.

• Review information regarding intervention rationale.• Review the intervention implementation data and determine

intervention steps that should be targeted using Participant Modeling based on adherence and quality data.

• Based on implementation data and the intervention itself, decide in what order to model the intervention steps. Pre-

208APPENDIX K: PARTICIPANT MODELING STRATEGY

pare a list of intervention steps in order according to most appropriate format. For example, intervention steps may be ordered sequentially or grouped according to difficulty. Note: It may be necessary to prepare more than one list if multiple formats may be appropriate.

MATERIALS: • Written list of discreet intervention steps (i.e., list of teach-

able steps from Direct Training) denoting “target” steps as determined by treatment integrity data.

• Any items needed to practice the intervention (e.g., writing utensils, forms, reinforcers, manual).


purpose

2. Describe the intervention and importance of imple-mentation

TALKING POINTSNote: Steps 1-4 occur outside the target setting.• Explain that the purpose of this meeting is to have a

chance to review and practice intervention implementation to be sure it is implemented as intended.

• Describe the steps of Participant Modeling - the imple-menter will observe the consultant model select interven-tion steps, practice those steps with support, and practice them independently.

• Discuss and collaboratively develop goals for Participant Modeling. These might include increased mastery of the intervention or improved confidence in implementation. Highlight how the steps of Participant Modeling will help meet the session goals.

• Describe how the selected intervention addresses the identified problem and will help the student reach his or her intervention goal.

• Explain how implementation of the intervention impacts student outcomes. That is, high levels of treatment integ-rity are related to more efficient improvements in student progress-monitoring data.


• Review the intervention steps in detail, explaining how each step will be implemented, why that step is used, and what implementing that step will serve to accomplish.

• Regularly provide the implementer with opportunities to ask questions about the steps and describe his or her experience with implementation.

• If the implementer requests minor adaptations to an in-tervention step, revise the step on the list and update the Implementation Plan as needed.

• Identify target intervention steps to be practiced during in-vivo practice by determining which steps are not being implemented consistently (according to treatment integrity data), and are an appropriate fit (e.g., steps like posting visual reminders in the classroom one time are likely not appropriate for in-vivo practice).

• Work with implementer to choose desired format for in-vivo practice. For example, will you model target inter-vention steps as the opportunity arises? Will you arrange the steps in a sequential order and model them in that order?

• Plan logistics for how you will enter/exit the target setting and suggest to the implementer that he/she may want to notify the target student(s) of your presence in advance.

Note: Steps 5-9 occur in the target setting.• Model how each of the targeted intervention steps

should look and sound, being mindful of pace and clarity. Be aware of body position and pronunciation to ensure that implementer can see and hear you clearly. In doing so, attend to the adherence, quality, and exposure for each intervention step.

3. Review the steps of the intervention

4. Identify target practice steps and prepare for in-vivo exercise

5. Complete in-vivo modeling


• Invite the implementer to practice the intervention steps with your assistance. Provide specific, positive, and cor-rective suggestions and feedback.

• Have the implementer continue to practice with your guidance until step is mastered.

• Repeat modeling and guided practice as needed until all target intervention steps have been rehearsed.

• After all targeted intervention steps have been modeled and practiced with support, invite implementer to inde-pendently practice.

• Ask the implementer to self-evaluate their independent practice.

• Provide constructive feedback regarding the implement-er’s independent practice. Be sure to reinforce successes and correct any implementation errors.

• If needed, repeat the guided and/or independent practice until the implementer successfully and confidently imple-ments each component of the intervention independently.

Note: Steps 10-11 occur outside the target setting.• Provide implementer with praise and feedback regarding

implementation of targeted intervention steps during independent practice.

• With the implementer, identify where and when imple-mentation of the intervention (or skills acquired through mastery of the targeted intervention steps) might be generalized to other settings or problems.

• Revisit the consultation goals and evaluate if those goals have been met through Participant Modeling.

6. Facilitate in-vivo supported practice

7. Allow implementer to independently practice


9. Repeat independent practice and feedback, if necessary

10. Discuss skill generalization

11. Close the Session


• Ask if the implementer has any questions or requires any additional resources or support (e.g., offer a summary of your session or the list of intervention steps).

• Provide positive feedback to the implementer about his or her participation in Participant Modeling.

212

Appendix KParticipant Modeling Treatment Integrity Protocol

174


Appendix F






213APPENDIX K: PARTICIPANT MODELING PROTOCOL 175





PARTICIPANT MODELING: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

PARTICIPANT MODELING: Treatment Integrity Data Sheet




1. Explain session purpose 3 2 1 0 NA 3 2 1 0 2. Describe the intervention and

importance of implementation 3 2 1 0 NA 3 2 1 0

3. Review the steps of the intervention 3 2 1 0 NA 3 2 1 0

4. Identify target practice steps and prepare for in-‐vivo exercise 3 2 1 0 NA 3 2 1 0

5. Complete in-‐vivo modeling 3 2 1 0 NA 3 2 1 0 6. Facilitate in-‐vivo supported

practice 3 2 1 0 NA 3 2 1 0

7. Allow implementer to independently practice 3 2 1 0 NA 3 2 1 0


9. Repeat independent practice and feedback, if necessary 3 2 1 0 NA 3 2 1 0

10. Discuss skill generalization 3 2 1 0 NA 3 2 1 0 11. Close the session 3 2 1 0 NA 3 2 1 0

Sum Columns


A

Sum Quality columns

A




Implementer Responsiveness

Always 100%

Mostly >51%

Rarely ≤50%

Never 0%



215

Raising Awareness Strategy

Appendix L

Raising AwarenessRaising Awareness aims to reorient the implementer’s percep-

tion of his or her ability to implement the intervention. During this meeting the consultant and implementer will focus on two primary activities. First, they will discuss the role of actual events, beliefs, and how beliefs shape our interpretations of these events. Second, they will review implementation data and research on intervention effectiveness.

Preparing for Raising Awareness • Review general guidelines for preparing for an Implementa-


• Prepare for Raising Awareness to take one session outside of the implementation setting.

• Review the intervention implementation data. Prepare the intervention implementation data (e.g., Treatment Integrity Across Sessions Graph and Treatment Integrity Across Inter-vention Steps Graph) and be prepared to share these figures, summarize the data, and answer questions in a constructive and supportive manner.

• Search libraries, credible websites, literature databases and/or your own resources for information relevant to demonstrating the effectiveness of the intervention when implemented as

216APPENDIX L: RAISING AWARENESS STRATEGY

planned across time. Be sure to use high-quality research, case studies, and/or literature reviews.

MATERIALS: • Intervention implementation data (e.g., Treatment Integrity

Across Sessions Graph and Treatment Integrity Across Interven-tion Steps Graph).

• Written intervention plan.• High-quality research related to importance of implementation

in intervention effectiveness.


purpose

2. Review implementation data

TALKING POINTS• Explain that you are meeting to review treatment integrity

and talk about the implementer’s perspective on imple-mentation.

• Provide an overview of Raising Awareness by describing steps

• Discuss and collaboratively develop goals for the Raising Awareness session. These might include increasing the implementers’ skills, confidence and motivation around intervention implementation. Highlight how the steps of Raising Awareness will help meet the session goals.

• Review intervention implementation data with implement-er by summarizing the Treatment Integrity Across Sessions Graph and the Treatment Integrity Across Intervention Steps Graph.

• Highlight intervention steps that were completed con-sistently and praise the implementer, and briefly review intervention steps that were not consistently implemented.

• Explore implementer’s impressions of the treatment in-tegrity data by asking for his or her feedback. Summarize, validate, and confirm implementer’s response.


• Ask implementer for his or her perspective on how con-sistently implementing the intervention has been going.

• Summarize implementer’s perspective and experience. Make links between the implementation data and the im-plementer’s perspective, as appropriate. Confirm with the implementer that your interpretation is correct.

• Describe that how we understand events influences the results of these events. In doing so, highlight the rela-tionship between (a) an actual event, (b) our beliefs, and (c) the results. Results might include insufficient stu-dent progress, discarding the current intervention, and/or looking for a new, more intensive intervention. These results are impacted not only by intervention implemen-tation (i.e., the actual event), but also by our beliefs about the student (e.g., too inattentive to benefit from the in-tervention), intervention (e.g., hasn’t worked in the past), and other factors (e.g., not enough time).

• To illustrate the relationship among these three factors, provide an example from a real life scenario (i.e., outside of intervention implementation). For instance, you may say that your decision to continue or change a work out plan (results) depends on your actual implementation of the work out plan (actual event) as well as how quickly you expected results and your impressions about your implementation of the plan (beliefs). (See Chapter 12 for additional examples.) Use the Raising Awareness Work-sheet to link these examples to the 3-step sequence.

• Ask the implementer about how the treatment integri-ty data and his or her perspective might fit into the (a) actual event, (b) beliefs, and (c) results sequence. Some implementers will be able to easily describe this 3-step sequence. For others, probe each step of the sequence specifically to build an understanding of the sequence.

3. Ask the implementer about implementation

4. Describe the 3-step sequence

5. Ask implementer to describe the 3-step sequence


• Summarize the implementer’s perspective on the 3-step sequence. If needed, help shape their response to fit this sequence. Use the Raising Awareness Worksheet to record the implementer’s responses. Confirm with the implementer that your summary is correct and update, if needed.

• Briefly present the implementer with a summary of high-quality research about effectiveness of the interven-tion when implemented as planned.

• Describe the research in terms of the 3-step sequence, particularly highlighting the actual event and result. That is, explain the importance of implementing all compo-nents of an intervention to positively impact student outcomes. Reflect on what the participants who imple-mented the intervention in the study might have held as beliefs.

• Ask the implementer about their impression of the research on implementation and how it relates to the 3-step sequence. Summarize and validate their perspective.

• Based on the review of research and the implementer’s perspective, transition to a discussion of strategies to support consistent implementation.

• Ask the implementer to identify strategies to help im-plement consistently in the future. Use the treatment integrity data to target this discussion. If appropriate, suggest identifying some ways to support or address the implementer’s beliefs about the student or intervention. Support the implementer to develop feasible strategies to deliver the intervention as planned. Update the Imple-mentation Plan, if appropriate.

• Based on the discussion, summarize and affirm the implementer’s perspective of (a) the intervention imple-

6. Present research about intervention implementation

7. Brainstorm strategies to improve implementation

8. Summarize the 3-step sequence


mentation research, (b) how these reasons fit into the 3-step sequence, and (c) strategies to assist with imple-mentation. Confirm that your summary is accurate.

• Ask the implementer to summarize the current and future intervention implementation in relation to the 3-step sequence.

• As needed, shape the implementer’s perspective by summarizing, validating, and suggesting things that the implementer might have missed.

• Review the meeting process and again highlight the 3-step sequence.

• Thank implementer for meeting with you, being open to new ideas, and critically examining the data, their beliefs, and research.


220

Appendix LRaising Awareness Treatment Integrity Protocol

174


Appendix F






221APPENDIX L: RAISING AWARENESS PROTOCOL 175





RAISING AWARENESS: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

RAISING AWARENESS: Treatment Integrity Data Sheet



Strategy Steps Adherence Quality* Complete Substantial Limited None NA Excellent Good Fair Poor 1. Explain session purpose 3 2 1 0 NA 3 2 1 0 2. Review implementation data 3 2 1 0 NA 3 2 1 0 3. Ask the implementer about

implementation 3 2 1 0 NA 3 2 1 0

4. Describe the 3-‐step sequence 3 2 1 0 NA 3 2 1 0 5. Ask implementer to describe

the 3-‐step sequence 3 2 1 0 NA 3 2 1 0

6. Present research about intervention implementation 3 2 1 0 NA 3 2 1 0

7. Brainstorm strategies to improve implementation 3 2 1 0 NA 3 2 1 0

8. Summarize the 3-‐step sequence 3 2 1 0 NA 3 2 1 0


Sum Columns


A

Sum Quality columns A

Number of

Applicable Steps x 3 B




100% Mostly

>51% Rarely ≤50%

Never 0%

Implementer was actively engaged. 3 2 1 0 Implementer cooperated with the intervention. 3 2 1 0

223

Motivational Consulting Strategy

Appendix M

Motivational ConsultingMotivational Consulting aims to increase the implementer’s

self-efficacy through providing information about intervention im-plementation as well as the consultant’s use of empathy and posi-tive regard. After this session, the implementer should feel positive and confident to maintain intervention implementation over time.

Preparing for Motivational Consulting: • Review general guidelines for preparing for an Implementa-


• Prepare for Motivational Consulting to take one session out-side of the implementation setting.

• Break down the intervention into teachable components.• Review implementer’s previously developed Implementation

Plan, Implementation Beliefs Assessment data, and treatment integrity data (adherence, quality, and exposure).

Materials: • A written list of intervention steps.• The Implementation Plan.• A summary of the Implementation Beliefs Assessment data,

noting salient themes and responses to items, and treatment integrity data, noting areas for improvement.

224APPENDIX M: MOTIVATIONAL CONSULTING STRATEGY

STEPS1. Explain session purpose

2. Elicit implementer’s goals for and perception of intervention implementation

3. Summarize and validate the implementer’s goals and reinforce change talk

TALKING POINTS• Explain that you are meeting to discuss the importance of

implementation and the implementer’s perspective. • Provide an overview of Motivational Consulting by briefly

describing steps including looking at Implementation Be-liefs Assessment and treatment integrity data and discuss-ing his or her impression of implementation.

• Collaboratively develop goals for the Motivational Con-sulting session. These might include increasing the im-plementers’ skills, confidence and motivation around intervention implementation. Highlight how the steps of Motivational Consulting will help meet the session goals.

• Ask the implementer to explain goals for intervention im-plementation. To do so, use open-ended questions.

• Ask the implementer to explain how implementation of the intervention has been going, continuing to use open-ended questions.

• Summarize the implementer’s perception of current im-plementation and goals for supporting the student in the future. Throughout your summary provide validation of the implementer’s perception and feelings.

• Reinforce any change talk. (Note: Change talk is any state-ment made by the implementer that indicates a desire, need, willingness, and/or commitment to change the cur-rent situation).

• Ask if your summary is correct. If not confirmed, ask the implementer to repeat their ideas, and again try to provide a summary. Repeat until the implementer confirms your summary as accurate.


• Ask if the implementer is willing to review assessment data together.

• Highlight and discuss themes from IBA and describe themes that indicate reservations or concerns about im-plementing the intervention.

• Utilize open-ended questions to ask the implementer for feedback about the assessment data and concerns or comments about implementing the intervention.

• Summarize and validate the implementer’s feedback and comments about the assessment data and intervention.

• Describe some aspects as “good things” and others as “less good.” Take your cue from the implementer about which aspects to describe as “good” and “less good.”

• Reinforce any change talk and ask if your summary is correct.

• Ask if the implementer is willing to review information about intervention implementation.

• Provide a general, but comprehensive explanation of the effectiveness of the intervention when implemented as planned, utilizing research and/or case examples.

• Frame your discussion of intervention implementation as a step toward achieving the goals the implementer de-scribed, helping to promote motivation to keep up with implementation.

• Review the Implementation Plan and treatment integrity data.

• Highlight intervention steps the implementer is imple-menting well.

• Suggest updates to the Action and Coping Plans to ad-dress intervention steps that are of concern to the imple-menter with regard to implementing the intervention.

4. Highlight Implementation Beliefs Assessment data and elicit implementer feedback

5. Summarize and validate the implementer’s feedback

6. Provide information regarding intervention implementation

7. Review Implementation Plan and treatment integrity data


• Ask the implementer to respond to your explanation of intervention implementation. To do so, ask a series of open-ended questions.

• Summarize and validate the implementer’s perspective of intervention implementation. Ask if your summary if correct.

• Provide more details about the intervention if the imple-menter’s responses indicate further questions.

• If the implementer still expresses reservations about intervention implementation, try to address the imple-menter’s concerns by (a) providing further information about sustained implementation of the intervention, (b) reviewing the intervention plan, or (c) encouraging the implementer to continue implementation to see if re-sults are similar to the research and case studies can be achieved.

• Ask the implementer to describe the relationship be-tween implementing the intervention and the goals for supporting the student(s). To do so, utilize open-ended questions.

• Summarize and validate the implementer’s perspective of the intervention and goals. Provide validation and rein-force change talk. Ask if your summary is correct.

• If the implementer still expresses reservations about the intervention, validate the implementer’s perception and encourage the implementer to try to continue sustained implementation of the intervention.

• Revisit the consultation goals and evaluate if those goals have been met through Motivational Consulting.

• Provide strong affirmations to implementer about his/her participation in Motivational Consulting and for his/her willingness and readiness for change.

8. Explore implementer’s thoughts about interven-tion implementation

9. Summarize, validate, and respond to implementer’s perspective

10. Ask implementer about the relationship between implementation and intervention goals

11. Summarize and validate the implementer’s perspective


227

Appendix MMotivational Consulting Treatment Integrity Protocol

174


Appendix F






228APPENDIX M: MOTIVATIONAL CONSULTING PROTOCOL 175





MOTIVATIONAL CONSULTING: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

MOTIVATIONAL CONSULTING: Treatment Integrity Data Sheet



Strategy Steps Adherence Quality* Complete Substantial Limited None NA Excellent Good Fair Poor 1. Explain session purpose 3 2 1 0 NA 3 2 1 0 2. Elicit implementer's goals for

and perception of intervention implementation

3 2 1 0 NA 3 2 1 0

3. Summarize and validate the implementer's goals and reinforce change talk

3 2 1 0 NA 3 2 1 0

4. Highlight implementation Beliefs Assessment data and elicit implementer feedback

3 2 1 0 NA 3 2 1 0

5. Summarize and validate the implementer's feedback 3 2 1 0 NA 3 2 1 0

6. Provide information regarding intervention implementation 3 2 1 0 NA 3 2 1 0

7. Review Implementation Plan and treatment integrity data 3 2 1 0 NA 3 2 1 0

8. Explore implementer's thoughts about intervention implementation

3 2 1 0 NA 3 2 1 0

9. Summarize, validate, and respond to implementer's perspective

3 2 1 0 NA 3 2 1 0

10. Ask implementer about the relationship between implementation and intervention goals

3 2 1 0 NA 3 2 1 0


Sum Columns


A

Sum Quality columns

A





100% Mostly

>51% Rarely ≤50%

Never 0%



230

Performance Feedback Strategy

Appendix N

Performance FeedbackPerformance Feedback aims to increase the implementer’s imple-

mentation of all intervention steps through a discussion of treat-ment integrity and progress monitoring data, detailed review of difficult implementation steps, and collaborative problem-solving to address challenges to implementation. After the session, the im-plementer should have developed a new perspective of his or her implementation of the intervention and feel confident implement-ing the intervention steps moving forward, including steps that have previously been difficult to implement consistently.

Preparing for Performance Feedback • Review general guidelines for preparing for an Implementa-


• Prepare for Performance Feedback to take one session outside of the implementation setting, but remember that it is typical-ly provided on an on-going basis.

• Review the intervention implementation data and student outcome data.

• Prepare the Treatment Integrity Across Sessions Graph and Treatment Integrity Across Intervention Steps Graph and be prepared to share these figures, summarize the data, and an-swer questions on a constructive and supportive manner.

231APPENDIX N: PERFORMANCE FEEDBACK STRATEGY

STEPS1. Explain session purpose

2. Elicit implementer

feedback about the intervention

3. Ask the implementer about student responsive-ness

4. Review implementation data

TALKING POINTS• Explain that you are meeting to discuss the intervention

and its implementation and evaluate student progress.• Provide an overview of Performance Feedback. Tell the

implementer you will look at data, identify areas that are going well, steps that seem to be challenging, and set goals for the next meeting.

• Ask implementer to evaluate the intervention process.• Address any questions or concerns the implementer has

regarding the intervention or implementation.

• Ask the implementer about his or her impressions of the student’s progress and response to the intervention, focus-ing on desired student outcomes.

• Address any questions or concerns the implementer has regarding student progress.

• First, review the intervention implementation data by providing a summary of the Treatment Integrity Across Sessions Graph to the implementer.

• Next, review the Treatment Integrity Across Intervention Steps Graph with implementer. Explain the graph displays

• Prepare the Progress-Monitoring Graph and be prepared to share and summarize the data.

• Review and re-familiarize yourself with the intervention and be prepared to explain the intervention steps as related to student outcomes and implementation data.

MATERIALS: • Treatment Integrity Across Sessions Graph and Treatment Integrity Across

Intervention Steps Graph• Progress-Monitoring Graph


the percentage of days each intervention step was imple-mented according to the intervention plan.

• Highlight intervention steps that were completed consis-tently and praise the implementer.

• Review intervention steps that were consistently imple-mented.

• Review the student progress-monitoring data by describ-ing the Progress-Monitoring Graph to implementer.

• Discuss student progress-monitoring data as they relate to the intervention implementation data, focusing on the relationship between the two graphs.

• Review the intervention steps that were not consistently implemented using the intervention plan for reference.

• For each step, describe (a) the procedures for imple-mentation, (b) when it is to be implemented, and (c) any materials needed.

• Confirm that the implementer’s understands the interven-tion steps reviewed.

• Ask the implementer for feedback about the implementa-tion of steps that have not been consistently implemented.

• Based on implementer’s feedback, work collaboratively to discuss strategies to address the challenges the imple-menter is experiencing implementing these steps.

• Confirm that the implementer feels prepared and com-mitted to increase his or her implementation of the intervention, especially those steps that have not been implemented consistently.

• Summarize the objectives of the session and highlight the link between progress monitoring data and treatment

5. Review progress- monitoring data

6. Review intervention steps

and confirm implementer understanding

7. Problem-solve strategies for implementation im-provement

8. Confirm implementer commitment to increasing implementation



integrity as well as the intervention steps review and problem-solving.

• Ask the implementer if he or she has questions.• Close the meeting by reiterating your support and the

implementer’s commitment to implementation.

234

Appendix NPerformance Feedback Treatment Integrity Protocol

174


Appendix F






235APPENDIX N: PERFORMANCE FEEDBACK PROTOCOL 175





PERFORMANCE FEEDBACK: Treatment Integrity Data SheetImplementer: Consultant: Date: Start time: End time:

PERFORMANCE FEEDBACK: Treatment Integrity Data Sheet



Strategy Steps Adherence Quality* Complete Substantial Limited None NA Excellent Good Fair Poor 1. Explain session purpose 3 2 1 0 NA 3 2 1 0 2. Elicit implementer feedback

about the intervention 3 2 1 0 NA 3 2 1 0

3. Ask the implementer about student responsiveness 3 2 1 0 NA 3 2 1 0

4. Review implementation data 3 2 1 0 NA 3 2 1 0 5. Review progress monitoring

data 3 2 1 0 NA 3 2 1 0

6. Review intervention steps and confirm implementer understanding

3 2 1 0 NA 3 2 1 0

7. Problem-‐solve strategies for implementation improvement 3 2 1 0 NA 3 2 1 0

8. Confirm implementer commitment to increasing implementation

3 2 1 0 NA 3 2 1 0


Sum Columns


A

Sum Quality columns

A





100% Mostly

>51% Rarely ≤50%

Never 0%


Implementer cooperated with the intervention.

3 2 1 0

PRIME - Implementation Science · PRIME was developed to help educators initiate and maintain...

Documents

Transcript of PRIME - Implementation Science · PRIME was developed to help educators initiate and maintain...