Washington PBIS Conference Rob Horner University of Oregon 2012.
Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of...
-
Upload
maria-collins -
Category
Documents
-
view
213 -
download
0
Transcript of Implementing Evidence-based Practices at Scales of Social Significance Rob Horner University of...
Implementing Implementing Evidence-based Practices at Evidence-based Practices at Scales of Social SignificanceScales of Social Significance
Rob HornerUniversity of Oregon
www.pbis.org
GoalsGoalsConsiderations for the emergence of
“evidence-based practices”
Six features for taking EBP technology to scale
Define “implementation” as a unique technology.
Use School-wide Positive Behavior Support as one example
Considerations in the Considerations in the Emergence of Evidence-Emergence of Evidence-based Practicesbased Practices
Defining “practices”
Defining “evidence-based”
Why is it useful to define “Evidence-based Practices?”
Considerations in the Emergence Considerations in the Emergence of Evidence-based Practicesof Evidence-based Practices
Defining a “practice”
◦ A “practice” is a procedure, or set of procedures, designed for use in a specific context, by individuals with certain skills/features, to produce specific outcomes for specific individuals.
◦ A practice can be at any of an array of sizes: Individual technique Intervention package Intervention program
Are these “practices?”Are these “practices?”For whom, in what context, to what For whom, in what context, to what outcome?outcome?
Phonics-based early reading instructionWhole Language Reading InstructionPositive reinforcementInclusionNCLBDiscrete trial trainingGeneralizationPositive behavior supportFunctional analysisApplied behavior analysis
Why should we be invested in Why should we be invested in the emergence of Evidence-the emergence of Evidence-based Practices?based Practices?
Researchers should continually debate and discuss the definition and details.
Implementers need to be able to identify practices that are validated.
Take great care in distinguishing between◦ Documentation that a practice is evidence-
based, and ◦ Documentation that one practice is better
than some other practice.
Taking Evidence-based Practices Taking Evidence-based Practices to Scales of Social Significanceto Scales of Social Significance
The value of a science of human behavior◦ Basic principles that help us describe,
interpret and establish effective patterns of behavior
While behavior analysis is among the most powerful approaches for achieving social change, too often ABA is viewed as relevant only within a narrow range of applications
(we are a niche or boutique technology)
Key Features to AchieveKey Features to AchieveLarge-scale Implementation of Evidence-based Large-scale Implementation of Evidence-based PracticesPractices
1. Focus on comprehensive outcomes defined by the values of the social system
2. Expand the unit of analysis
3. Measure process as well as outcome
4. Use ABA principles to build effective and accessible practices
5. Establish a technology for implementation
6. Define practices for scaling up practices that are evidence-based.
What isWhat is School-wide Positive Behavior School-wide Positive Behavior Support?Support?
School-wide PBS is: A systems approach for establishing the social
culture and individualized behavioral supports needed for schools to achieve both social and academic success while preventing problem behavior
Evidence-based features of SW-PBS Prevention Define and teach positive social expectations Acknowledge (reward) positive behavior Arrange consistent consequences for problem behavior On-going collection and use of data for decision-making Continuum of intensive, individual interventions. Administrative leadership – Team-based
implementation (Systems that support effective practices)
School-wide Positive Behavior Support:School-wide Positive Behavior Support:Current ImplementationCurrent Implementation
School-wide Positive Behavior Support7500 schools in 44 states
Team Coach Curriculum emphasizing prevention: Define
and teach appropriate social behavior to all students
Formal system for rewarding appropriate behavior
Intensive, individual interventions based on behavioral function
On-going data collection and use of data for active decision-making
7500 Schools across 44 states implementing school-wide positive behavior support
Primary Prevention:School-/Classroom-Wide Systems for
All Students,Staff, & Settings
Secondary Prevention:Specialized Group
Systems for Students with At-Risk Behavior
Tertiary Prevention:Specialized
IndividualizedSystems for Students
with High-Risk Behavior
~80% of Students
~15%
~5%
SCHOOL-WIDE POSITIVE BEHAVIOR
SUPPORT
SWIS summary 07-08 SWIS summary 07-08 (Majors Only)(Majors Only)2,532 schools; 1,300,140 students; 1,139,119 2,532 schools; 1,300,140 students; 1,139,119 ODRsODRs
Grade Range Number of Schools
Mean Enrollment per school
Mean ODRs per 100 per school day
K-6 1762 444 .35 (sd=.46)
(1 /300 / day)
6-9 482 653 .92 (sd=1.42)
(1/ 110 / day)
9-12 176 914 1.06 (sd=.1.57)
(1/105 / day)
K-(8-12) 312 401 1.00 (sd=.1.86)
(1/ 155 / day
N = 1679 443 163 246 Elementary Middle High K (8-12)
10%
N = 1679 443 163 246
Elementary Middle High K (8-12)
School-wide PBSSchool-wide PBS1. Focus on comprehensive valued
outcomes Social competence Academic achievement Safety
2. Expand the unit of analysis◦ Whole school
Classroom Groups of “at-risk” students
Individual students needing intensive support
School-wide PBSSchool-wide PBS3. Measurement/ Evaluation
◦ Include both process and outcome measures Outcomes: Office Discipline
Referrals/Academics Process (implementation): Team Checklist Research: System-wide Evaluation Tool (SET)
SWIS
TIC data
SET data
Comprehensive Measures
Progress Monitoring Measures
FidelityMeasures
OutcomesMeasures
XX (ABA)
Comprehensive Measures
Progress Monitoring Measures
FidelityMeasures
XX XX
OutcomesMeasures
XX XX (ABA)
Iowa Checklist 01-05, PK-6 % Fully & Partially Implemented
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
05-A
ug-0
3
05-N
ov-0
3
23-F
eb-0
4
22-J
an-0
4
01-F
eb-0
5
02-J
un-0
5
12-A
ug-0
4
24-N
ov-0
4
01-M
ar-0
5
12-S
ep-0
2
31-O
ct-0
2
28-F
eb-0
3
21-A
pr-0
3
01-S
ep-0
3
05-N
ov-0
3
05-A
ug-0
3
11-S
ep-0
3
07-N
ov-0
3
06-F
eb-0
4
01-S
ep-0
3
01-N
ov-0
3
01-M
ar-0
4
03-A
ug-0
4
08-N
ov-0
4
08-M
ar-0
5
03-J
un-0
5
1 1 1 2 2 2 3 3 3 4 4 4 4 4 4 5 5 5 5 6 6 6 7 7 7 7
Start Up Full Implementation Start Up Part Implementation
School-wide PBSSchool-wide PBS
4. Use ABA principles to Establish Accessible Evidence-based Practices◦ Use the language of the implementation context
◦ Combine technologies needed to achieve valued outcomes. ABA + Person-centered planning + Organizational
Systems + Bio-Medical
◦ Collaborate with other disciplines Mental Health, Juvenile Justice, School Psychology,
Sociology
◦ Use Single-case Designs to Document Evidence-based Practices
Academic, Behavioral, and Academic, Behavioral, and Functional Predictors of Functional Predictors of Chronic Problem Behavior in Chronic Problem Behavior in Elementary GradesElementary Grades
Kent McIntosh University of Oregon
40
5th Grade ORF Trajectories by Function (n = 47)Peer Attention vs. Escape Task
80
90
100
110
120
130
140
150
Fall 03-04 Winter 03-04 Spring 03-04
Mean
Co
rrect
Wo
rds p
er
Min
ute
or 1 ODRs 0
Non-target
Students
5th Grade ORF Trajectories by Function (n = 47)Peer Attention vs Escape Task
80
90
100
110
120
130
140
150
160
Fall 03-04 Winter 03-04 Spring 03-04
Mean
Co
rrect
Wo
rds p
er
Min
ute
.Peer Attn
or 1 ODRs 0
ORF Trajectories by Function (n = 47)
80
90
100
110
120
130
140
150
160
Fall 03-04 Winter 03-04 Spring 03-04
Mean
Co
rrect
Wo
rds p
er
Min
ute
.Peer Attn
Esc. Task
or 1 ODRs 0
School-wide PBSSchool-wide PBS
4. Use ABA principles to Establish Accessible Evidence-based Practices◦ Use the language of the implementation context
◦ Combine technologies needed to achieve valued outcomes. ABA + Person-centered planning + Organizational
Systems + Bio-Medical
◦ Collaborate with other disciplines Mental Health, Juvenile Justice, School Psychology,
Sociology
◦ Use Single-case Designs to Document Evidence-based Practices
Embedding Bully-Proofing Embedding Bully-Proofing in School-wide PBSin School-wide PBS
Scott RossRob Horner
University of Oregonwww.pbis.org
A Functional Approach to Bully A Functional Approach to Bully Prevention:Prevention:Remove the social reinforcersRemove the social reinforcers
Define, teach and reward school-wide behavior expectations.
Teach all children to identify and label inappropriate behavior.
Not respectful, not responsible, not safe
Teach all students a “stop signal” to give when they experience problem behavior.
What to do if you experience problem behavior (victim, recipient)
What to do if you see someone else in a problem situation (bystander)
Teach all students what to do if someone delivers the “stop signal”
29
0
2
4
6
8
10
0
2
4
6
8
10
Baseline Acquisition Full BP-PBS Implementation
0
2
4
6
8
10
0
2
4
6
8
10
0
2
4
6
8
10
Num
ber of Incidents of Bullying B
ehavior
School Days
0
2
4
6
8
10
School 1
Rob
Bruce
Cindy
Scott
Anne
Ken
School 2
School 3
3.14 1.88 .88 72%
BP-PBS, Scott Ross 30
Conditional Probabilities of Victim Responses to Problem Behavior
0%
10%
20%
30%
40%
50%
"Sto
p"
"Wa
lk"
Po
sitiv
e R
esp
on
se(l
au
gh
ing
/ch
ee
rin
g)
Ne
ga
tive
Re
spo
nse
(cry
ing
/fig
htin
gb
ack
)
No
Re
spo
nse
Pro
bab
ilit
y o
f R
esp
on
seBaseline
BP-PBS
28% increase 19% decrease
BP-PBS, Scott Ross 31
Conditional Probabilities of Bystander Responses to Problem Behavior
0%
10%
20%
30%
40%
50%
"Sto
p"
"Wa
lk"
Po
sitiv
e R
esp
on
se(l
au
gh
ing
/ch
ee
rin
g)
Ne
ga
tive
Re
spo
nse
(cry
ing
/fig
htin
gb
ack
)
No
Re
spo
nse
Pro
bab
ilit
y o
f R
esp
on
seBaseline
BP-PBS
21% increase
22% decrease
School-wide PBSSchool-wide PBS
4. Use ABA principles to Establish Accessible Evidence-based Practices◦ Use the language of the implementation context
◦ Combine technologies needed to achieve valued outcomes. ABA + Person-centered planning + Organizational
Systems + Bio-Medical
◦ Collaborate with other disciplines Mental Health, Juvenile Justice, School Psychology,
Sociology
◦ Use Single-case Designs to Document Evidence-based Practices
Use single-case research to document evidence-based practices.
Define protocol for measuring “effect size”
Define professional standards for defining a practice as “evidence-based” using single case research.
School-wide PBSSchool-wide PBS5. Build a functional technology of Implementation
◦Define conditions for implementation◦Define conditions for high fidelity
Coaching, Policies, Administrative Contingencies
◦Establish implementation with low cost◦Establish procedures for sustainability
and continuous regeneration right from the beginning.
The Role of Behavior The Role of Behavior Specialists in the Development Specialists in the Development of Function-based Behavior of Function-based Behavior Support PlansSupport Plans
Leah BenazziUniversity of Oregon
Three knowledge areas needed Three knowledge areas needed for a behavior support planfor a behavior support plan
Behavior Support Plan
Knowledge about The Student
Knowledge aboutThe Setting
Knowledge about Behavioral Theory
DesignDesignThree conditions in which
behavior support plan recommendations were built from simulated cases (descriptive information, functional assessment information)
5 counterbalanced simulations
◦Team alone 12 plans◦Specialist alone 12 plans◦Team with Specialist 12 plans
MeasuresMeasures
Technical Adequacy: Are elements of behavior support plan consistent with functional assessment hypothesis? (1-6) (range 3-18)◦ ABA Experts (published studies employing
functional analysis)
Contextual Fit: Are elements of plan consistent with values, skills, resources, administrative support?◦ 16 questions (8 factors): ◦ Scored on 1-6 scale: Total (16-96)
Team member Ranking of Plans based on preference for implementation (1,2,3)
Results: Results: Technical AdequacyTechnical AdequacyMean Expert Rating (6-18)Mean Expert Rating (6-18)
Mean Expert Ratings of Technical Soundness
0
2
4
6
8
10
12
14
16
Team Alone Team + Spec Spec Alone
Me
an
Te
ch
nic
al S
ou
nd
ne
ss
* Team alone plans were statistically different from plans that included behavior specialist.*Team + Specialist and Specialist Alone were not statistically significantly different.
Table 3Repeated Measures Analysis of Variance Summary Table for the Effects of Plan Developer on Technical Adequacy Scores
Source df SS MS F
Plan DeveloperBeh Spec Invol Unpredicted
211
285.88279.276.62
142.94279.276.62
32.89*64.26* 1.52
Team 11 45.51 4.14
Error 22 95.61 4.35
Total 35 427.00
*p < .01.
Post-hoc analysis of Post-hoc analysis of Technical AdequacyTechnical Adequacy
Behavior Team Team +
F Specialist Alone Specialist
Problem Behavior 3.38 0.95 0.68 0.78
Antecedents identified 4.40 1 .83 1
Identified Function 14.14** 1 0.7 1
Prevention Strategies 16.2** 0.98 0.58 0.98
Teaching Strategies 5.51 0.92 0.68 0.92
Extinction Strategies 55.3** 0.93 0.28 0.84
Positive Reinforcement 5.57 0.94 0.78 0.98
Person Responsible 134.16** 0.93 0.11 0.83
Assess Fidelity 1.5 0.03 0 0.05
Assess Impact 163.83** 0.93 0.13 0.84
** p< Bonferroni family-wise alpha .05
Results: Results: Contextual FitContextual FitMean Team Rating (0-100)Mean Team Rating (0-100)
Mean Team Ratings of Contextual Fit
70
72
74
76
78
80
82
84
86
88
90
Team Alone Team +Spec Spec Alone
Me
an
Co
nte
xtu
al F
it
* Specialist Alone plans were statistically different from plans that included team members.* Team Alone and Team + Specialist plans were not statistically significantly different
Post-hoc Analysis of Contextual Post-hoc Analysis of Contextual FitFit
Contextual Fit Domain F Behavior Team Team +
Specialist Alone Specialist
Knowledge of BSP Elements 15.99** 5.08 5.76 5.67
Skills to perform BSP 9.72 5.35 5.62 5.56
Values consistent with BSP 52.62** 4.69 5.85 5.76
Resources to implement 2.59 4.62 4.89 4.9
Administrative Support 10.68 4.93 5.23 5.32
BSP expected to be effective 29.78** 4.29 5.25 5.4
BSP in best interest of student 30.21** 4.78 5.74 5.77
BSP Efficient to Implement 13.10** 4.32 5.04 4.97
** p< Bonferroni family-wise alpha .05
Implementation Implementation TechnologyTechnologyRole of Coaching
Importance of Policy
Training Outcomes Related to Training Components
Training Outcomes
Training Components
Knowledge of Content
Skill Implementation
ClassroomApplication
Presentation/ Lecture
PlusDemonstration
Plus Practice
Plus Coaching/ Admin SupportData Feedback
10% 5% 0%
30% 20% 0%
60% 60% 5%
95% 95% 95%
Joyce & Showers, 2002
Discipline Foundation Policy: Discipline Foundation Policy: LAUSDLAUSD
School-Wide Positive Behavior Support NUMBER: BUL-3638.0 ISSUER: Donnalyn Jaque-Antón, Executive Officer, Educational Services DATE: March 27, 2007
POLICY: Every student, pre-school through adult, has the right to be
educated in a safe, respectful and welcoming environment. Every educator has the right to teach in an atmosphere free from disruption and obstacles that impede learning. This will be achieved through the adoption and implementation of a consistent school-wide positive behavior support and discipline plan for every school in LAUSD.
All school level discipline plans will be consistent with the Culture of Discipline: Guiding Principles for the School Community (Attachment A) and Culture of Discipline: Student Expectations (Attachment B). This will include: teaching school rules and social-emotional skills; reinforcing appropriate student behavior; using effective classroom management and positive behavior support strategies by providing early intervention for misconduct and appropriate use of consequences.
School-wide PBSSchool-wide PBS
6. Define practices for scaling up
Efficacy Effectiveness
Dean Fixsen and Karen Blase
Scale Model
Fixsen et al
Leadership Team
FundingVisibility Political
Support
Training Coaching Evaluation
Local Demonstration Schools
Active Coordination
BehavioralExpertise
School-wide PBSSchool-wide PBS Document effects in multiple formats
◦ Single case, Descriptive, RCT, etc.
Documentation via randomized control-group Design◦ Provide research outcomes that address
multiple audiences Families Administrators Teachers Scientist from all disciplines
The Effects of School-wide The Effects of School-wide PBS within a Randomized PBS within a Randomized Control Effectiveness TrialControl Effectiveness Trial
Rob Horner, George Sugai, Keith Smolkowski, Lucille Eber, Jean Nakasato, Anne Todd, Jody Esperansa
OSEP TA Center on Positive Behavior Support
www.pbis.orgIn press in the Journal of Positive Behavior Intervention
Research QuestionsResearch QuestionsCan SWPBS be implemented to
criterion by typical state trainers?
If SWPBS is implemented are schools perceived as safer settings?
If SWPBS is implemented do students benefit academically?
MethodMethodRandomized Control Trial
◦ 30 Elementary Schools in Illinois, and 30 Elementary Schools in Hawaii
◦ Random assignment of schools to (Initial SWPBS training; and Delayed SWPBS
training)◦ Replacement (7 schools) randomly assigned
Data collected across three years◦ Time 1: No SWPBS training for any schools◦ Time 2: Initial Treatment Schools get training◦ Time 3: Delay Schools get training (problems…)
MeasuresMeasures Office Discipline Referrals (ODR)
Implementation of SWPBS◦ School-wide Evaluation Tool (SET)
Sugai et al. Perceived School Safety
◦ School Safety Survey (SSS) Sprague, Colvin & Irvin
Academic Success◦ Proportion of Students Meeting State Reading
Standards (SAT – 9 in Hawaii; ISAT in Illinois)
Randomized Waitlist Controlled Randomized Waitlist Controlled TrialTrial
Assessment Time Period
Group T 1 T 2 T 3
Treatment (N = 30) O X O O
Control/Delay (N = 30) O O X O
(T = time (by year), O = observation, X = implementation of SWPBS training)
Finding #1: Finding #1: Implementation by regular Implementation by regular personnelpersonnelMean SET score (Total) [Mean SET score (Total) [Bold indicates post interventionBold indicates post intervention]]
T1 T2 T3
Treatment Group
.381N = 33
.785N = 30
.823N = 30
Control/Delay Group
.388N= 26
.459N= 27
.640N= 23
Random coefficients analysis (Murray, 1998; Singer & Willett, 2003):
Time X Condition p < . 0001; r = .67; d = 1.78
Random coefficients analysis (Murray, 1998; Singer & Willett, 2003): Time X Condition p < . 0001; r = .67; d = 1.78
ResultsResults: With training by regular state : With training by regular state trainers, schools are able to implement trainers, schools are able to implement SWPBS to criterionSWPBS to criterion..
Implementation of SWPBS
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
T1 T2 T3
Mea
n S
ET
Sco
res
Initial (N = 33) Delayed (N = 28)
Random coefficients analysis: p <.0001; d = 1.78
Initial
Training
Delay
Training
* *
Finding #2: Finding #2: SWPBS is associated with SWPBS is associated with increased perception of safety: increased perception of safety: School Safety School Safety Survey: Risk FactorSurvey: Risk Factor
T1 T2 T3
Treatment Group
.370N= 24
.344N = 29
.343N = 25
Control/Delay Group
.387N = 19
.415N= 24
.358N = 20
Time X Condition p = .0154 r = - .40 d = - .86
ResultsResults: Perceived Social : Perceived Social Risk Factors decreasedRisk Factors decreased when when SWPBS was implemented with fidelity. SWPBS was implemented with fidelity.
Perceived Risk Factor Score from School Safety Survey
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
T1 T2 T3
Mea
n S
cho
ol
Saf
ety
Su
rvey
Ris
k S
core
s
Initial (N = 24) Delay (N = 19)
Random coefficients analysis p = .0154; d = -.86
* *
Finding #3: Finding #3: SWPBS associated with SWPBS associated with increase in proportion of students meeting state increase in proportion of students meeting state reading standardreading standard
T1 T2 T3
Treatment Group
.455N = 33
.529N= 31
.536N = 31
Control/Delay Group
.38N = 28
.402N= 27
.436N= 23
T2 Treatment vs. Control: p = .032 r = .28 d = .58
ResultsResults: The percentage of 3: The percentage of 3rdrd graders graders meeting the state reading standard increased meeting the state reading standard increased with SWPBS implementationwith SWPBS implementation
Percentage of 3rd Graders meeting State Reading Standard
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
T1 T2 T3Per
cen
tag
e o
f 3r
d G
rad
ers
mee
tin
g s
tate
rea
din
g
stan
dar
d
Initial (N = 33) Delay ( N= 28)
N.S. p = .032; d = .58
* *
SummarySummary
◦ Never stop development of the rigorous, precise science of human behavior.
◦ Expand the unit of analysis to address socially relevant outcomes Address the full set of outcomes
defined as important for a context/ community
◦ Expand the research methods/questions to address socially important concerns. Sustainability Scalability
SummarySummaryInvest in a technology of “implementation” that will focus on taking evidence-based practices to scale.