Post on 16-Dec-2015
Susan BarrettJerry Bloom
Sheppard Pratt Health Systemwww.pbis.org
www.pbismaryland.org
Building Evaluation Capacity for States and Districts
2
PBS Systems Implementation Logic
Leadership Team
Management Team
FundingMarketing
Visibility Political Support
Training Coaching Evaluation
Active Coordination
Local School Teams/Demonstrations
Braiding Initiatives
EvaluationWhat are your questions?Do you have the tools to answer?Can you get the answer quickly?Easy, Efficient, RelevantEconomic BenefitsBehaviorAchievementRegular Feedback to all Stakeholders- MD
example
•How many schools are implementing SW-PBS ? 1.Number by cohort2.Number by local district3.Number by implementation phase4.Number of Active and Inactive by local district5.Number of schools reporting data (rate of return)
•Are schools implementing SW-PBS? 1.State and District needs to determine process tool (i.e. team implementation checklist, implementation inventory, benchmarks of quality, EBS self assessment)
•Are schools in beginning or advances stages of implementation?1.Implementation Phase Inventory used to track level
•How do reported rates of problem behavior in local district or state compare to the national average? (SWIS)1.by Elementary, Middle, High, Alternative
•What is the percentage of students who receive 0 or 1 Office referrals/year? (SWIS)1.By Elementary, Middle, High, Alternative
•What are the problem behaviors? (SWIS)1.Elementary, Middle, High, Alternative
•Where are problem events occurring? (SWIS)1.By Elementary, Middle, High, Alternative
•What time do problem events occur? (SWIS)1.By Elementary, Middle , High, Alternative
•What are the discipline patterns by gender? (SWIS)•What are the discipline patterns by ethnicity? (SWIS)•What are the discipline patterns for students on an IEP? (SWIS)
Evaluation TemplateFairfax County Example
Forms/Surveys and ChecklistsTier I
Surveying Staff- EBS Survey
Fidelity Checks- Team Implementation Checklist, SET, BOQ, IPI
Safety Surveys- Oregon Safety Survey
Outcome Measures- -ODR, Suspension, Attendance, Special
Ed, GPA, Honor Roll, AYP
Forms /Checklists to TRAINING MATERIALS
EXAMPLE– MD Training Materials
Tools
State Database-Data Entry, Report Generation
Various Levels POC Coach State Team
www.pbssurveys.orgwww.pbseval.orgwww.swis.org
Marketing and VisibilityWho are your stakeholders?Do you have a spokesperson?Using the data to create newsletters,
presentations, fact sheets, elevator business cards- important you can get access to what you need to make your case on the fly!!
www.pbismaryland.orgMaryland Newsletter
Be Careful
11
Multiple levels of VisibilityState and Local Level: Presentations,
Trainings, Stakeholder meetings, Interagency efforts, (Transformation; Mental Health Integration; Wraparound)
Multiple Media: Visual, Face to Face, Written, Website
Multiple Audiences: School Administrations and Instructional Leaders; University staff; Legislators, Potential alternative funders; State and Local Political appointees; Juvenile Justice; Vendors in the System of Care; Parent and other advocacy organizations; Community Members
PBIS Maryland Partner Meeting
Management Team meets with Assistant Superintendent of Student Services and Special Education
7
18
30
5257
81
90
116110
310
18
47 47
64
82
113110
43%
56%60%
90%
82%79%
91%
97%100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
20
40
60
80
100
120
140
1999 2000 2001 2002 2003 2004 2005 2006 2007
Trained and Implementing by Cohort
Trained Implementing % of Trained Implementing
NOTE: Retrained schools will be counted in the most recent retraining year only.
13
65
50
72
18
5
19
9
31
7
27
5
11
48
4
32
50
7 8
30
12
2
1412
1013
64
33
70
18
2
17
6
30
1
27
3
11
45
4
31
37
7 6
24
9
2
1311 10
100% 98%
66%
97%100%
40%
89%
67%
97%
14%
100%
60%
100%
94%
100%97%
74%
100%
75%80%
75%
100%
93% 92%
100%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0
10
20
30
40
50
60
70
80
Alle
gany
Anne
Aru
ndel
Balti
mor
e Ci
ty
Balti
mor
e Co
.
Calv
ert
Caro
line
Carr
oll
Ceci
l
Char
les
Dor
ches
ter
Fred
eric
k
Gar
rett
Har
ford
How
ard
Kent
Mon
tgom
ery
Prin
ce G
eorg
es
Que
en A
nnes
Som
erse
t
Spec
ial
St. M
arys
Talb
ot
Was
hing
ton
Wic
omic
o
Wor
cest
er
Implementing/Inactive by LSS
Trained Implementing % of Trained Implementing
91% Rate of Return
0%10%20%30%40%50%60%70%80%90%
100%
1999 2000 2001 2002 2003 2004 2005 2006 2007
% of schools submitting data
30
60
97286
IPI Fall 2007
Preparation Initiation Implementation Maintenance
473
# of schools reporting:
Evaluation Question: Are schools in beginning or advanced stages of implementation?Data Source: Implementation Phases InventoryOverall, the IPI data suggest a relatively advanced level of self-reported implementation among the schools in Maryland
Evaluation Question: Are schools implementing School-wide PBIS?Data Source: Team Implementation Checklist (Target Criterion = 80%)Based on the total average of the 327 checklists submitted, school teams report that 78% of items are in place.
T eam Implementation C hec klis tS pring 2007
85.6 86.874.8 76.7 71.9
95.0
80.8 77.0 75.5
51.9
78.3
63.170.6
90.0 85.0 85.0 90.0100.0
83.394.0
72.5
0.010.020.030.040.050.060.070.080.090.0
100.0
Alle
gany
Ann
eA
rund
elB
altim
ore
City
Bal
timor
eC
o.
Cal
vert
Car
olin
e
Car
roll
Cha
rles
Fre
deric
k
Har
ford
How
ard
Mon
tgom
ery
Prin
ceG
eorg
esQ
ueen
Ann
es
Som
erse
t
Spe
cial
St.
Mar
ys
Tal
bot
Was
hing
ton
Wic
omic
o
Wor
cest
er
LS
S A
vera
ges
Evaluation Question: Are schools implementing SW-Positive Behavior Support?Data Source: SET (Objective Criterion = 80%)
-All regions met objective criterion-A pre-post comparison regional average shows a 69% increase.
K-6 6-9 9-120
0.2
0.4
0.6
0.8
1
1.2
0.22
0.600000000000001
0.5
0.19
0.45 0.48
0.34
0.980.93
Office Discipline Referrals (ODRs)Per 100 Students Per School Day
MD Average 2006 - 2007 MD Average 2007 - 2008 National Average
Data from the School-Wide Information System (SWIS)
Evaluation Question: Do 80% of the students in MD Elementary PBIS schools using SWIS receive 0 or 1 office referrals?Data Source: SWIS PBIS Elementary schools in Maryland using SWIS report that 92% of their students receive 0 or 1 office referral.
Evaluation Question: Where is the location of behavior problems in Middle Schools?Data Source: SWISThe majority of the problem behaviors reported occur in the classroom followed by hallways.