Practical Problem Solving: Drilling Down Into School...
Transcript of Practical Problem Solving: Drilling Down Into School...
Central Oregon PBIS ConferenceBend, OregonApril 22, 2019
Practical Problem Solving:Drilling Down Into School Data for
Improved Decision-Making
Alan Cook – University of OregonJessica Daily– University of Oregon
Presenters
Alan Cook
[email protected] Applications Training Team
Jessica Daily
[email protected] Team Lead
Objectives
A. Provide language to describe the rationale for building comprehensive data collection and decision systems for monitoring data related to SWPBIS implementation
B. Describe the role of a data analyst in gathering and summarizing data prior to a team meeting to maximize meeting minutes
C. Describe the core features of data analysis or “drilling down” into data sources (e.g., fidelity surveys, referral data)
D. Practice conducting a drill down using referral data using the School-Wide Information System (SWIS)
3
Rationale for Data-Based Decision Making in SWPBIS
Research on Sustainability of SWPBIS
What single factor is most related to high sustainability of SWPBIS?
5
The frequency with which DATA are presented to all school staff
Pinkelman, S., McIntosh K., Rasplica, C. K., Berg, T., & Strickland-Cohen, M. K. (2015).
Outcome/Goal
SWPBIS and Outcomes
6
Desired Outcome: Enhanced social competence& academic achievement for all students.
How do we get to our desiredoutcome?
SystemsStaff Behavior Supports
PracticesStudent Behavior Supports
DataDecision Making Supports
SWPBIS Continuum of Supports (Tiers)
7
All specialized interventions are more effective and more durable with universal, school-wide behavioral expectations as a foundation.
Tertiary—intensive, individualized5% of student population
Secondary—targeted, small group15% of student population
Universal—primary prevention provided to all students, effective for approximately 80%
Types of Information (Data)
Fidelity Data
Outcome Data
Did we implement the systems and strategies we agreed upon?
Is the plan resulting in progress toward our goals? 8
Connecting Fidelity & Outcome Data
Lucky SustainingPositive outcomes, low understanding of how they were achieved
Replication of success is unlikely
Positive outcomes, high understanding of how they were achieved
Replication of success likely
Losing Ground LearningUndesired outcomes, low understanding of how they were achieved
Replication of failure likely
Undesired outcomes, high understanding of how they were achieved
Replication of mistakes unlikely
Out
com
es
Fidelity9
Data-Based Decision Making
Decisions are more likely to be effective and
efficient when they are based on....
DATA
The quality of data-based problem solving depends most on the first stepDefine the problem to be solved with:
• Precision(who, what, where, when, how often, why)
• Clarity(general agreement on priorities)
10
Why Use Data for Decision Making?
Data help place the “problem” in the local context rather than on the students.Data help us ask the right questions…they do not provide the
solutions. Use data to:
o Identify problemsoRefine problems and understand the contextoDefine the questions that lead to a solution
11
Problem Solving with Precision
The statement(s) of a problem is important for team-based problem solving.o Everyone must be working on the same problem with the same assumptions.
Problems are often framed in the “primary” form. o Raises awarenesso Not useful for problem solving
Precise problem statements result from a detailed data review and are solvable.
Data Team Roles
Data Collectors
All staff are part of PBIS data collection at some level and should receive training, coaching, and materials on procedures!
Survey coordinators and responders (e.g., Tiered Fidelity Inventory, School Climate Survey)
Universal Screening measures
Behavioral incident referrals
Targeted intervention data (e.g., CICO points)
Individualized intervention data (e.g., behavior support plan data)
14
Team Facilitators/Leaders
Before meetingso Schedule and communicate expectationso Coordinate tasks with minute taker, data analyst, or others
During meetingso Keep meeting focused on agenda and scheduleo Encourage active participation across members
After/Between meetingso Ensure that information (meeting minutes and tasks) are sent outo Regularly organize updates at all-staff, grade-level, and other meetings
15
Data Analyst
Standing Meetings (PBIS team, grade-level teams, data teams)o Provide standard set of reports with updated progresso Provide special-request reports on request or when potential problems are
identified o Summarize findings in person or at least in writing to support interpretation
Special Meetings (e.g., parent-teacher conferences)o Provide special-request reports on requesto Provide support in interpreting data
16
Data Integrity and Confidentiality
Inaccurate Data
False Assumptions
Poor Decisions
Accurate Data
Better Assumptions
Good Decisions
What is Data Integrity?
Data Integrity• Maintaining the accuracy and
consistency of data over its entire life cycle
• Critical aspect to the design, implementation, and usage of any system that stores, processes, or retrieves data
For data to be useful, it should reliably match overall perceptions across staff, students, and families.
18
Garbage in, garbage out
19
Garbage Data
Garbage Results
Incredible System
What is Confidentiality?
Data ConfidentialityProtection of potentially identifiable information
(In SWIS): o Person-identifiable
o Site/Building-identifiable
o Account access
Data about staff and students are highly confidential
Regular review of district policies related to student, staff, and school-level data
Keep passwords confidential
Log out every time you step away
20
Drilling Down into Your Data
Drilling Down and The Game of Clue
22
When you drill down into your data, you are like a detectivewho follows clues to isolate a problem context and function until it is precise but still socially important to address.
“Drill Down” Approach
The Drill Down approach starts with the big picture and looks for potential “red flags” that might indicate a problem that might need to be addressed. Then we identify and address the problem context and finally the function
1.What is our current reality?
2.Is there a problem (red flag)?
3.What is the precise context & function of the problem?
What, where, when, who, how often, why? Be efficient, don’t “drown” by getting too specific (e.g., red heads with blue shoes)
23
Primary
System Update/Big Picture
Red Flag
Identify Problemwith
Precision
Use DATA to define…
a PRIMARYsummary statement
a PRECISEproblem
statementthen
24
Creating a Precise Problem Statement
Drilling Down with Different Tools
Step 1: Look at overall picture What are the levels, trends,
peaks, and/or valleys?
How does this year compare with last year?
How do our data compare with national/regional norms?
How do our data compare with our preferred/expected status?
Step 2: Filter to a more precise problem statement by adding in additional data/“clues”
25
Drilling Down with the TFI (Steps 1 & 2)
Step 1. Overall (Total Score) Step 2. Breakdown by Tier
26Levels, trends, peaks?
Drilling Down with the TFI (Step 3)
Step 3. Breakdown by Tier & Subscale
27
Drilling Down with the TFI (Step 4)Step 4. Review Specific Items Use the items to build an action plan
for areas that can be improved
28
Our TFI Total Score was below 80%o Which tiers?o Which subscale(s)?o Which items?o Compared to other/previous fidelity data
“We’ve had a decreasing trend in our Tier 1 Scale for the last three months. Looks like the subscale of Implementation is the area we need to focus on, specifically with defining and teaching expectations, discipline policies, as well as feedback/ acknowledgement. Additionally, our faculty and student/family/community involvement have decreased over the same time period…….The staff are reporting similar areas as ‘not in place’ and as ‘priorities for improvement’ on the SAS.”
Asking the Right Questions: Fidelity
Primary Statement
Precise Problem
Statement
Outcomes/Referrals at Tier I (Step 1)
Step 1. Summary of overall referral level patterns, trends, & peaks
30
Outcomes/Referrals at Tier 1 (Step 2 & 3)
Step 2: Review Core Data to get an overall picture of what is happening
31
Step 3: Identify “Red Flag” to investigate further
Outcomes/Referrals at Tier I (Step 4)
Step 4. Isolate red flag and search for contextual factors (one at a time)
32
Outcomes/Referrals at Tier I (Step 5)
Step 5. Based on the context of the behavior (what, where, when, who) identify the perceived functionor motivation
Filters: Date Range: 9/1-10/31 Behavior: Defiance Time: Across DaysLocation: Mostly Classroom Grades: Mostly 6th – 8th
33
Prioritizing Multiple Problems
Prioritizing problems will likely be based upon criteria such as:
oSeverity/intensity – How dangerous is the behavior(e.g., aggression vs. inappropriate language)?
oFrequency – How often is it currently occurring?
oTrend – Is it decreasing, increasing, or consistent?
TIPS II Training Manual (2013)
Drill Down Worksheet
Drill Down Worksheet
This Drill Down Worksheet allows you to track the “steps” of the drill down • Record-keeping• Efficient
“back-tracking” • Replication
36
SWIS Drill Down Activity:
Log into: www.pbisapps.orgSWIS Demo
account“We do” & “You Do”
37
SWIS Drill Down Activity:
As we go through the Drill Down process record our steps on the SWIS Drill Down worksheet
38
565 studentsGrades 6, 7 & 8
Harbor Haven MS: Is there a problem?
Lately, the faculty buzz around Harbor Haven is that student behavior is getting worse. When we started the school year, everything seemed fine. But several months later, the teachers are complaining that they are constantly dealing with students messing around and not getting to class in a timely manner after lunch and after other breaks and transitions.
What kind of statement is this….primary or precise?
What additional information do I need to get more precise?
15
Harbor Haven MS: Is there a problem?
Is there a problem?
If so, what is it?
16SWIS Core Reports
Scho
ol-w
ide
Data
Harbor Haven MS: SWIS Core Reports A
18
What Do I Know?
43
• I know pieces of information.
• But I do not know if any of this information is connected.
• I need to drill down to look for connections.
Choosing a Red Flag
What? Inappropriate language, defiance, disruption
Where? Hallway, playground, commons, cafeteria
Who? Majority of referrals are from 8th graders
When? 8:30am-10:30am, 11am-11:30am, & 1:30pm-2:15pm
44
Data Drill Down
23
Add Hallway to our “Include in Dataset” field.
Data Drill Down
24
Change the graph type to change the lens of analysis.
Data Drill Down
25
Data Drill Down
25
Use the summary to analyze problem size.
Data Drill Down
28
Add the clues you learn to the dataset to analyze with more precision and clarity.
Change the graph type to change the lens of analysis.
Use the summary to analyze problem size.
Data Drill Down
26
Data Drill Down
26
Data Drill Down
26
Data Drill Down for Connections
Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.
30
Precise Problem Statement & Solution Development
Hallway 6th Graders Tardies at 1:00
Possibly Attention Maintained
Drill Down
54
2018-2019X
X
X
X
X
X
Hallways
6th graders
1:00
Tardy
Obtain Peer Attention
Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.
1015
SWPBIS Basic Decision Guidelines
What are indicators that a student needs additional supports for externalizing social behavior?
o Tier I 0-1 major ODRFewer than X minor/total
o Tier II 2-5 major ODRsX-X minor/total
o Tier III 6+ major ODRsX+ minor/total
55
Is the problem a system-level or student-level issue (rule of 10)?
o System Issue10+ students within a similar context (e.g., location, time of day, grade) engaging in similar behavior(s)
o Individual/Small GroupFewer than 10 students within similar context engaging in similar behaviors
Solution Development
Target Area(s): Problem behaviors in the hallways
Goal: Reduce referrals for tardies in the hallways by 50%
Solution Component Action Step(s)
Prevention-Increase active supervision. Vice-principal available at that time to monitor hallway. Grade levels establish a supervision rotation between classes.
Teaching -Reteach behavioral expectations in the hallway areas.- Retrain staff on identifying possible motivation.
Recognition-Increase recognition for appropriate behavior.-Provide feedback tickets that can be collected and used for spirit wear.-Dance party for the classroom with the fewest referrals.
Extinction-Post weekly grade-level SWIS data.-Encourage all students to work for the dance party making peer attention for problem behavior less likely.
Correct. Consq. -Active supervision and continued early consequence (ODR)
Data collection-Survey staff and students to determine whether strategies (e.g., active supervision, teaching, and recognition) are implemented consistently-Use weekly SWIS data to evaluate change
Precise Problem Statement: Many 6th graders are having issues with being tardy in the hallway around 1:00, and the behavior might be maintained by peer attention.
Solution Components
What are the Action Steps?Who is
Responsible?By When?
How will Fidelity be Measured?
Notes/Updates
PreventionIncrease active supervision Vice Principals
All staffOngoing Staff visible in the hallways,
especially during the 1:00 passing period
Teaching
Teach behavioral expectations for the commons and hallways
Teachers will teach hallway expectations to their homeroom classes
January 15 Staff sign-off sheet next to the PBIS bulletin board in the Faculty Lounge to indicate completion
Incentives for the class that can have 10 students tell the principal the expectations first
RecognitionDance PartyRecognition tickets for school store
PBIS Team will coordinate Party by January 31Tickets by January 10
Dance Party earnedtickets used at the Spirit Store
ExtinctionPost weekly dataEncourage all students to work for the Dance Party
All staff Ongoing Weekly data posted in the commons and hallway
Correct. Consq.Active supervision and continued early consequence (minor/major ODR)
All staff Ongoing Staff visible in the hallways
What data?Who is responsible for
gathering the data?When/How often will
data be gathered?Where will data be
shared?Who will see the
data?
Data CollectionODR record SWIS Data Entry person and principal
share report with teachers.Weekly Emailed to staff and posted in the
hallways and commons for students
All staff and students
Goal: Reduce referrals for 6th grade tardiness in the hallways by 50%
Big Idea About Sharing Data
58
Know your audience• What does the team need to know
to make good decisions?
• What do staff need to know? Is there anything they need to do?
• Who else would benefit from seeing an update on student behavior (e.g., students, families, parent group)?
Tell the storyHelp paint the picture with data: What, where, when, who, how often, why?
Big Idea About Sharing Data
59
Drill DownExample #2
Drill Down #2
Let’s see what happens when we start our drill down procedure with a problem behavior instead of a location.
Staff are reporting that there is a rise of inappropriate language in the K-3 wing of the school.
I’ll open up the Drill Down Tool
For this demo, I’m going to change the dates to last school year. You can choose to limit or expand the time frame depending on your search.
Don’t forget, I’m going to record my steps carefully using the SWIS Drill Down Worksheet. I can always use this sheet to backtrack if I need to.
61
Drill Down #2
71
Tools to Get Organized with
SWIS
Drill Down Video Tutorial
73https://www.pbisapps.org/Resources/Pages/Data-Drill-Down-How-To.aspx
Drill Down for Disproportionality
74PBISApps.org | Ethnicity: Data Drill Down
Drill Down Worksheet
75
SWIS Schedule
76
You can also check out the back cover of your SWIS manual for a reporting schedule. This will come in handy for delegating which reports will be run and by whom.
Connect With Your SWIS Facilitator
77
Your SWIS facilitator has been specifically trained on how to effectively use the Drill Down tool. If you have any questions or need any pointers, you can always get in touch with them.
Not using Drill Down?
78
Not using Drill Down is like throwing darts blindfolded. Good luck hitting the target!
~Tia Rupe, SWIS Facilitator
Review
Questions? Comments? Thanks for attending our Drill Down session today. Please don’t hesitate to reach out to us either at [email protected]
or [email protected] if you need any assistance or have further questions!
79