Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum
description
Transcript of Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum
![Page 1: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/1.jpg)
Evaluation in Michigan’s Model
Steve [email protected]
National PBIS Leadership Forum
October, 2010
http://miblsi.cenmi.org
![Page 2: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/2.jpg)
MiBLSi Evaluation Team(the people that make this presentation possible)
• Anna Harms, Evaluation Coordinator
• Ed Huth, Data Analyst
• Jennifer Rollenhagen, PBIS Assessments Coordinator
• Terri Metcalf, Reading Assessments Coordinator
• Nikki Matthews, Data Entry, PBIS Surveys Support
• Nancy Centers, DIBELS, AIMSweb Support
• Donna Golden, Data Entry
• Steve Goodman, MiBLSi Co-Director
![Page 3: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/3.jpg)
Mission StatementTo develop support systems and sustained implementation of a data-driven, problem solving model in schools to help students become better readers with social skills necessary for success.
![Page 4: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/4.jpg)
Cumulative Totals of Supported MiBLSi
Schools
![Page 5: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/5.jpg)
Building StaffBuilding StaffBuilding StaffBuilding Staff
Building Building Leadership Leadership
TeamTeam
Building Building Leadership Leadership
TeamTeam
LEA District LEA District Leadership Leadership
TeamTeam
LEA District LEA District Leadership Leadership
TeamTeam
Collecting information to evaluate implementation effects and using this information for continuous improvement
•Fidelity of implementation (across schools)•Systems integrity (district-LEA)•Student success (district-wide)
•Fidelity of implementation (state)•Systems integrity (project)•Student success (project-wide)
•Fidelity of implementation (across grades)•Systems integrity (school)•Student success (school-wide)
•Student success/Intervention effectiveness
ISD ISD Leadership Leadership
TeamTeam
ISD ISD Leadership Leadership
TeamTeam
MiBLSi ProjectMiBLSi ProjectMiBLSi ProjectMiBLSi Project
•Fidelity of implementation (across districts)•Systems integrity (district-ISD)•Student success
![Page 6: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/6.jpg)
Several Purposes of MiBLSi Assessments
• Audit
– for “taking stock” of current strengths/weaknesses and action planning
• Formative evaluation
– for improving program while it is in the process of being implemented
• Summative evaluation
– for improvement of future reiterations
![Page 7: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/7.jpg)
Assessments
Elementary Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• Benchmarks for Advanced Tiers (BAT)• Dynamic Indicators of Basic Early Literacy
Skills (DIBELS)• Planning and Evaluation Tool (PET) for
Effective Schoolwide Reading Programs• Effective Reading Support Team
Implementation Checklist• Special Education Data Collection Form• Schoolwide Reading Analysis Support Page
Middle/Junior High Schools• Major Discipline Referrals• PBIS Self-Assessment Survey• PBIS Team Implementation Checklist• Benchmarks of Quality (BOQ)• Schoolwide Evaluation Tool (SET)• ORF/MAZE through AIMSWeb• School-Wide Evaluation and Planning
Tool for Middle School Literacy (SWEPT)
• Middle School Reading Team Implementation Checklist
• Special Education Data Collection Form
![Page 8: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/8.jpg)
![Page 9: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/9.jpg)
![Page 10: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/10.jpg)
![Page 11: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/11.jpg)
Building LevelBuilding Level
![Page 12: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/12.jpg)
Assist Teams in Using Data for Decision-making
• First Year– Winter systems review
– Spring Data Review
• Second Year– Fall data review
– Winter data review
– Spring data review
• Third Year– Fall data review
– Winter data review
– Spring data review
![Page 13: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/13.jpg)
Assessment Booklet
• Description of assessments
• Data collection schedule
• Data summary• Data forms and
assessment forms
![Page 14: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/14.jpg)
Team Evaluation of Outcome, Process and Systems Data
![Page 15: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/15.jpg)
Assessment Schedule (for Cohort 7 from MiBLSi website)
![Page 16: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/16.jpg)
Video examples for completing and submitting PBIS assessments
![Page 17: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/17.jpg)
Improving the accuracy and Consistency of Recording Office Discipline Referrals
![Page 18: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/18.jpg)
Developing Fluency with Discipline Referral Categories
Example Exercise 2: Match the example situation below to the correct problem behavior on the discipline categories answer sheet. Write the letter in the column for Exercise 2.
Example Situation
A. Issac picked up his lunch and notice that another student was sitting in his usual spot at the table. Issac squeezes into his spot forcing the student to abruptly slide off the seat and onto the floor.
B. Ian comes to school wearing a T-shirt with a sexually suggestive picture.
C. During the test, Timothy is looking at Jennifer’s paper and copying her answers.
D. Taneshia walks to school each morning. She plays in the yard across from the school until the bell rings. After she sees al l the student run into the building, she walks towards the school, late again.
E. During lunch, Franz collects the empty milk cartons from his classmate. He sets them on the floor in a line and then stamps on the cartons making a loud noise with each stomp .
F. Alexandra has been following Ben around the playground. She sometimes makes various noises (that seems to bother him) when she is near. Sometime she tells him how she and her brother will stop him on the way home from school and beat him up.
G. Susan and Jill are best friends. Jill lives several houses down from the school. During lunch Jill talks Susan into coming over to see her new puppy. The school has a rule about staying on school property at lunch.
H. Billy’s teacher asks the class to take out their math book. Billy groans and says that math is stupid while not taking out his book. His teacher then tells him he needs to get his book from his desk. Billy says he hates math, he hates school and he is not going to do it.
I. Mr. Walter happens to walk into the boy restroom to find Matt and Chris wadding up paper towels and stuffing them into the toilet.
J. Rebecca has been having difficulty with her assignments at school. Her teacher, Ms. Swanson, sent a note home with Rebecca to be signed by her parents and returned the next day. However, when the note was returned the signature was Rebecca’s mother’s name but written in Rebecca’s handwriting.
K. The first grade class writes in a journal. The topic focuses on a particular letter each day. During th e letter “F” day, Sarah writes a word she has learned from some tough kids on the playground. She shows her journal to all of the students around her.
Example Situation
A. Issac picked up his lunch and notice that another student was sitting in his usual spot at the table. Issac squeezes into his spot forcing the student to abruptly slide off the seat and onto the floor.
B. Ian comes to school wearing a T-shirt with a sexually suggestive picture.
C. During the test, Timothy is looking at Jennifer’s paper and copying her answers.
D. Taneshia walks to school each morning. She plays in the yard across from the school until the bell rings. After she sees al l the student run into the building, she walks towards the school, late again.
E. During lunch, Franz collects the empty milk cartons from his classmate. He sets them on the floor in a line and then stamps on the cartons making a loud noise with each stomp .
F. Alexandra has been following Ben around the playground. She sometimes makes various noises (that seems to bother him) when she is near. Sometime she tells him how she and her brother will stop him on the way home from school and beat him up.
G. Susan and Jill are best friends. Jill lives several houses down from the school. During lunch Jill talks Susan into coming over to see her new puppy. The school has a rule about staying on school property at lunch.
H. Billy’s teacher asks the class to take out their math book. Billy groans and says that math is stupid while not taking out his book. His teacher then tells him he needs to get his book from his desk. Billy says he hates math, he hates school and he is not going to do it.
I. Mr. Walter happens to walk into the boy restroom to find Matt and Chris wadding up paper towels and stuffing them into the toilet.
J. Rebecca has been having difficulty with her assignments at school. Her teacher, Ms. Swanson, sent a note home with Rebecca to be signed by her parents and returned the next day. However, when the note was returned the signature was Rebecca’s mother’s name but written in Rebecca’s handwriting.
K. The first grade class writes in a journal. The topic focuses on a particular letter each day. During th e letter “F” day, Sarah writes a word she has learned from some tough kids on the playground. She shows her journal to all of the students around her.
Problem Behavior Exercise 1 Exercise 2 Exercise 3
Inappropriate language
Fighting physical injury
Defiance/Disrespect
Lying/Cheating
Harassment/Tease/Taunt
Disruption
Excessive Tardiness
Truancy
Property Damage
Forgery/ Theft
Other
![Page 19: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/19.jpg)
District LevelDistrict Level
![Page 20: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/20.jpg)
Increase 8%
Decrease 14.6%
Focus on Implementing with Fidelityusing Benchmarks of Quality (BoQ)/ODR ’06-’07 and ’07-’08
![Page 21: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/21.jpg)
District Implementation Tracking Form
![Page 22: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/22.jpg)
Leadership-Implementation Support Team Self-Assessment
![Page 23: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/23.jpg)
Project LevelProject Level
![Page 24: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/24.jpg)
File Maker Pro Data Base
![Page 25: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/25.jpg)
One major activity of MiBLSi involves professional development
Over 422 training days are currently scheduled for the 2010-2011 school year.
![Page 26: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/26.jpg)
![Page 27: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/27.jpg)
On-Line evaluation
• Trainer evaluation of trainer workdays
• Participant evaluation of training sessions
![Page 28: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/28.jpg)
On-Line Evaluation Form
![Page 29: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/29.jpg)
Trainer Work Day Questions1. The training goals were clearly defined and reviewed frequently
with checking for understanding.
2. The trainers were knowledgeable about the training content and were able to respond to participants' questions and share experiences to support understanding.
3. The trainers presented the content in such a way that promoted active engagement and opportunities for processing, working, and/or learning the content.
4. The materials were accessible in a timely manner (posted two weeks prior to trainer work day).
5. The trainer notes and activities of the day were a valuable use of my time as it relates to preparing for this upcoming training.
6. Potential challenges that participants may experience were highlighted with some ideas for addressing those challenges.
7. The big ideas of the day's training were emphasized, and areas to cut or condense were described in enough detail so that I am confident about how to adjust for different groups.
![Page 30: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/30.jpg)
MiBLSi Project Data
![Page 31: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/31.jpg)
MiBLSi Project Data
• Behavior and Reading Interaction
![Page 32: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/32.jpg)
Proficiency on 4th Grade High Stakes Reading Test and Percent of Major Discipline Referrals
from Classroom: 132 Elementary Schools
![Page 33: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/33.jpg)
Average MEAP Reading Scores and fidelity in PBIS implementation based on Benchmarks of Quality
*29 Elementary Schools from multiple districts
![Page 34: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/34.jpg)
MiBLSi Project Data
• Implementation Fidelity
![Page 35: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/35.jpg)
Comparison of Schoolwide Evaluation Tool (SET) Scores after training and after MiBLSi
Implementation
![Page 36: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/36.jpg)
MiBLSi School-wide Evaluation Tool (SET) Average Scores for Elementary and Middle Schools
for 2009-10
![Page 37: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/37.jpg)
MiBLSi Project Data
• Student Outcome
![Page 38: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/38.jpg)
Percent of Students meeting DIBELS Spring Benchmark for Cohorts 1 - 5
(combined grades)
![Page 39: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/39.jpg)
Percent of Students at DIBELS Intensive Level
across year by Cohort
![Page 40: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/40.jpg)
Elementary Schools with complete data sets: Average Major Discipline Referrals per 100
Students per Day
![Page 41: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/41.jpg)
Spring Reading Curriculum-Based Measurement “Established Level” for Cohort
4-6 Middle Schools
![Page 42: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/42.jpg)
Special Education Referral and Eligibility
Rates for Cohort 1 - 4 Schools (Comparison of 2007-08 and 2008-09)
*n = 84 schools
![Page 43: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/43.jpg)
Middle Schools with Complete Data Sets: Average Major Discipline Referrals per 100 Students per Day
![Page 44: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/44.jpg)
Lesson Learned
• Teams need to be taught how to analyze and use data
• Emphasis on directing resources to need and removing competing activities
• As we grow, it is even more important to systematic gather data that is accurate and then act on the data for continuous improvement
![Page 45: Evaluation in Michigan’s Model Steve Goodman sgoodman@oaisd National PBIS Leadership Forum](https://reader036.fdocuments.us/reader036/viewer/2022062520/56815a6f550346895dc7d0ca/html5/thumbnails/45.jpg)
“Even if you’re on the right track, you’ll get run over if you just sit there”
- Will Rogers