Elana R. Weinberger, M.Ed. Melissa L. Greene, Ph.D. William J. Matthews, Ph.D. John M. Hintze, Ph.D....
-
Upload
isaac-buchanan -
Category
Documents
-
view
216 -
download
1
Transcript of Elana R. Weinberger, M.Ed. Melissa L. Greene, Ph.D. William J. Matthews, Ph.D. John M. Hintze, Ph.D....
Elana R. Weinberger, M.Ed. Melissa L. Greene, Ph.D.William J. Matthews, Ph.D.John M. Hintze, Ph.D.
PROGRAM EVALUATION OF POSITIVE BEHAVIOR SUPPORT IN AN ALTERNATIVE SCHOOL
National Association of School Psychologists Annual ConventionFebruary 27th, 2009
Boston, MA
Positive Behavior Support (PBS)
Rooted in the field of behaviorism and applied behavior analysis (ABA)
Defined as the “application of positive behavioral interventions and systems to achieve socially important behavior change” (Sugai et al., 2000)
Systems approach to promoting positive behaviors of students and preventing and responding to school and classroom discipline problems
By using positive reinforcement primarily, PBS creates and maintains safe learning environments where teachers can teach and students can learn
Use of PBS has become more common and necessary in recent years as a method of preventing school violence
Purpose of Evaluation
To evaluate the strengths and weaknesses of a PBS program as implemented in an alternative school, and to implement changes to the program in an effort to improve program effectiveness and positive outcomes for students
Design
Program Evaluation using an exploratory case study design
Three phase model• Phase 1: initial evaluation (11/07-12/07)• Phase 2: intervention (1/08-3/08)• Phase 3: follow-up (4/08-5/08)• All data collected in phase 1 were collected again
in phase 3
Evaluation Questions
1. Are desired behaviors defined in observable terms?
2. Do students understand behavioral expectations?
3. Do staff understand behavioral expectations?
4. Is there inter-rater reliability among staff in terms of point allocations for behaviors?
5. Is the school-wide positive behavior support system implemented consistently?
Evaluation Questions (continued)
6. Do staff buy-in to the importance of the positive behavior support program?
7. Do staff find the positive behavior support program effective?
8. Does the school-wide positive behavior support system lead to positive student behaviors?
9. Does the school-wide positive behavior support system foster a safe school environment?
Setting
12-months per year, five-days per week, comprehensive interdisciplinary, psychiatric day treatment program
Collaboration between a private hospital and local school district
4 classrooms, referred to as teams in which students are grouped by grade and ability, 8:1:1 ratio
Program accommodates 25 students, ages 5-12 Diverse staff that includes professionals from a variety of
disciplines Students receive milieu, individual and group therapy, as
well as medication management and educational services
Challenges before PBS
Two separate systems under one roof, trying to provide different services to the children• Different point of reference for educational and
clinical staffs• Each system uses its own “language”
Not enough focus on the positives Inconsistent behavioral expectations throughout the
program Poor communication among staff
School-Wide PBS
Joined New York State PBIS initiative in 2004 Required commitment of hospital and education
administration Received support from PBIS coaches Created multidisciplinary committee Created matrix of behavioral expectations in
collaboration with educational and clinical staff
School-Wide PBS
Daily Level System• Students are given 0, 1 or 2 points every 30 minutes in four areas
• Respect, Responsibility, Safety, & Citizenship• Points are allocated by teacher, TA and MHW in student’s
classroom• Daily reward system based on accumulation of points and
corresponding level earned (2-5) Students can automatically be dropped to Level 2 for unsafe behaviors Staff is encouraged to use positive statements when interacting with
children at all times Students are encouraged to take Cool Downs to manage their behavior Staff instruct students to take Time Out when behavior cannot be
managed in classroom Weekly rewards given based on accumulated daily levels
Point Sheet
Time: Children earn 0,1,2 for each category every half hour
I was safe I was responsible
I was Respectful
I was a good citizen
Staff initials: 1:1 feedback Codes: GW= Good work ☺ CO=Cool down TO =Time out NFD= Not following directions Ag= Aggression (staff or peers) D= Disrespectful (staff or peers) ↓2 = Level 2 drop
8:30-9:00
9:00-9:30
9:30-10:00
Participants
Students and staff• Staff consent, parental verbal consent and
student assent obtained prior to enrolling
N ~ 50• n = ~25 students (ages 7-12)• n = ~25 staff
Data Collection Procedures:Quantitative
Frequencies• Daily levels earned by students• Level 2 drops (instances of unsafe behavior)
Inter-rater reliability • 25% of point sheets completed by two staff
members independently for a period of 2 weeks, for each classroom during phases 1 and 3
Data Collection Procedures:Quantitative/Qualitative
Staff Survey assessed 1) Buy-in to the PBS program, 2) understanding of the PBS program, and 3) implementation of the PBS program
Combination of 20 Likert-scale items and 8 open-ended questions
• Likert scale items regarding buy-in were rated on a scale of 1-3 (not important to very important)
• Likert scale items regarding understanding and implementation of program were rated on a scale of 1-4 (strongly disagree to strongly agree)
Surveys were completed anonymously
Data Collection Procedures:Qualitative
Staff focus groups were completed separately for educational and hospital staff
Facilitated by one of the evaluators Two independent note takers recorded staff
comments Thematic elements were summarized by the
evaluators after each focus group
Focus Group Protocol
In your opinion, what is the purpose of the PBS program?
Does the current implementation of the PBS program lead to positive student behaviors, and a safe school environment? Please explain?
Do you feel that you have received adequate training and support to reliably score point sheets and deliver time-outs and level 2 drops?
What elements of the PBS program would you change in order to make it more effective and supportive of staff and students?
Data Collection Procedures:Qualitative
Student Focus Groups• What do respect, responsibility, safety
and citizenship mean?• What behaviors do you have to do to
earn Level 4 or 5?• What behaviors would cause you a get
Level 2 drop?
Results: Phase 1 Assessment (11/07-12/07)
Overall reliability of point sheets is good (.86)• Reliability by behavioral category ranged from .81 to .92• Areas for improvement: Overall reliability can be better, and better
agreement on “Responsibility” (.82) and “Respect” (.81) Survey revealed that staff as a whole is invested in PBS
program (average score = 2.6/3.0) Understanding of program (average score = 2.5/4.0) and
implementation of program (average score = 2.6/4.0) need improvement
Level 2 drops account for 14% of total number of levels earned Levels 4 or 5 account for 70% of total number of levels earned
Feedback from Focus Groups
Staff was in agreement on understanding purpose of PBS program
• General agreement on modeling and teaching positive behavior
• Increase the focus on education Mixed responses regarding the success of the
current implementation and creating safe learning environment
• Some felt that it works, but we were not fully implementing the program, while others felt that it was not effective
• Strong sentiment that program was not as effective as it had been in past and we were not implementing it consistently
Feedback from Focus Groups
Across the board, staff felt that they had not received sufficient training• Most staff felt that they had not been sufficiently
trained, but learned on the job• This led to discrepancies in implementation
• Most staff felt they personally could benefit from ongoing training
Feedback from Focus Groups
When asked what they would change, staff discussed need for additional staff and training
In addition, staff identified need for modifications to reward system, and use of more visual reminders and tools for classroom and milieu
Feedback from Student Focus Groups
Children were aware of rules of conduct, but unclear as to distinctions between four categories of behavior
• Safety was most clearly understood
Younger children unclear about distinctions between use of cool down, time out, and level drop
• Lack of clarity between behavior and consequence
Phase 2: Interventions(1/08-3/08)
Met with PBS committee and staff regarding results of assessment
Conducted 5 staff trainings Worked with teachers to provide “refreshers” to
students around behavioral expectations Created new posters to hang in classrooms and
milieu to remind students about expected behaviors Updated matrix of behaviors Updated staff manual and created staff training
protocol for new staff
Results: Phase 3 Follow up(4/08-5/08)
Overall reliability of point sheets remains the same (.87)• Reliability by behavioral category ranged from .82 To .92• Reliability improved on “Responsibility” (.85) and “Respect” (.83), indicating
better understanding of these categories Survey revealed that staff investment in PBS program diminished
slightly (average score = 2.4/3.0) Understanding of program (average score = 3.0/4.0) and
implementation of program improved (average score = 3.1/4.0) Level 2 drops account for 17% of total number of levels earned, which
is a slight increase from phase 1 Levels 4 or 5 account for 66% of total number of levels earned, which
is a slight decrease from phase 1 Non-parametric (Wilcoxon) test revealed that these differences were
not statistically significant
Feedback from Phase 3 Focus Groups
Staff continues to be in agreement on the purpose of the PBS program
• General agreement on modeling and teaching positive behaviors
• Promote a safe environment for staff and students• Increase focus on learning
General consensus that the current PBS program may not be an adequate approach for some of the students at Bard House, but is working well for others
PBS would be more effective if it were reinforced at home Some feel that PBS promotes a safe environment, others feel
that the program is not a safe environment despite PBS
Feedback from Phase 3 Focus Groups
Active supervision on PBS is needed Some issues were indirectly related to PBS effectiveness
• need for additional staff• staff burnout• poor communication between educational and hospital staff
Across the board, staff felt that the trainings that were given were helpful and adequate
Trainings helped improve confidence, understanding and implementation of program
Some staff still feel that they are inconsistencies regarding what constitutes a level drop
Feedback from Phase 3 Focus Groups
When asked what they would change about the program at this point, staff discussed:• Need for additional staff • More effective and consistent rewards for
students• Better communication among staff• Positive reinforcement for students at home
Discussion
The evaluation was successful in that we were able to identify strengths and weaknesses, and identify areas of concern to be addressed through interventions
We were able to implement a variety of interventions, and received feedback that interventions were successful
• For example, interventions increased staff confidence, understanding, and implementation of the program
• We also increased accuracy of utilizing point sheets Through this evaluation, we were not able to effectively change
overall safety of program or student behaviors We were able to develop plan for future trainings and program
modifications that we were not able to implement during our limited time frame
Limitations
Incomplete data collection due to variety of factors.• Fewer surveys were returned in phase 3 than in phase 1• Student focus group could not be completed in phase 3 due
to scheduling Internal evaluation, inherent biases and censoring of
participants Ongoing staff and program changes Timeline may not have been sufficient to see an effect on
student outcomes Evaluation was affected by the larger systems within which the
program exists (hospital, public school)
Recommendations
Ongoing evaluation of the program Ongoing staff trainings to increase reliability of point allocations
and level drops Continue to tailor PBS to meet the needs of program Increase communication among staff Increase involvement of parents and caregivers with the PBS
program Continue to implement program changes over summer and
next academic year