FINAL OUTCOMES REPORT MEASURING THE EFFECTIVENESS OF · PDF fileFINAL OUTCOMES REPORT ....

1

FINAL OUTCOMES REPORT

MEASURING THE EFFECTIVENESS OF THE YORK CENTRE’S MAKING A DIFFERENCE -

ANTI-BULLY PROGRAM

BRUCE JOHNSON, COMMUNITY OUTREACH

HEATHER BYERS, COMMUNITY OUTREACH

KAREN MCKINNEY, COMMUNITY OUTREACH

CARRIE MAZZEI, COMMUNITY OUTREACH

DEAN ROKOS, CLINICAL DIRECTOR

DILYS HANER, PhD CANDIDATE, CLINICAL DEVELOPMENTAL PSYCHOLOGY

MARIE-JOSEE EMARD, RESEARCH ASSOCIATE, THE ONTARIO CENTRE OF

EXCELLENCE FOR CHILD & YOUTH MENTAL HEALTH

CATHERINE CAPPADOCIA, STUDENT CONSULTANT

TANYA WITEVEEN (Researcher, The Centre of Excellence)

DATE: October 31, 2011

Making a Difference Anti-Bully Program Delivered to Select Grade 5 and 6 classes within both the York Region District and York Catholic District School Boards

2

EXECUTIVE SUMMARY Project Lead: DEAN ROKOS, CLINICAL DIRECTOR Organization Name: The York Centre for Children, Youth & Families Program Title: Community Outreach – Making A Difference The Community Outreach team assessed the effectiveness of their Making A Difference Anti-

Bullying Program. The program was delivered, for one hour per week for eight consecutive

weeks, to 41 grades five and six classes. The Making a Difference program is designed to

educate students on the definition of bullying and the understanding of bullying situations so that

they can develop strategies to protect themselves and others against bullying.

The Purpose:

To determine the effectiveness of the Making A Difference anti-bullying program with regards to:

(1) decreasing subjective reports of bullying victimization at school, (2) decreasing reports of

school absenteeism due to bullying, and (3) increasing knowledge of effective bullying reporting

procedures and as positive/assertive responses to situations in which participants are

victimized or witness bullying. The findings will be used to support future service planning and

delivery, and to contribute to best practices in the delivery of bullying prevention programs.

The Program:

The York Centre is a children’s mental health centre located in Richmond Hill, Ontario. The

Community Outreach team consists of four staff members who provide anti-bullying workshops

to classrooms within Public and Catholic schools of York Region. This program has been

designed to reduce the reported incidents of bullying and to assist students with managing

situations in which they are victims of bullying, witnesses to bullying, or perpetrators of bullying.

A modification of the Climate Assessment Survey, both pre- and post-intervention, is

administered to participants as a measure of program effectiveness. The program runs for eight

consecutive sessions over eight weeks. Each session is 60 minutes and targets a different

dynamic of bullying. Grades five and six participants develop a definition of bullying, understand

3

the difference between tattling and telling, learn strategies to get help, and develop assertive

responses to bullying.

The Plan:

This evaluation project began in November 2010. Fidelity checklists were implemented by all

four facilitators. Short-term outcomes were measured by comparing pre- and post- intervention

surveys upon completion of program delivery, which ended in June 2011. Teachers were given

a post-evaluation survey to provide a measure of adults’ perceptions of the program’s

effectiveness and to support the outcome and process evaluations.

The Product:

We collected data from participants in 41 classrooms in 25 schools. Pre-intervention data was

collected from 968 students and post-intervention data was collected from 914 participants. Z

tests for proportions were carried out and revealed no statistically significant differences before

the intervention between groups for gender or grade.

There was a statistically significant positive change from pre- to post-test on all questions that

measured knowledge of reporting procedures. Although results were statistically significant for

each of these items, effect sizes were small. This evaluation did not find significant results with

regards to subjective reports of victimization, witnessing, or perpetration of bullying. While we

continue to believe in the utility of the training materials and the delivery of our facilitators, we

are not entirely surprised by these findings. It is rare that bullying interventions find significant

differences in behaviour during such a short time line (our study comprised eight weeks only).

Future evaluators may wish to tweak the wording of these questions to further determine if the

construct being assessed is children’s knowledge or comfort with reporting procedures. In the

future, we would want to collect data at two additional time points: four weeks prior to the

intervention (baseline), and four weeks after the intervention (follow-up).

Amount Awarded: $34,000 Final Report Submitted: October 31, 2011 Region: MCYS-Central

5

Table of Contents

Final Report Introduction and Literature Review Description with Logic Model………………………………………………………………….. 7 Detailed Summary Session One ……………………………………………………………………………………. 9 Session Two ……………………………………………………………………………………. 10 Session Three ………………………………………………………………………………….. 11 Session Four …………………………………………………………………………………… 11 Session Five …………………………………………………………………………………… 12 Session Six …………………………………………………………………………………….. 12 Session Seven ………………………………………………………………………………… 15 Session Eight ………………………………………………………………………………….. 16 Evaluation ……………………………………………………………………………………… 17 Key Process Evaluation Questions …………………………………………………………. 19 Literature Review Week One ……………………………………………………………………………………… 20 Week Two ……………………………………………………………………………………… 21 Week Three ……………………………………………………………………………………. 22 Week Four ……………………………………………………………………………………… 22 Week Five ……………………………………………………………………………………… 22 Week Six ……………………………………………………………………………………….. 23 Week Seven …………………………………………………………………………………… 24 Week Eight …………………………………………………………………………………….. 25 Methodology ……………………………………………………………………………………26 Measures ………………………………………………………………………………………. 29 Limitations ……………………………………………………………………………………… 29 Results Knowledge of Bullying Reporting Procedures ……………………………………………… 31 Subjective Reports of Bullying ……………………………………………………………….. 32 Bullying and School Absenteeism …………………………………………………………… 34 Positive/Assertive Responses to Bullying …………………………………………………... 35 Knowledge Exchange Internal Linkages ……………………………………………………………………………… 38 External Linkages …....................................................................................................... 38 Conclusions and Recommendations/Next Steps …………………………………………. 40

6

Insights …………………………………………………………………………………………43 References …………………………………………………………………………………… 45 Appendices Appendix A – Power Point Slides …………………………………………………………... 50 Appendix B – Fidelity Checklists …………………………………………………………… 51 Appendix C – Logic Model ………………………………………………………………….. 59 Appendix D – Outcome Evaluation Matrix ………………………………………………… 60 Appendix E/F – School Surveys…………………………………………………………… 63 Appendix G – Community Group, Coleader Evaluation ………………………………… 85 Appendix H – Process Matrix ……………………………………………………………… 88 Appendix J – Teacher Satisfaction Data …………………………………………………. 93

7

Final Report Introduction and Literature Review

Description with logic model

The Making A Difference program is an eight-session workshop delivered to classrooms from

grades four to eight across both York Region Public and Catholic School boards. The program

was initially designed by Bruce Johnson, Community Outreach Worker for The York Centre for

Children, Youth & Families. Mr. Johnson initiated this program in 2000 and provided two

classes to 19 students within this pilot year. Over the past 11 years, the program has been

developed and expanded. Bruce has since trained the other members of the Community

Outreach Team (Heather Byers, Carrie Mazzei & Karen McKinney) to deliver the program.

Training was initiated in a one-to-one setting where Bruce reviewed the theme of the program,

the eight individual sessions, and the research to support the content and activities. Training

continued in the field through observation of Bruce (and later Heather) delivering the program.

Since 2000, 498 classes and 12,757 students in York Region have received the Making A

Difference program.

Over the past two years, the Community Outreach team has held three, full-time equivalent

positions dedicated to delivering this program. Modifications have been made to the program

involving the inclusion of visual aids, such as the power point presentation and an increase in

small and large group activities. These activities were added to support recommendations from

our stakeholder survey, which included co-leading teachers and the Community Outreach Team

members themselves. Prior to the implementation of the research project, the team met weekly

from September through to November 2010 to consolidate and finalize power point presentation

materials (Appendix A) and fidelity checklists for each of the eight sessions that ensure the

program can be delivered consistently by all facilitators (Appendix B).

8

The Making A Difference Program – is two-fold in its design. The first four sessions are

designed to increase students’ knowledge regarding the definition of bullying, the various forms

of bullying, and the tactics used by people who bully. The latter four sessions are designed to

increase students’ awareness and understanding of how to respond assertively and how to

report bullying situations (Logic Model- See Below/Appendix C; Outcome Evaluation Matrix –

Appendix D).

Program Logic Model-long term outco

9

Detailed Summary of Sessions One Through Eight

Week One – Definition of Bullying

• The students are given the pre-intervention survey [Making A Difference, School Safety

Pre-survey; Appendix E(i) YRDSB and E(ii) YCDSB] to assess their sense of bullying

issues in the school community.

• The students draw a bullying situation and are encouraged to use speech and thought

bubbles to indicate what characters are thinking and feeling.

• The four-4 part definition of bullying is taught through direct instruction and it is

reinforced by the power point presentation and a video, “Katie’s First Day, “ from the

program Stop Bullying Now (http://www.stopbullying.gov/kids/webisodes/index.html).

• While the four-part definition of bullying is taught through direct instruction, it is

reinforced by the power point presentation, a video, and the students’ personal

drawings.

• The four-part definition of bullying is: 1. People who bully misuse their power; 2. People

who bully have an intention to harm one or more people; 3. The target(s) are hurt; 4.

People who bully repeat their misuse of power and actions. Students are also informed

that bullying can occur through a number of group dynamics such as: one person targets

another, one person targets a group, a group targets one person or a group can target a

group.

• The key point: Bullying is a chosen behaviour in which people misuse and abuse

their power to intentionally target others. By using the definition students are

better able to identify bullying situations.

10

Week 2- We are Loveable & Capable

• The class reviews the previous lesson and participates in a large group discussion.

• The Real Friends video from the Stop Bullying Now series

(http://www.stopbullying.gov/kids/webisodes/index.html) is presented and students are

asked to watch and determine how one of the characters makes a difference to help

another. The video is reviewed through a class discussion.

• As a whole-class activity, students are asked to share how they have made a difference

in the past week to help someone in need.

• The power point presentation focuses on what makes all of us loveable and capable.

• Students colour a poster and identify and share personal experiences about: 1. Two

people who love and care for them; 2. Three people they can turn to for help; 3. Three

activities they enjoy.

• This lesson is designed to empower students and help them see that they are all

loveable and capable of making healthy decisions, and have the right to be treated with

respect at all times.

• The key point: All people are special. People who bully target what makes us

special to maintain their misuse of power. We are all lovable and capable and we

all deserve to be treated with respect.

11

Week 3 – Forms of bullying

• In a whole-class activity, students are asked to share how they made a difference in the

past week to support someone in need of help.

• The class reviews and discusses the previous lesson as a group.

• Students respond to the questions, “What are some physical bullying behaviours?” and

“What are some verbal bullying behaviours?”

• Students work in groups of five or six. There are five home groups created and then

students within these groups split off to colour-coded expert groups. Each expert group

learns about one form of bullying (such as: social, electronic, racial, religious or

disability). Experts then go back to their home group and share the one form of bullying

they have just learned. All forms of bullying are then reviewed by the Community

Outreach Worker.

• A Video is presented to close the session (Don’t Laugh At Me by: Project Respect Steve

Seskin/Allen Shamblin Cross Keys Publishing).

• The key point: There are many ways people choose to bully others. By

understanding the various forms of bullying, students are better able to identify

bullying situations and seek appropriate help.

Week 4 – Tactics

• Students are asked to share how they made a difference over the past week to support

someone in need of help.

• The entire class discusses and reviews the lessons of weeks one through three.

• Students are taught the difference between direct and indirect bullying and the seven

tactics that can be used by people who bully to stop others from reporting such

incidents.

12

• Students learn to identify the above mentioned tactics through role plays.

• Students watch the video (K.B.’s Day, Stop Bullying Now series

(http://www.stopbullying.gov/kids/webisodes/index.html) and identify the forms of

bullying, tactics, and direct or indirect behaviours noticed.

• The key point: People who bully will use a number of tactics to convince

bystanders and targets to not seek support.

Week 5- Responding to Bullying Behaviours

• Students are asked to share with the class how they made a difference over the past

week to support someone in need of help.

• The entire class discusses and reviews the lessons of weeks one through four.

• Through direct instruction, students are taught the three ways people respond to bullying

behaviours (passive, aggressive, and assertive).

• Students role play various assertive responses to a number of bullying situations.

• Students are asked to watch a video (Miltons’ Dream, Stop Bullying Now Series,

http://www.stopbullying.gov/kids/webisodes/index.html) and identify the ways one

character responds to a number of bullying situations.

• The key point: People’s responses to bullying behaviours help to determine

whether the situation worsens or improves.

Week 6 – Safety in Numbers – Who Should We Support?

• A puzzle activity opens this session. The word “safety” is printed on the display

(black/white board) with random numbers surrounding it. The answer is “safety in

numbers”. The students are asked to try and figure out the puzzle as the answer is the

theme of this week’s session.

http://www.stopbullying.gov/kids/webisodes/index.html

13



• The class reviews the previous week’s workshop on response styles.

• Through the use of the power point presentation students are visually introduced to all

the players involved in bullying situations (people who bully, people who are targeted;

and the three types of bystanders: those who support the people bullying, those who say

nothing and do nothing, and those who support the people being targeted).

• Students brainstorm in small groups to come up with a variety of reasons why people

choose to support the various players or choose to do nothing. This is then discussed in

with the entire class.

Large Group Activity, three parts:

1. Pictures of faces of the person who initiated the bullying, active supporters of the bully

and pictures of people who passively respond to the bullying are handed out by the

facilitator.

• Students are randomly given pictures of a face, which represents the various roles of

people involved in a bullying situation. The student who received the picture of the face

of the person who initiated the bullying is asked to stand on one side of the classroom.

The student who has the picture of the targeted person stands on the opposite side of

the classroom. The students who received the remaining pictures are asked to move to

the side of the classroom that their person pictured would support.

• Once this is completed students who have the active supporters faces are standing

beside the person who started the bullying; some students with the passive responders

are usually standing beside the person targeted or are milling around not going to either

14

side. The leader asks: “How is being quiet and doing nothing helping the person

targeted? (The response is “it doesn’t.” Then the leader asks: “Who benefits when

people do nothing?” (The response is the person who is bullying because if no one helps

the target, the bullying most likely will continue). The students who were milling around

or are on the “side of the target” are asked to move to the other side.

• The facilitator asks the children to sit down and asks the class: “Who is missing?

• (The response expected is “the people who will help the target”.)

• The facilitator points out “this is one of the main reasons why bullying continues. People

who could help don’t!”

2. The facilitator then states: “We will do this activity again.” The procedure is the same

with the addition the faces of people who help also now being distributed. This second

time through, the passive supporters quickly go to the side of the person who initiated

the bullying and the people who have the faces of the people who will help the target are

standing beside the target.

• The leader asks the person who has the face of the target if they like to have help.

(Response expected is: “Yes.”)

3. The leader then asks everyone to sit back down and states: “We will do this activity once

again”.

• The leader then randomly hands out the face of the person who initiated the bullying,

and the face of the target and directs them to opposite sides of the classroom. Then

loudly the leader states: “Bullying has just occurred, pick a side, hurry get up and go”.

15

• Once the children have chosen their side the leader states that behaviour is judged and

helping someone to hurt others is wrong. Helping someone who is hurt feel better and

safe is a good thing.

• The leader then asks the teacher if this visual demonstration of students is generally

how children in their class respond to bullying situations. After the teacher responds, the

leader then sends any children who chose to support the bullying to move to the other

side and then ask the students “If all the people in this class said, ‘stop bullying’ do you

think the amount of bullying would increase or decrease? (Response expected is:

“Decrease.”)

The key point: The display board puzzle is pointed out to the students and they are left

with the message: “there is safety in numbers and the more people who take a stand and

support those who are being targeted, the less power the person who bullies has and the

more likely bullying will decrease”. “The more people to get involved to take a stand

against bullying, the more able we are to make a difference to reduce bullying and make

our school and community a safer place.”

Week 7 – Reporting Bullying


someone in need of help

• The entire class reviews the previous week’s lesson on who students should support in

bullying situations.

16

• Students watch the video (Power in Numbers, Stop Bullying Now,

http://www.stopbullying.gov/kids/webisodes/index.html) and are asked to identify the key

players in the bullying situation. Students are also asked to identify the many assertive

responses that occur to reduce the harm of what was intended.

• Students are asked to provide an example of a bullying situation and then the student

providing the example is asked to rate the level of harm the target(s) incurred from 0 (no

harm) to 100 (maximum harm). The scale is displayed on the power point presentation

and the individual student can look to his/her classmates to help determine the

estimated amount of harm.

• After a couple of examples of maximum harm (such as injury or death) are discussed.

Students are informed that we all have the power to make a difference and anything we

can do to reduce the harm from bullying is a positive response.

• Students work in small groups to brainstorm and mind-map all the assertive strategies

they could use to reduce the harm from specific bullying situations. Each group has a

different bullying scenario. The group scenarios are reviewed with the entire class and

each group presents their assertive responses.

• Reporting – One Assertive response is to report the bullying incident. Students are

given direct instruction on how to use the 5W-H strategy (who, what, where, when, why

& how) to report bullying incidents. The more information bystanders give to trusted

adults the better able others are to make a difference.

• The key point: When we respond to bullying behaviours, we should always aim to

decrease the harm, stay safe, and seek help when needed.

Week 8 – Review & Future Capacity Planning



17

• In a game show format, students are placed in groups and the entire Making a

Difference program is reviewed via teams responding to questions.

• The Anti-Bullying Pledge – Students are given a written pledge to read and sign.

• Students, along with their teacher, are asked to brainstorm how they can inform the

students outside of their classroom about the Making A Difference program and how to

teach others to take a stand against bullying. Possible projects are discussed.

• Students are given the Making A Difference School Safety Post-Survey [Appendix F(i)

YRDSB and F(ii) YCDSB], an 18-question survey to evaluate their current feelings

around bullying at their school. The classroom teacher is also given an evaluation to

complete to provide feedback to the Community Outreach Team (Community Group Co-

leader Evaluation – Making a Difference; Appendix G).

• The key point: Every student has the power to Make A Difference and take a stand

against bullying.

Evaluation

The purpose of this evaluative study is to compare the pre-intervention and post-intervention

survey results to determine if the Making A Difference program had a positive impact on

participants’ behaviours. Specific evaluation questions to support the process evaluation are:

1. Does the Making A Difference program increase student participants’ knowledge of

effective bullying reporting procedures?

2. Since the start of the Making A Difference sessions, do student participants report a

decrease in bullying victimization at their school?

3. Has the Making a Difference Program had a positive effect on school absenteeism due

to bullying?

18

4. Have student participants reported increased positive/assertive responses to situations

in which they are victimized or witness bullying since the beginning of the Making A

Difference Program?

5. Do teachers observe a decline in bullying behaviours since the beginning of the Making

A Difference Program?

To further evaluate our research outcomes and support future planning of the Making a

Difference program, process evaluation questions were created to provide feedback from all

involved: The Community Outreach Workers (facilitators), the students (participants) and the co-

leaders (teachers).

19

Key Process Evaluation Questions

Facilitators (Fidelity Check, see Appendix B):

1. Did the program facilitators deliver the same program consistently?

2. What activities worked well? What activities did not work so well?

Teachers (Community Group, Co-Leader Evaluation - Making A Difference; Appendix G):

1. Were teachers satisfied with the quality of the Making A Difference Program?

2. Were teachers satisfied with the amount of information students received?

Students [Making A Difference School Safety Post-Survey, Appendix F(i) and F(ii)]:

1. Did the students feel the program was helpful?

Stakeholders

Stakeholders include both the York Region District School Board (YRDSB) and the York

Catholic District School Board (YCDSB). In total, 41 classrooms and 25 schools were served

throughout this evaluation. The Care and Treatment Program Facilitator for YRDSB, the

Behaviour Resource Specialist for YCDSB and the Behavioural Resource Consultants for both

boards as well as COMPASS have a vested interest in this project as they support the York

Centre’s Community Outreach Team and refer our services to specific schools and classes on

an ongoing basis. Other stakeholders involve our alliance centres such as: Blue Hills Child and

Family Centre, Kinark Child and Family Services, and the Children’s Aid Society of York Region

who also work with students within York Region and could benefit from future training with the

Making A Difference program. Finally our financial supporters, particularly the Ministry of

Children and Youth Services and the United Way are informed of this project and are awaiting

results.

20

Literature Review

A comprehensive review of the literature was conducted. This process was started by preparing

an outline of each of the Eight Making a Difference sessions (see above). This outline was used

to guide the literature search to support or disprove the use of our adopted practices.

Week 1

Dr. Debra Pepler is a clinical-developmental and research psychologist specializing in child

development and a world-renowned expert in the field of bullying. She has indicated that it is

not healthy to label children as “bullies” or “victims”. The language within the Making A

Difference program therefore refers to “people who bully” and “people who are targeted”. All

facilitators are taught early in the training process to avoid the terms “bully” and “victim.” Dr.

Pepler has also indicated that bullying programs should offer tailored versions of the program to

ensure content and delivery are developmentally appropriate for the age group of the

participants (Pepler, 2006).

Cross et al., (2008) found that it is important to have a clear definition of bullying; therefore, the

focus of the first session of the Making A Difference program is to provide students with a

definition of bullying and to support their understanding of how bullying situations can occur.

The definition used by the program is informed by research that has indicated that bullying is

defined by three elements: aggression (the intent to harm), a power differential, and repetition

(O’Connell, Pepler, & Craig, 1999; Olweus, 1993; Smith & Thompson, 1991).

Throughout the program students also gain insight as to how bullying is defined as a

“relationship problem” (Burgess, Garbarino, & Carlson, 2006; Craig & Pepler, 2007; Pepler et

al., 2006; Smith, 2008). In 2006, Pepler et al., conceptualized bullying “as a relationship

problem that requires relationship solutions because it is a form of aggression that unfolds in the

context of a relationship in which one individual asserts interpersonal power through

aggression”.

21

Week 2

In week two of the program, the focus of the session is to develop rapport with the students and

to support their sense of belonging, feelings of empowerment as well as to facilitate empathy.

Building the capacity for empathy is supported as an effective component of bullying prevention

programs (Cross et al., 2008). Research indicates that bullying behaviours are associated with

low levels of empathetic responsiveness, while empathy is related to helping victimized peers

(Szalavitz, 2010). Prosocial children show more empathy than peers who bully or are

victimized (Gini, Albiero, Benelli, & Altoe, 2007; Jolliffe & Farrington, 2006; Warden &

Mackinnon, 2003).

Peer victimization is associated with low levels of peer acceptance and high levels of peer

rejection (Boulton & Smith, 1994; Perren & Hornung, 2005; Salmivalli, Lagerspetz, Bjorkqvist,

Osterman, & Kaukiainen, 1996). As students decorate their “Loveable and Capable” poster,

they are asked to think about and later report; two people who love and care for them, three

people they can turn to for help, and three activities they enjoy. As the session progresses they

realize how students are the same and different from each other and that, despite their

differences, all students have the right to be accepted, respected, loved and have capabilities to

make a positive difference to help themselves and others. The final slide of session two Power

Point presentation asks students: “What kind of friend are you and how will you make a

difference?” The message they are left with is: respect everyone, honour everyone’s

uniqueness, and celebrate our goodness.

22

Weeks 3 and 4

In weeks three and four, the Making a Difference program offers interactive activities (group

work and role plays) to learn about the different forms of bullying and tactics used by people

who bully to maintain their power. Research suggests that indirect and direct aggression are

perpetrated differentially across gender and age (Card, Stucky, Sawalani, & Little, 2008;

DeSouza & Ribeiro, 2005; Pepler et al., 2006, 2008; Scheithauer, Hayer, Petermann, & Jugert,

2006; Woods & White, 2005). Indirect bullying includes social and relational forms of

aggression such as ignoring, gossiping, and deliberately leaving others out of a group, whereas

direct bullying includes verbal and physical forms of aggression such as hitting or insulting,

respectively. According to the research, boys tend to engage in physical forms of bullying and

sexual harassment more than girls (DeSouza & Ribeiro, 2005; Pepler et al., 2006). With

regards to electronic, verbal and social bullying Woods & White (2005) found similar patterns

exhibited by both boys and girls.

There is evidence that some peer groups have hierarchical structures in which some students

are leaders while others are marginalized and fear being ostracized from the group. (Craig &

Pepler, 2003). Within these groups, the leaders (not the members) decide who is included.

However, the majority of students take part in peer groups that are inclusive and lack formal

leadership roles (Adler, 1996).

Week 5

Whereas the first four sessions support students understanding of bullying behaviours, the final

four sessions focus on strategies and responses people can use to reduce the harm from

bullying. Among empirically supported bullying programs, a discussion of response styles when

being bullied is supported as an important component (Cross et al., 2008). Session five teaches

students there are three main communication response styles (passive, aggressive, assertive)

23

that people use to respond to bullying. The students learn about each response style and then

through group discussion, talk about which response style would be the best to use and why.

Facilitators support the students to understand that assertive response styles are most likely to

reduce the harm from bullying. Research evidence has supported an association between

victimization and more passive/non-assertive response styles and behaviours (Schwartz et al.,

1993; Sharp, 1996). For example, Sharp (1996) found that students with high self-esteem and

active response styles are as likely to be bullied as students with low self-esteem and passive

reactions; however the extent and effect of victimization are less severe for students with high

self-esteem and active response styles. As there is evidence that assertiveness training

reduces victimization among peers (Arora, 1991; Sharp & Cowie, 1994), the Making A

Difference facilitators’ primary focus is to support the students’ understanding and development

of assertive response styles.

Week 6

Research has noted that the peer group provides the relational context for bullying (Craig &

Pepler, 1995; Olweus, 1993; Salmivalli, Lagerspetz, Björkqvist, Österman, & Kaukiainen, 1996).

In Week six, the Making A Difference program, focuses on the role bystanders play in bullying

situations. Students learn that bystanders who watch a bullying situation and then “say nothing

or do nothing” to help those being targeted reinforce the bully behaviour. In 1999, O’Connell,

Pepler, & Craig found that peers were present in 85-88% of bullying interactions taking place on

the school playground. The more peers who gather to watch the bullying episode, the longer the

episode tends to last (O’Connell, Pepler, & Craig, 1999). When children gather to watch, they

spend most of the time passively watching, and they focus attention on the child/children who

bully, not the child being victimized. This peer attention reinforces the bullying behaviour. In the

majority of bullying episodes observed (57%), peer intervention stopped bullying within 10

seconds (Hawkins, Pepler, & Craig, 2001). In week six, the students investigate why

24

bystanders support the people bullying, why some say nothing and do nothing and why others

support the people being targeted. This lesson is both discussion and activity based. The

primary focus is to help students understand that bystanders who do nothing are actually

supporting the people who bully. The second message of this lesson is to help students

understand there is power and safety in numbers and the more people who get involved and

support those being targeted the less power those who bully have and a reduction in bullying

incidences are more likely to occur.

Week 7

This lesson continues to focus on developing assertive strategies, including proper reporting

techniques, for dealing with a variety of bullying situations.

Reporting to friends and trusted adults has been established as an important factor in reducing

the impact of bullying . Pellegrini & Bartini, (2000) and Pellegrini, Bartini, & Brooks(1999) assert

that having at least one friend is a protective factor against bullying. Furthermore, peers play a

large role in maintaining and/or stopping bullying episodes. In majority of bullying episodes

(57%), intervention by a peer stopped bullying within 10 seconds, regardless of what was said

by the intervener (Hawkins, Pepler, & Craig, 2001). Session 7 supports students to understand

the benefits of reporting and provides them with specific information on how to report bullying.

Research also helps to identify the importance of the teacher as a co-leader within the Making A

Difference program. Teachers and school staff are in a unique and influential position to

promote healthy relationships and to intervene in bullying situations (Pepler, 2006; Salmivalli,

Kaukiainen, & Voeten, 2005). The goal of the Making A Difference program is not only to

support students’ understanding of bullying and how to deal with bullying situations but to also

to have teachers present to build the capacity of the messages being taught to the student

25

participants as well as transfer this knowledge to other students and teachers within the larger

school setting. According to Alsaker, (2004), teachers who participated in a bullying prevention

program felt more confident about handling bullying problems, had more supportive attitudes

toward victimized students, and felt more positively about working with parents regarding

bullying problems than those who did not participate.

Week 8

During the final week of the Making A Difference program students reflect on the information

that was presented in weeks one through seven. This is done through a game show-like activity

wherein students work together in small groups to answer bullying related questions. The

session ends with an anti-bullying pledge and a class brainstorming session on projects or ways

in which the class can share the Making A Difference information with others in the school.

Encouraging students to support their own school is considered empowering and believed to

make a difference in social relationships. According to Stoneman (2002) and Zeldin (2004),

youth may provide more appropriate and credible solutions than adults regarding problems that

affect other children and youth, such as bullying. Youth are more likely to follow the example of

effective problem-solving skills when these are modeled by peers rather than by adults, and are

more likely to be interested and committed to prevention programs when these are led by peers

(Lerner & Galambos, 1998).

26

Methodology

Descriptives and Outcome Indicator Chart

We collected data from participants in 41 classrooms in 25 schools. Pre-intervention data was

collected from 968 students of whom 502 were boys (52.2%) and 459 were girls (47.8%). Out of

these 968 participants, 601 were in grade 5 (62.1%) and 367 were in grade 6 (37.9). Post-

intervention data was collected from 914 participants: 490 were boys (53.6%) and 424 were

girls (46.4%). Out of these 914 participants, 555 were in grade 5 (60.7%), 359 were in grade 6

(59.3%). Z tests for proportions were carried out and revealed no statistically significant

differences before intervention between groups for gender or grade.

Outcome Indicator Chart

OUTCOME EVALUATION MATRIX

Evaluation Questions

Short-Term Outcomes

Indicator(s) Measurement tool

Method to Collect Data & Frequency

Person responsible for data collection? Data source?

Dates of data collection (specify month/year)

STUDENTS 1. Does the Making A Difference Program increase student participants knowledge of effective bullying reporting procedures?

Increased knowledge of effective bullying reporting procedures for participants at school

Questions 4, 9, 10, 13*, 15*, (18 qualitative responses on post survey) Comparison between pre and post surveys

Making a Difference School Safety Survey

-Students complete the pre survey on the first session (week 1) of the Making A Difference program -Students complete the post survey on the last session (week 8) of the Making A Difference Program

Program Facilitators

-November 2010 through June 2011

2. Since the commencement of the Making A Difference program have student

Decreased subjective reports of bullying victimization at school for

Questions 5, 6, 7 Comparison between pre and post




27

participants reported a decrease of subjective reports of bullying victimization at their school?

participants surveys

3. Has the Making a Difference Program had a positive impact on school attendance due to bullying?

Decreased reports of school absenteeism due to bullying

Question 8 Making a Difference School Safety Survey



4. Have student participants reported increased positive/assertive responses to situations in which they are victimized or witness bullying since the commencement of the Making A Difference Program?

Students will increase their positive/assertive responses to situations in which they are victimized or witness bullying

Questions 12, 14, (18 qualitative responses on post survey)




TEACHERS Did the teachers observe a decline in bullying behaviours since the commencement of the Making A Difference Program

Teachers will observe a decline in bullying behaviours

Coleader Evaluation (Quantitative) Comparing

quantitative responses: 6 vs 8, 9 vs 10 to determine if 80% teachers report decreased bullying behaviours

Coleader Evaluation (Qualitative) Review

qualitative responses to question 7

-Coleader Evaluation

-Program Facilitators will provide the evaluations to classroom teachers and collect on the same day) -Teachers complete the evaluation -On the last day of the program (week 8)


January through June 2011

28

Evaluation Questions Method to (specify month/year)

Intermediate Outcomes





1. Have student participants increased their assertive strategies and decreased passive/aggressive strategies in their responses to bullying behaviour?

Increase use of assertive strategies and decrease passive/aggressive strategies to respond to bullying behaviour

TBD (To be determined)

TBD TBD TBD TBD

2.Has there been an increase in supportive and helping behaviours for those who witness bullying?

Increase helping and supportive behaviour for those who witness bullying

TBD TBD TBD TBD TBD

3.Have student participants from the original evaluation continued to report a decrease in bullying victimization at school?

Continued decrease in reports of bullying victimization at school for participants

TBD TBD TBD TBD TBD

4.Has there been a decrease in reported bullying behaviours throughout the school since the commencement of the Making A Difference program involving specific classes?

Decreased subjective reports of bullying victimization for those students in classrooms other than those that received the intervention

TBD TBD TBD TBD TBD

29

Measures:

We used a version of The Safe Schools Survey for grades five and six students (PREVNet,

2007). This survey is used annually by both the public and Catholic school boards as part of

their review of children’s developmental functioning. The Safe Schools Survey was developed

by psychologists at PREVNet, a Network Centre of Excellence that aims to reduce bullying in

Canada.

Sources of information from students and teachers:

Fidelity measures (Appendix B) were implemented to assure the program was delivered as

intended across the four presenters. Weeks one and two had nine and eight items respectively

on their fidelity checks. Weeks three through eight each had ten items. A dichotomized variable

was created to describe whether there was high or low adherence to the program. If three or

more items on the checklist were not adhered to for four or more of the eight sessions, the

classroom was coded as ‘low adherence.’ Otherwise, a code of ‘high adherence’ was given on

this variable. A similar dichotomous variable was created for teacher engagement. If the

facilitator rated the teacher as being engaged for five or more of the eight sessions, the

classroom was coded as having an ‘engaged teacher.’ If the facilitator rated the teacher as

being engaged for four or less of the eight session, a code of ‘non-engaged teacher was given.’

The final measure, using the teacher/co-leader evaluations (Appendix G) involved feedback on

how satisfied teachers were with the Making A Difference Program (Appendix H; Process

Evaluation Matrix) and their rating of the severity of bullying incidences within the school before

and after the program was delivered. T tests were used to analyze the teachers’ perceptions of

frequency and severity of bullying incidents at the school.

Limitations:

When conducting research with partner organizations, there are always limitations. Our study

made use of children’s self-report data collected via questionnaires, which although effective for

assessing the direct opinions and feelings of children, are also limited by children’s reading level

30

and limited attention. The questionnaires we used included closed-answer questions that did not

allow for qualification or elaboration of answers. Also, findings based solely on self-report data

must be interpreted with caution because of shared method variance. For example, students

were the only ones reporting their own behaviour and it is possible that they underreported their

involvement in bullying. Although not included in our analyses, we asked teachers to fill in a

brief questionnaire at the end of the intervention to provide us with their perceptions of the

program and its effectiveness with their students. This teacher questionnaire focused on general

classroom and peer functioning rather than reports about individual children. Therefore, the data

collected for each individual child was in isolation and without validation from other sources.

Another limitation of this study is that we were restricted from collecting data at more than two

time points: at the beginning of session one and the end of session 8. This means that post-

intervention data was collected immediately after the final lesson and that there was no time for

this lesson to be translated into meaningful attitudes and behaviours in the lives of the

participants. Furthermore, we were unable to take a baseline measurement of behaviour prior to

the intervention. Taking baseline data four-to-six weeks before the intervention allows

researchers to better determine if confounding factors may have influenced any changes in

rates of bullying, witnessing, and victimization. In future research, it will be important for us to

attempt to take data at baseline and a four-to-six week follow-up to better understand the impact

of the intervention.

Finally, this study was limited by restrictions of the two school boards regarding matching pre-

and post-intervention data to individual students. If we had been able to use an identifier on

surveys (such as a first name or number assignment), we could have used repeated-measures

ANOVAs to analyze the data. Because data was not matched, we used an independent-

measures design, which is less sensitive to detecting a significant statistical difference when

31

one truly exists. In future studies, it will be important build on the trust we gained with the school

boards and individual schools to allow us to use an identifier in the surveys and successfully

match pre- and post- data.

Results

Knowledge of Bullying Reporting Procedures

Our first evaluation question regards the investigation of participants’ knowledge of effective

bullying reporting procedures as reflected in survey questions 4, 9, and 10. With regard to

question 4, “If you were bullied or saw someone being bullied, would you know how to report it

at your school?” we ran a Chi-square test for goodness-of-fit to investigate if the number of

participants who answered “yes” differed from pre-test to post-test. 89.2% of participants who

answered this question before intervention and 94.6% who answered it after intervention said

that they knew how to report bullying at their school. A Pearson Chi square with Yates

correction indicated that this is a significant difference (Yates = 17.424, p < .001) indicating that

more participants knew how to report bullying in their school after the intervention than before.

This change is associated with a small effect size (Phi = .10, p < .001).

With regard to question 9, “There is an adult at this school who I would feel comfortable

speaking to if I am bullied,” participants responded using a 5-point Likkert scale with the

following indicators: 1=strongly disagree, 2=agree, 3-not sure, 4=disagree, 5=strongly disagree.

A one-way ANOVA was conducted to explore the impact of the intervention on levels of

confidence in the above statement. These data violated the homogeneity of variance

assumption for one-way ANOVA, so the Welch statistic was substituted (Levene = 44.53, p <

.001). There was a statistically significant difference at the p = .05 level in scores between the

pre- and post-intervention conditions [Mpre = 2.64, Mpost = 2.94; F(1,1922) = 21.64, p < .001).

32

This indicates that students’ confidence in their ability to identify a trustworthy adult with whom

they could comfortably discuss being victimized increased after the intervention. Although the

change from pre- to post-test was significant, an eta squared analysis revealed a very small

effect size (.01).

With regard to question 10, “There is an adult at this school who I would feel comfortable

speaking to if I see someone else being bullied,” participants responded using the above Likkert

scale. A one-way ANOVA was conducted and no assumptions of the test were violated because

the variances between the two groups were homogenous (Levene = 5.62, p = .018). There was

a statistically significant difference at the p = .05 level in scores between the pre- and post-

intervention conditions [F(1, 1911.229) = 8.14, p = .004). This indicates that students’

confidence in their ability to identify a trustworthy adult with whom they could comfortably

discuss witnessing bulling increased after the intervention. Although the change from pre- to

post-test was significant, an eta squared analysis revealed a very small effect size (.004).

Subjective Reports of Bullying

Our second evaluation question regards the subjective reports of participants’ victimization,

witnessing, and perpetration of bullying and are reflected in survey questions 5, 6, and 7. For

these survey questions, participants are asked to indicate how often they encounter specific

types of bullying using an ordinal scale of “never,” “sometimes,” “often,” or “very often.” To

obtain a global indication of how often they are bullied, witness bullying, and commit bullying

offenses respectively, we created a numerical ranking using a Likkert scale and found the mean

for each report.

33

In each of these questions, students responded to the following items according to the following

scale: never =0, sometimes=1, often=2, very often=3.

a. Threats from other students

b. Insults, name calling, rude gestures

c. Physical bullying by one person

d. Physical bullying by a group

e. Stealing or damaging your stuff

f. Being excluded or left out

g. Being made to look silly in front of others

h. Someone using the Internet or a cell phone...

Their responses for items a through h were averaged (mean) for questions 5, 6, and 7.

With regards to frequency of victimization (question 5), the survey question was, “In the last 4

weeks at school, how often have you been bullied in these ways?” A two-way ANOVA was

calculated to test for a main effect between time one and time two (before and after the

intervention) and any interaction effects between gender and grade. A Bonferroni correction was

applied to correct for type 1 error with multiple analyses (α = .05/2 = .025). However, when the

data failed to meet the criteria of homogeneity of variance (Levene (3, 1860) = 20.74, p < .001),

a more stringent alpha level was set at .01. A Bonferroni correction was then applied to the new

alpha level (α = .01/2 = .005). There was no interaction effect between sex and time [F(1,1860)

= 3.55, p = .06] and no interaction effect between grade and time [F(1, 1860) = 1.12, p = .289].

Furthermore, there was no main effect for time (the intervention, we might refer to this as the

difference between time1 and time2) either [F(1, 1860) = 1.49, p = .223]. This indicates that

participants levels of perceived victimization did not change after the intervention.

34

With regards to frequency of witnessing bullying (question 6), the survey question was, “In the

last 4 weeks at school, how often have you seen bullying (witnessed it) in these ways?” Two-

way between groups ANOVAs were conducted to explore the impact of gender, grade and the

intervention on self-reported levels of witnessing bullying and a Bonferroni correction was

applied to reduce type 1 error with multiple analyses (α = .01/2 = .005). The assumption of

homogeneity of variances was met. There was no interaction effect between sex and time

[F(1,1860) = 1.51, p = .219], nor was there an interaction effect between grade and time

[F(1,1918) = .08, p = .778].There was no main effect for time [F(1, 1860) = 1.31, p = .252). This

indicates that participants’ subjective perceptions of witnessing bullying did not change after the

intervention.

With regards to frequency of bullying others (question 7), the survey question was, “In the last 4

weeks at school, how often have you bullied others in these ways?” Two-way between groups

ANOVAs were conducted to explore the impact of gender, grade and the intervention on self-

reported levels of bullying aggression and a Bonferroni correction was applied to reduce type 1

error with multiple analyses (α = .01/2 = .005). The assumption of homogeneity of variances

was not met so a more stringent alpha level was set at .01. A Bonferroni correction was then

applied to the new alpha level (α = .01/2 = .005). There was no interaction effect for time and

gender [F(1, 1860) = 1.22, p = .269] and no interaction effect for time and grade [F(1, 1860) =

.31, p = .579]. There was no main effect for the time [F(1,1860) = 6.09, p = .014]. This indicates

that participants’ subjective perceptions of their own levels of bullying behaviour did not change

after the intervention.

Bullying and School Absenteeism

Our third evaluation question was in regard to how often children missed school or wanted to

miss school as a result of bullying and is reflected in survey question 8, “In the last 4 weeks,

have you ever stayed away, or wanted to stay away from school because you were being

35

bullied?” With regards to whether or not the program was effective in reducing reports of

absenteeism due to bullying, we analyzed the responses to this question (“yes” or “no”) using a

Chi Square for Independence. 618 participants answered this question before the intervention

and 601 answered it after the intervention for a total of 1219 responses. A Yates correction

indicated a significant result (Yates = 9.59, p = .002) with a small effect size (Phi = -.09, p =

.002). This result indicates that fewer participants stayed away from school or wanted to stay

away from school due to bullying after the intervention.

Positive/Assertive Responses to Bullying

Our final research question was in regard to participants’ use of prosocial responses when

being victimized or when witnessing bullying and are reflected in questions 12 and 14 of the

survey. Participants indicated which reactions they had from a list of 13 choices including 9

positive/assertive and 4 negative/antisocial responses. (The items “I have not been bullied” and

“other” were not included in these analyses.) For questions 12 and 14, a z test for proportions

was conducted for each of the 13 items, and a Bonferroni correction was applied to reduce type

1 error for multiple analyses (α = .05/13 = .0038). Table 1 displays the results for question 12, in

which participants were asked, “Think of the last time you were bullied. What did you do?

(Check ALL that are true for you.)” There were no significant differences in how many students

indicated any particular response between pre- and post-test. Table 2 displays the results for

question 14, in which participants were asked, “Think of the last time you saw bullying happen.

What did you do? (Check ALL that are true for you.)” There were no significant differences in

how many students indicated any particular response between pre-and post-test. However, the

final item, “I got back at the person who was doing it,” would have been significant at the p = .05

level (without the Bonferroni correction). While it is debated in the literature if a Bonferroni

correction should be uniformly applied to pilot data in the social sciences, we have chosen to

36

use the more stringent method of analysis in our research. We report this result merely as a

curiosity, and with the idea in mind that this item may be of interest in further study.

Data were analyzed further to investigate the frequency of participant who used at least one

positive/assertive response and at least one negative/anti-social response. These data are

presented in Tables 3 and 4. Z tests for proportions revealed no significant differences among

assertive and non-assertive responses between pre- and post-test conditions.

Table 1. Responses to Victimization Response Before (n=952) After (n=900) Significant? *I ignored it 304 (31.9%) 276 (30.7%) No I told my parents or guardian about it

392 (41.2%) 353 (39.2%) No

I told my teacher about it

264 (27.3%) 250 (27.8%) No

I told my principal or vice principal about it

150 (15.8%) 145 (16.1%) No

I told an adult from school about it

180 (18.9%) 178 (19.8%) No

I told an adult outside of school about it

175 (18.4%) 168 (18.7%) No

I told another student about it

306 (32.1%) 330 (36.7%) No

I called a helpline 10 (1.1%) 8 (0.9%) No *I fought back 155 (16.3%) 136 (15.1%) No I got someone to help stop it

219 (22.6%) 219 (24.3%) No

*I made a joke out of it 99 (10.4%) 114 (12.7%) No I stood up to the person who was doing it

306 (31.6%) 281 (31.2%) No

*I got back at the person who was doing it

99 (10.4%) 74 (8.1%) No

*indicates a negative/antisocial response

37

Table 2. Responses to Witnessing Bullying Response Before (n=932) After (n=898) Significant? *I ignored it 129 (13.8%) 120 (13.4%) No I told my parents or guardian about it

187 (20.1%) 168 (18.7%) No

I told my teacher about it

269 (28.9%) 235 (26.2%) No

I told my principal or vice principal about it

124 (13.3%) 114 (12.7%) No

I told an adult from school about it

197 (21.2%) 183 (20.4%) No

I told an adult outside of school about it

113 (12.1%) 88 (9.8%) No

I told another student about it

254 (27.3%) 257 (28.6%) No

I called a helpline 10 (1.1%) 12 (1.3%) No *I fought back 112 (12.0%) 91 (10.1%) No I got someone to help stop it

229 (24.6%) 208 (23.2%) No

*I made a joke out of it 31 (3.3%) 31(3.3%) No I stood up to the person who was doing it

260 (26.9%) 260 (26.9%) No

*I got back at the person who was doing it

73 (7.8%) 44 (4.9%) No** (z = 2.563826, p = .0052)

*indicates a negative/antisocial response **result would be significant if the Bonferroni correction had not been applied

Table 3. Frequencies of Assertive and Non-Assertive Responses to Bullying

Type(s) of responses Pre (n=987) Post (n=925) Assertive Males = 290

Females =283 Total = 577

266 273 539

Grade 4 = 21 Grade 5 = 360 Grade 6 = 194 Total = 577

17 327 195 539

Non-assertive Males = 252 Females = 182 Total = 437

220 181 401

38


9 243 149 401

Table 4. Frequencies of Assertive and Non-assertive Responses to Witnessing Bullying

Type(s) of responses Pre (n=965) Post (n=923) Assertive Males = 294

Females =294 Total = 592

249 278 527


12 319 196 527

Non-assertive Males = 151 Females = 107 Total = 259

140 79 219


3 133 83 219

Knowledge Exchange

Internal Linkages

Knowledge exchange has occurred internally via staff meetings in which we have shared the

original work plan, research process and current findings. Through this experience our agency

has continued to develop a culture of learning and, most importantly, an appreciation for the

value of research.

External Linkages

This experience has led us to make some unique contacts which include:

1. Michael Jonny , Manager, Knowledge Mobilization, York University,

2. Representatives from the La Marsh Research Centre,

3. Promoting Relationships and Eliminating Violence Network (PREVNet) (Deb Pepler &

Joanne Cummings),

39

4. Fred Faber (The York Public Board Coordinator of Bully Prevention).

5. Robert Hache, Vice President Research and Innovation at York University

6. York Region District School Board (YRDSB)

7. York Catholic District School Board (YCDSB).

8. Care and Treatment Program Facilitator for YRDSB,

9. Behaviour Resource Specialist for YCDSB

10. Behavioural Resource Consultants for both boards

11. Community Partners in Schools (COMPASS)

12. Blue Hills Child and Family Centre,

13. Kinark Child and Family Services,

14. The Children’s Aid Society of York Region

15. The United Way

16. The Ministry of Children and Youth Services

These linkages to stakeholders within the field of bullying have enabled our Community

Outreach Team to enrich our network which has ultimately benefitted our program delivery. Our

bullying prevention program had previously been offered in isolation of other service providers.

By defining our working relationships with some of the key stakeholders we have been better

able to coordinate the delivery of our bullying prevention workshops within the school boards

and even York Region. Our plan is to publish our findings within our agency newsletter that is

circulated to our partners and the general public within the broader community. In collaboration

with the bully prevention coalition of York Region we will present our findings and provide an

opportunity for knowledge exchange. Using this experience we plan to continue our capacity

building regarding the delivery and evaluation of the Making A Difference program within both

York Region school boards. It is our continuing goal to build capacity within the York Region

schools so all students feel safe and have the appropriate resources to take a stand against

bullying.

40

Conclusion & Recommendations/Next Steps

The Making a Difference intervention aimed to reduce subjective reports of bullying and

victimization, while also teaching children knowledge of reporting procedures and assertive

responses to bullying when in victim or bystander roles. The intervention appears to have been

most successful in its goal of teaching knowledge of reporting procedures. This is likely due to

the fact that acquiring knowledge and skills is often a prerequisite to changing behaviour.

There was a statistically significant positive change from pre- to post-test on all questions that

measured knowledge of reporting procedures. Questions 4, 9, and 10 are phrased in such ways

that they not only measure children’s knowledge (of reporting procedures and of trustworthy

adults at their school) but also their confidence in their abilities and sufficient levels of comfort in

reaching out for help. Although we did not directly ask which adults students identified as

trustworthy, we hypothesize that many children would identify their classroom teacher if this

adult was present and mindfully involved while the intervention was delivered. Seeing their

teacher model appropriate responses and actively engage in the subject material is likely a

helpful in aiding children to identify them as a trustworthy adult to whom they would speak if

they dealing with bullying. With regard to question 4, “If you were bullied or saw someone being

bullied, would you know how to report it at your school?” the effect size was small. This may be

evidence of a ceiling effect considering that 89.2% of participants at pre-test answered “yes” to

this item, leaving little room for a larger increase. Interestingly, effects sizes were also small for

both questions 9 and 10, which specifically tap into children’s ability to identify a specific adult at

their school with whom they would feel safe reporting bullying. Because the means in both pre-

and post-test responses for these questions were near the middle of the ranges of scores, we

cannot interpret this small effect as being due to a ceiling effect. It may be that children are

aware of reporting procedures in the theoretical sense, but when bullying involves them

41

personally, they may be reluctant to identify someone with whom they are comfortable trusting.

Future evaluators may wish to tweak the wording of these questions to further determine if the

construct being measured is children’s knowledge or comfort with reporting procedures.

This evaluation did not find significant results with regards to subjective reports of victimization,

witnessing, or perpetration of bullying. While we continue to believe in the utility of the training

materials and the delivery of our facilitators, we are not entirely surprised by these findings. It is

rare that bullying interventions find significant changes in behaviour during such a short time line

(our study comprised eight weeks only). In the future, we would attempt to collect data at two

additional time points: four weeks prior to the intervention (baseline), and four weeks after the

intervention (follow-up). Because behaviours are difficult to change and often require a change

in thinking or beliefs, it makes sense that behavioural differences may not be evident in the data

immediately after the intervention (quite literally at the end of the final session).

Follow-up data can provide information on subjective experiences of bullying after the lessons of

the intervention have been reinforced by the classroom teacher and additional whole-school

projects (which are planned but not executed during the final session). Another consideration

with children’s self-reports of bullying and victimization is that few children are actually involved

in high-frequency bullying. Often the data is positively skewed, with most children reporting low

frequencies of involvement in bullying and a few children reporting higher frequencies of chronic

bullying problems. Often researchers will correct for this positive skew by using a Log10

transformation on the data (which did not make a statistical difference in the case of our data).

Ideally, researchers match pre- and post-test data by participant using a code number or other

identifier so that more sensitive statistical analyses may be performed (e.g. with matched data,

repeated measures ANOVA may be used in place of independent samples ANOVA).

Unfortunately, we were unable to retain permission of the involved school boards to use any

42

form of identifier that could be used to match pre- and post-surveys. This inability to match data

over time points may have resulted in our inability to detect a difference in children’s behaviour

that truly exists. In future studies, we will make a strong case to the school boards about the

importance of matching data and find a creative solution to satisfy the boards’ concern for

anonymity of participants. It is important to note that although the results were not significant,

we did find a positive trend in the data for victimization, witnessing, and perpetration of bullying.

The use of more sensitive analyses in future studies will help us to identify if these trends are

significant.

Positive trends were also found in questions that tapped into children’s responses to being

bullied and witnessing bullying. Specifically, there was an increase in self-reported assertive

responses and a decrease in self-reported negative responses. Again, these changes were not

statistically significant; however, this may be due in part to the necessity of performing

Bonferroni corrections due to multiple analyses. In fact, the final item, “I got back at the person

who as doing it,” (which is a negative response) would have been found to have decreased from

pre- to post-test with regards to responding to witnessing bullying if the Bonferroni correction

had not been applied. Furthermore, both survey questions 12 and 14 contain items that children

told us were repetitive and confusing. For example, children have 15 choices for a response to

the question, “Think of the last time you were bullied. What did you do?” and may indicate as

many responses as they wish. Items such as “I told the Principal or Vice Principal” and “I told an

adult at school” may be seen as confusing, perhaps encouraging children to indicate all items or

to select items in a random manner. Future research requires the “tightening up” of this list of

response items. Also, items such as “I made a joke of it” and “I ignored it” were confusing to

some children. Many participants believed that these negative responses were actually

prosocial because they were not “fighting back” but still respected their level of comfort by not

43

confronting someone of whom they are afraid. Clarification of these items may yield different

results in the future.

Feedback from the teachers indicated a high incidence of satisfaction with the Making A

Difference program (Appendix I: Teacher Satisfaction Data). 94.6% of teachers reported that

they would recommend this program to others and 88.9% indicated that the sessions were long

enough while 82.9% reported satisfaction with the number of sessions being appropriate for the

intended purpose.

Feedback from the Community Outreach facilitators must also be noted. Reviewing and

analyizing four facilitators’ fidelity checklists indicated high adherence (91% of the classrooms

received the intervention as intended). However, the most often cited problem on the fidelity

checklist involved disengaged teachers, only 62% of the classrooms had a engaged teacher.

(Appendix J: Teacher Engagement -Data). Since so much of the program is dependent on the

teacher reinforcing the program’s messages and being a trusted adult to whom to report

bullying, the importance of an engaged teacher is paramount to the success of the program.

This is an important consideration as we move forward and encourage teachers to be invested,

engaged, and supportive.

Insights about our experience in conducting this evaluation

Insights learned from conducting this evaluation have been plentiful. The process has

reinforced the value of sustaining an up-to-date working knowledge of the developments of best

practices in our field. Completing our literature review has highlighted the importance and

necessity in keeping the Making A Difference program current and up to date with evidence-

based best practices.

44

This evaluation increased the communication within the Community Outreach Team as the

team members created the fidelity checklists. The process was at times challenging due to

individual teaching styles; however, it was a necessary process to streamline the program.

This project has led us to question what constitutes an appropriate evaluation tool. Due to the

ethics approval from the boards of education the research was limited by the survey tool. While

considered ethical and approved for the Community Outreach team to deliver to students, the

survey’s ability to truly assess the program’s effectiveness may have been limited. This has, in

return, created dialogue within the Community Outreach team. Staff members have questioned

whether the program is less effective than hoped for or whether the tool itself didn’t specifically

measure our target goals. As a newly formed “research team” we have made a decision to

adopt a new research tool to continue evaluating effectiveness of our program.

A significant insight that must be shared is the overwhelming commitment required when taking

on an evaluation project of this magnitude. The York Centre has limited research resources and

without the help of Dilys Haner and support of Marie-Josee Emard the project would not have

been possible. Dilys offered the team her time, research expertise and data analysis software.

Her support was needed and highly valued. Should the York Centre pursue future research

projects this is an insight that must remembered. Staff members aligned with this project were

at times pulled in too many directions (i.e., the requirements of their daily professional roles and

then the additional work and time needed to support the research process). It would be highly

beneficial for The York Centre to apply for funding that would allow the agency to have a

permanent on-site research position whereby this worker could fully commit to the project and

not be pulled in a multitude of directions. Such a position would improve access to literature

data bases and research tools as well as everyday communication, support and training for

building capacity in evaluation and research.

45

References

Adler, P. A. (1996). Preadolescent clique stratification and the hierarchy of identity. Sociological

Inquiry, 66, 111-142.

Alsaker, F. (2004). Bermese programme against victimisation in kindergarten and elementary

schools. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in Schools: How successful

can interventions be? (pp. 289-306). Cambridge: Cambridge University Press.

Arora, T. (1991). The use of victim support groups. In P. K. Smith & D. A. Thompson (Eds.),

Practical Approaches to Bullying. London: David Fulton.

Bauman, S. & Del Rio, A. (2006). Preservice teachers' responses to bullying scenarios:

Comparing physical, verbal, and relational bullying. Journal of Educational Psychology,

98(1), 219-231.

Boulton, M. J., & Smith, P. K. (1994). Bully/victim problems in middle-school children: Stability,

self-perceived competence, peer perceptions, and peer acceptance. British Journal of

Developmental Psychology, 12, 315-329.

Card, N. A., Stucky, B. D., Sawalani, G. M., & Little, T. D. (2008). Direct and indirect aggression

during childhood and adolescence: A meta-analytic review of gender differences,

intercorrelations, and relations to maladjustment. Child Development, 79, 1185-1229.

Craig, Wendy M.; Pepler, Debra J.. Identifying and targeting risk for involvement in bullying and

victimization. The Canadian Journal of Psychiatry / La Revue canadienne de psychiatrie

48. 9 (Oct 2003): 577-582.

Craig, W., & Pepler, D. (1995). Peer processes in bullying and victimization: An observational

study. Exceptionality Education Canada, 5, 81-95.

Craig, W. & Pepler, D. J. (2007). Understanding bullying: From research to practice. Canadian

Psychology, 48, 86-93.

46

Cross, D., Shaw, T., Pearce, N., Ecreg, E., Waters, S., Pintabona, Y., et al. (2008). School-

based interventions research to reduce bullying in Australia 1999-2007: What works, what

doesn’t, and what’s promising? In D. Pepler & W. Craig (Eds.), Understanding and

Addressing Bullying: An International Perspective (pp. 289-310). Bloomington, IN:

Authorhouse.

DeSouza, E. R., & Ribeiro, J. (2005). Bullying and sexual harassment among Brazilian high

school students. Journal of Interpersonal Violence, 20, 1018-1038.

Gini, G., Albiero, P., Benelli, B., & Altoe, G. (2007). Does empathy predict adolescents bullying

and defending behavior? Aggressive Behavior, 33, 467-476.

Hawkins, D. L., Pepler, D. J., & Craig, W. M. (2001). Naturalistic observations of peer

interventions in bullying. Social Development, 10, 512-527.

Jolliffe, D., & Farrington, D. P. (2006). Examining the relationship between low empathy and

bullying. Aggressive Behavior, 32, 540-550.

Lerner, R. M., & Galambos, N. L. (1998). Adolescent development: Challenges and

opportunities for research, programs, and policies. Annual Review of Psychology, 49, 413-

446.

O’Connell, P., Pepler, D., & Craig, W. (1999). Peer involvement in bullying: Insights and

challenges for intervention. Journal of Adolescence, 22, 437-452.

Olweus, D. (1993). Bullying at school: What we know and what we can do. Oxford: Blackwell.

Pellegrini, A., & Bartini, M. (2000). A Longitudinal Study of Bullying, Victimization, and Peer

Affiliation During the Transition from Primary School to Middle School. American Educational

Research Journal, 37(3), 699-725.

Pellegrini, A.D., Bartinia, M. & Brooks, F. (1999). School Bullies, Victims, and Aggressive

Victims: Factors Relating to Group Affiliation and Victimization. Journal of Educational


47

Pepler, Debra J. (2006). Bullying interventions: A binocular perspective. Journal of the Canadian

Academy of Child and Adolescent Psychiatry / Journal De l'Académie Canadienne De

Psychiatrie De l'Enfant Et De l'Adolescent, 15(1), 16-16-20.

Pepler, D. J. (2006). Bullying interventions: A binocular perspective. Journal of the Canadian

Academy of Child and Adolescent Psychiatry, 15, 16-20.

Pepler, D. J., Craig, W. M., Connolly, J. A., Yuile, A., McMaster, L., & Jiang, D. (2006). A

developmental perspective on bullying. Aggressive Behaviour, 32, 376-384.

Pepler, D., Craig, W., Cummings, J., Yamada, S., Cappadocia, M. C., Vaughan, A., & Ma, J.

(2008). Foundations for Bullying Prevention: Evidence Based Fact Sheets and Implications.

Report prepared for the United States Department of Health and Human Service Substance

Abuse and Mental Health Services Administration.

Perren, S., & Hornung, R. (2005). Bullying and delinquency in adolescence. Swiss Journal of


PREVNet (2007). The Safe Schools Survey and The Bullying Quiz, (unpublished).

Salmivalli, C., Kaukiainen, A., & Voeten, M. (2005). Anti-bullying intervention: Implementation

and outcome. British Journal of Educational Psychology, 75, 465-487.

Salmivalli, C., Lagerspetz, K., Bjorkqvist, K., Osterman, K., & Kaukiainen, A. (1996). Bullying as

a group process: Participant roles and their relations to social status within the group.

Aggressive Behavior, 22, 1-15.

Scheithauer, H., Hayer, T., Petermann, F., & Jugert, G. (2006). Physical, verbal, and relational

forms of bullying among German students: Age trends, gender differences, and correlates.

Aggressive Behavior, 32(3), 261-275.

Sharp., S. 1996. Self-esteem, Response Style and Victimization: Possible Ways of Preventing

Victimization through parenting and School Based Training Programmes. School

Psychology International, 17, 347-357.

48

Sharp, S., & Cowie, H. (1994). Empowering Students to take Positive Action against Bullying. In

P. K. Smith & S. Sharp (Eds.), School Bullying: Insights and Perspectives. London:

Routeledge.

Schwartz, D., Dodge, K.A., & Coie, J.D. (1993). The Emergence of Chronic Peer Victimization

in Boys’ Play Groups. Child Development, 64, 1755-1772.

Smith, D. (2008). Promoting a positive school climate: Restorative practices for the classroom.

In D. Pepler & W. Craig (Eds.), Understanding and Addressing Bullying: An International

Perspective (pp.132-143). Bloomington, IN: Authorhouse.

Smith, P. K., & Thompson, D. (1991). Practical Approaches to Bullying. Great Britain: David

Foulton.

Stoneman, D. (2002). The role of youth programming in the development of civic engagement.

Applied Developmental Science, 6, 221-226.

Szalavitz, M. (2010). How not to raise a bully: The early roots of empathy. Time, April 17th,

2010. Retrieved October 1st, 2011 from

http://www.time.com/time/health/article/0,8599,1982190,00.html

Warden, D. & Mackinnon, S. (2003). Prosocial children, bullies, and victims: An investigation of

their sociometric status, empathy and social problem-solving strategies. British Journal of

Developmental Psychology, 21, 367-385.

Woods, S., & White, E. (2005). The association between bullying behaviour, arousal levels and

behaviour problems. Journal of Adolescence, 28, 381-395.

Zeldin, S. (2004). Preventing youth violence through the promotion of community engagament

and membership. Journal of Community Psychology, 32, 623-641.

http://www.time.com/time/health/article/0,8599,1982190,00.html

Appendix A – Power Point Presentation

50

Making a DifferenceMaking a Difference

Week 1Week 1

Please see attached email : Making A Difference- Power Point- Attachments-Weeks 1-8

Appendix B – Fidelity Checklists

51

MAKING A DIFFERENCE FIDELITY CHECK

WEEK 1 Date _____________________________ Class _____________________________ School _________________ Time Frame _____________________________ Number of Students: _________ Guidelines for Group Facilitators: Please ensure that the entire activity (listed below) has been completed prior to placing a checkmark in the “yes” column.

Did I: Yes NO

1. Provide introduction (myself, York Centre, Making A Difference) ___ ___ 2. Deliver Pre-survey ___ ___

4. Read the entire survey aloud to the students ___ ___ 5. Follow the power point ___ ___ 6. Have students complete the Think Pair Share

(what are some positive qualities of a good friend) ___ ___ 7. Have the students complete the drawing activity

(picture of a bullying situation or evolution game) ___ ___ 8. Review the 4 part definition of bullying ___ ___ 9. Review the forms of power ___ ___ 10. Discuss the 4 situations whereby bullying can be repeated

(1:1; 1:group, group:group; group:1) ___ ___ 9. Show the K.B. video (K.B.’s First Day) to review misuses of power ___ ___ 10. Have an engaged teacher during the lesson? ___ ___ 11. Ensure the Making A Difference Class envelope is marked with Class, school, grade, date started and date completed information ___ ___

Comments (were there any distractions during this session - e.g. announcements):


52



Did I: Yes NO

1. Show video – Real Friends -How did Melanie make a difference? ___ ___ 2. Ask: Has anyone observed someone in need of help and

did someone (without using names) step in to make a difference? ___ ___ 3. Follow the power point – Loveable and Capable ___ ___ 4. Have the students complete the Poster Activity? ___ ___ 5. Have some of the students share their posters? ___ ___ 6. Complete Activity 2 – Are we all the same? ___ ___ 7. Rip the facilitator’s poster (point out differences & rip) ___ ___ 8. Have an engaged teacher during the lesson? ___ ___

Comments:


53


WEEK 3

Date _____________________________ Class _____________________________ School _________________ Time Frame _____________________________ Number of Students: _________

Guidelines for Group Facilitators: Please ensure that the entire activity (listed below) has been completed prior to placing a checkmark in the “yes” column.

Did I: YES NO

1. Ask: Has anyone observed someone in need of help and did

anyone step in to make a difference? ___ ___ 2. Review-4 part definition and remind – we are all loveable & capable ___ ___ 3. Follow the power point – 4. Initiate a Group discussion – review physical & verbal bullying ___ ___ 5. Initiate the Group Activity - Group work – forms of bullying ___ ___ 6. Provide students with Forms of bullying worksheet ___ ___ 7. Provide teacher with information sheet (forms of bullying) ___ ___ 8. Review and discuss the group work activity ___ ___ 9. Show the Video – Don’t Laugh At Me ___ ___ 10. Have an engaged teacher during this lesson ___ ___ Comments:


54



Did I: Yes NO

1. Ask: Has anyone observed someone in need of help? ___ ___ 2. Review ___ ___ 3. Follow the power point ___ ___ 4. Teach direct vs indirect bullying situations-Role plays

(i.e., direct bullying-verbal/physical; indirect bullying-gossip) ___ ___ 5. Teach the tactics used by people who bully– direct instruction ___ ___ 6. Engage students in the role play activity

-Group work – Role plays: various tactics used by people who bully ___ ___ 8. Show the Video – K.B.’s Day (review forms of bullying; direct vs. indirect) ___ ___ 9. Have an engaged teacher during this lesson ___ ___ Comments:


55



Did I: Yes NO

1. Ask: Has anyone observed someone in need of help? ___ ___ 2. Review – previous lessons ___ ___ 3. Follow the power point ___ ___ 4. Demonstrate the 3 communication response styles ___ ___

(Passive- Aggressive –Assertive) 5. Initiate role plays – involving facilitator ___ ___ 6. Support a group process - what’s likely to happen if we respond to bullying

behaviour using a passive – aggressive – assertive response? ___ ___ 7. Initiate a review of the various “assertive” responses ___ ___ 8. Show the video – Milton’s Dream ___ ___ 9. Have an engaged teacher during this lesson ___ ___

Comments:


56



Did I: Yes NO

1. Put the puzzle on blackboard “Safety in Numbers” -theme of the day ___ ___

2. Ask: Has anyone observed someone in need of help? ___ ___ 3. Review – 3 Communication Styles/Responses to bullying behaviour. ___ ___ 4. Follow the power point – ___ ___ 5. Group Activity (Why?)Why do some people support

someone who bullies? Why do some people support the target? Why do some people choose to say nothing and do nothing? ___ ___

6. Engage students in the Activity Who Do You Support? ___ ___ 7. Review the theme of the day? There is safety

in numbers and the more people to take a stand and support someone being targeted the less power the person bullying has and the more likely bullying will decrease. ___ ___

8. Have an engaged teacher during this lesson ___ ___

Comments:


57



Did I: Yes NO

1. Ask: Has anyone observed someone in need of help? ___ ___ 2. Review – Who should we support? ___ ___ 3. Show the Video–Power in Numbers ___ ___ 4. Follow the power point – ___ ___ 5. Initiate Activity #1 – Group process -How much harm? ___ ___ 6. Initiate Activity #2 - Small group process - strategies to

make a difference ___ ___ 8. Reporting – 5W-H ___ ___ 9. Have an engaged teacher during this lesson ___ ___

Comments:


58



Did I: Yes NO

1. Ask: Has anyone observed someone in need of help? ___ ___ 2. Review – Weeks 1-8 – Group process – Oral Quiz ___ ___ 3. Follow the power point ___ ___ 4. Provide students with the anti-bullying pledge ___ ___ 5. Complete the post-survey – read aloud to students ___ ___ 6. Complete teacher/facilitator evaluation ___ ___ 7. Develop a future project plan for class to Make A Difference ___ ___ 8. Have an engaged teacher during this lesson ___ ___

Comments:

Appendix C – Logic Model

59

Program Logic Model-long term outco

Appendix D – Outcome Evaluation Matrix

60

OUTCOME EVALUATION MATRIX


Short-Term Outcomes





STUDENTS 1. Does the Making A Difference Program increase student participants knowledge of effective bullying reporting procedures?

Increased knowledge of effective bullying reporting procedures for participants at school

Questions 4, 9, 10, 13*, 15*, (18 qualitative responses on post survey) Comparison between pre and post surveys


-Students complete the pre survey on the first session (week 1) of the Making A Difference program -Students complete the post survey on the last session (week 8) of the Making A Difference Program



2. Since the commencement of the Making A Difference program have student participants reported a decrease of subjective reports of bullying victimization at their school?

Decreased subjective reports of bullying victimization at school for participants

Questions 5, 6, 7 Comparison between pre and post surveys




3. Has the Making a Difference Program had a positive impact on school attendance due to bullying?

Decreased reports of school absenteeism due to bullying

Question 8 Making a Difference School Safety Survey



4. Have student participants reported increased positive/assertive responses to situations in which

Students will increase their positive/assertive responses to situations in which they are victimized

Questions 12, 14, (18 qualitative responses on post survey)





61

they are victimized or witness bullying since the commencement of the Making A Difference Program?

or witness bullying

TEACHERS Did the teachers observe a decline in bullying behaviours since the commencement of the Making A Difference Program

Teachers will observe a decline in bullying behaviours

Coleader Evaluation (Quantitative) Comparing

quantitative responses: 6 vs 8, 9 vs 10 to determine if 80% teachers report decreased bullying behaviours



-Coleader Evaluation




Evaluation Questions Method to (specify month/year)

Intermediate Outcomes





1. Have student participants increased their assertive strategies and decreased passive/aggressive strategies in their responses to bullying behaviour?

Increase use of assertive strategies and decrease passive/aggressive strategies to respond to bullying behaviour

TBD (To be determined)

TBD TBD TBD TBD

2.Has there been an increase in supportive and helping behaviours

Increase helping and supportive behaviour for those who witness

TBD TBD TBD TBD TBD


62

for those who witness bullying?

bullying

3.Have student participants from the original evaluation continued to report a decrease in bullying victimization at school?

Continued decrease in reports of bullying victimization at school for participants

TBD TBD TBD TBD TBD

4.Has there been a decrease in reported bullying behaviours throughout the school since the commencement of the Making A Difference program involving specific classes?

Decreased subjective reports of bullying victimization for those students in classrooms other than those that received the intervention

TBD TBD TBD TBD TBD

Appendix E(i) – Making a Difference School Safety Pre-Survey – York Region District School Board

63

Welcome! Thank you for completing this survey. This survey asks about you and your experiences at your school over the past 4 weeks. The York Region District School Board is interested in what you and other students have to say and would like your honest answers to the questions that follow. This survey will be completed by students doing the Making A Difference Program. Your feedback will help make schools better. We would like you to answer all of the questions; however, you can skip a question if it makes you uncomfortable or if you don’t know the answer. This is an anonymous survey and no one will know what you wrote. This survey will take about 10 minutes to complete.

1. What grade are you in? (Check one) 5 6

2. Are you a…? (Check one) Boy Girl

BULLYING

There are lots of different ways to bully someone, but a child who bullies wants to hurt the other person (it’s not an accident), and does so repeatedly and unfairly (the person who bullies has some advantage over the other person). Sometimes a group of students will bully another student. Bullying takes many forms: Physical bullying

• when someone hits, shoves, kicks, spits, or beats up on others • when someone damages or steals another student’s property

Verbal bullying

• name-calling, mocking, hurtful teasing • making someone feel sad or scared with words • making people do things they don’t want to do

Social bullying

• leaving someone out of a group • gossiping or spreading rumors about others • setting others up to look foolish • making sure others don’t hang out with the person

Cyber bullying

using computer, email or phone text messages, or pictures to

• hurt someone’s feelings • make someone look bad • scare someone


64

3. Please check the box that best matches how you feel:

Strongly Agree Agree Disagree Strongly

Disagree Not Sure

I feel safe at this school. I feel safe on the way to and from school. I feel safe in my neighbourhood. Yes No 4. If you were bullied or saw someone being bullied, would you know how to report it at your school?

5. In the last 4 weeks at school, how often have you been bullied in these ways?

Very Often Often Sometimes Never a. Threats from other students b. Insults, name calling, rude gestures c. Physical bullying by one person d. Physical bullying by a group or a gang e. Stealing or damaging your stuff f. Being excluded or left out of a group g. Being made to look silly in front of others h. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make you look bad or tell secrets about you

6. In the last 4 weeks at school, how often have you seen bullying (witnessed it) in these ways?

Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying by one person d. Physical bullying involving a group or a gang e. Stealing or damaging others stuff f. Being excluded or left out of a group g. Being made to look silly in front of others h. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make another look bad or tell secrets about them


65

7. In the last 4 weeks at school, how often have you bullied others in these ways? Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying towards one person d. Physical bullying involving a group or a gang e. Stealing or damaging others stuff f. Excluded or left someone out of a group g. Made someone look silly in front of others h. Used the Internet or a cell phone to e-mail or send text messages or pictures in order to make someone look bad or tell secrets about them

Yes No I have never been bullied

8. In the last 4 weeks, have you ever stayed away, or wanted to stay away from school because you were being bullied?

Please check the box that is closest to how you feel:


Disagree Not Sure

9. There is an adult at this school who I would feel comfortable speaking to if I am bullied.

10. There is an adult at this school who I would feel comfortable speaking to if I see someone else being bullied.

11. This school gives students who have been involved in bullying, a chance to talk about it and solve the problem.


66

12. Think of the last time you were bullied. What did you do? (Check ALL that are true for you.) I have not been bullied. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 13. If you did not do anything when you were bullied, what was the reason? (Check ALL that are true for you) I have not been bullied/harassed. I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would

get worse. Other reason (please explain) _______________________


67

14. Think of the last time you saw bullying happen. What did you do? (Check ALL that are true for you.) I have not seen bullying. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 15. If you did not do anything when you saw bullying, what was the reason? (Check ALL that are true for you) I have not seen bullying I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would

get worse. Other reason (please explain) _______________________ THANK YOU for completing our survey!

Appendix E(ii) Making A Difference School Safety Pre-Survey - York Catholic District School Board

68

Welcome! Thank you for completing this survey. This survey asks about you and your experiences at your school, over the past 4 weeks. The York Catholic District School Board is interested in what you and other students have to say and would like your honest answers to the questions that follow. This survey will be completed by students doing the Making A Difference Program. Your feedback will help make schools better. We would like you to answer all of the questions; however, you can skip a question if it makes you uncomfortable or if you don’t know the answer. This is an anonymous survey and no one will know what you wrote. This survey will take about 10 minutes to complete.



BULLYING



Verbal bullying


Social bullying


Cyber bullying




69



Disagree Not Sure



Very Often Often Sometimes Never i. Threats from other students j. Insults, name calling, rude gestures k. Physical bullying by one person l. Physical bullying by a group or a gang m. Stealing or damaging your stuff n. Being excluded or left out of a group o. Being made to look silly in front of others p. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make you look bad or tell secrets about you


Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying by one person d. Physical bullying involving a group or a gang e. Stealing or damaging other’s stuff f. Being excluded or left out of a group g. Being made to look silly in front of others h. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make another look bad or tell secrets about them


70






Disagree Not Sure





71

12. Think of the last time you were bullied. What did you do? (Check ALL that are true for you.) I have not been bullied. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 13. If you did not do anything when you were bullied, what was the reason? (Check ALL that are true for you) I have not been bullied/harassed. I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would

get worse. Other reason (please explain) _______________________


72

14. Think of the last time you saw bullying happen. What did you do? (Check ALL that are true for you.) I have not seen bullying. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 15. If you did not do anything when you saw bullying, what was the reason? (Check ALL that are true for you) I have not seen bullying I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would

get worse. Other reason (please explain) _______________________ THANK YOU for completing our survey!

Appendix F(i) Making A Difference School Safety Post-Survey - York Region District School Board

73

Welcome! Thank you for completing this survey. This survey asks about you and your experiences at your school over the past 4 weeks. The York Region District School Board is interested in what you and other students have to say and would like your honest answers to the questions that follow. This survey will be completed by students doing the Making A Difference Program. Your feedback will help make schools better. We would like you to answer all of the questions; however, you can skip a question if it makes you uncomfortable or if you don’t know the answer. This is an anonymous survey and no one will know what you wrote. This survey will take about 10 minutes to complete.



BULLYING



Verbal bullying


Social bullying


Cyber bullying




74



Disagree Not Sure



Very Often Often Sometimes Never q. Threats from other students r. Insults, name calling, rude gestures s. Physical bullying by one person t. Physical bullying by a group or a gang u. Stealing or damaging your stuff v. Being excluded or left out of a group w. Being made to look silly in front of others x. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make you look bad or tell secrets about you


Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying by one person d. Physical bullying involving a group or a gang e. Stealing or damaging others stuff f. Being excluded or left out of a group g. Being made to look silly in front of others h. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make another look bad or tell secrets about them


75






Disagree Not Sure





76

12. Think of the last time you were bullied. What did you do? (Check ALL that are true for you.) I have not been bullied. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 13. If you did not do anything when you were bullied, what was the reason? (Check ALL that are true for you) I have not been bullied/harassed. I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would get worse. Other reason (please explain) _______________________


77

14. Think of the last time you saw bullying happen. What did you do? (Check ALL that are true for you.) I have not seen bullying. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 15. If you did not do anything when you saw bullying, what was the reason? (Check ALL that are true for you) I have not seen bullying I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would get worse. Other reason (please explain) _______________________

A lot A Little Not Very Much

Not At All

16. Did the Making A Difference Programme help?

Please comment why the Making a Difference programme helped you a lot, a little, not very much or not at all.


78

17. What did you learn from the Making A Difference programme: 18. How would you deal with bullying in the future? _________________________________________________________________ _____

THANK YOU for completing our survey!

Appendix F(ii) Making a Difference School Safety Post-Survey - York Catholic District School Board

79

Welcome! Thank you for completing this survey. This survey asks about you and your experiences at your school, over the past 4 weeks. The York Catholic District School Board is interested in what you and other students have to say and would like your honest answers to the questions that follow. This survey will be completed by students doing the Making A Difference Program. Your feedback will help make schools better. We would like you to answer all of the questions; however, you can skip a question if it makes you uncomfortable or if you don’t know the answer. This is an anonymous survey and no one will know what you wrote. This survey will take about 10 minutes to complete.



BULLYING



Verbal bullying


Social bullying


Cyber bullying




80



Disagree Not Sure



Very Often Often Sometimes Never y. Threats from other students z. Insults, name calling, rude gestures aa. Physical bullying by one person bb. Physical bullying by a group or a gang cc. Stealing or damaging your stuff dd. Being excluded or left out of a group ee. Being made to look silly in front of others ff. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make you look bad or tell secrets about you


Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying by one person d. Physical bullying by a group or a gang e. Stealing or damaging of others stuff f. Being excluded or left out of a group g. Being made to look silly in front of others h. Someone using the Internet or a cell phone to e-mail or send text messages or pictures in order to make another look bad or tell secrets about them


81

7. In the last 4 weeks at school, how often have you bullied others in these ways? Very Often Often Sometimes Never a. Threats to other students b. Insults, name calling, rude gestures c. Physical bullying towards one person d. Physical bullying involving a group or a gang e. Stealing or damaging of others stuff f. Excluded or left someone out of a group g. Made someone look silly in front of others h. Used the Internet or a cell phone to e-mail or send text messages or pictures in order to make another look bad or tell secrets about them





Disagree Not Sure





82

12. Think of the last time you were bullied. What did you do? (Check ALL that are true for you.) I have not been bullied. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 13. If you did not do anything when you were bullied, what was the reason? (Check ALL that are true for you) I have not been bullied/harassed. I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would get worse. Other reason (please explain) _______________________


83

14. Think of the last time you saw bullying happen. What did you do? (Check ALL that are true for you.) I have not seen bullying. I ignored it. I told my parent(s) or guardian(s) about it. I told my teacher about it. I told my principal or vice-principal about it. I told an adult at school about it. I told an adult outside of school about it. I told another student about it. I called a helpline. I fought back. I got someone to help stop it. I made a joke of it. I stood up to the person who was doing it. I got back at the person who was doing it. Other (please explain) _______________________________________ 15. If you did not do anything when you saw bullying, what was the reason? (Check ALL that are true for you) I have not seen bullying I was afraid I didn’t know what to do or who to talk to. I thought if I told someone, they wouldn’t do anything about it. I don’t like to tell on other people. I didn’t think the bullying/harassment was so bad. I didn’t want to get in trouble for telling. I didn’t think it would make a difference. I thought I would get bullied/harassed more or that the bullying/harassment would get worse. Other reason (please explain) _______________________

A lot A Little Not Very Much

Not At All

16. Did the Making A Difference Programme help?

Please comment why the Making a Difference programme helped you a lot, a little, not very much or not at all.


84

17. What did you learn from the Making A Difference programme: 18. How would you deal with bullying in the future? _________________________________________________________________ _____

THANK YOU for completing our survey!

Appendix G – Community Group, Co-leader Evaluation – Making a Difference

85

The York Centre for Children, Youth & Families

COMMUNITY GROUP

CO-LEADER EVALUATION MAKING A DIFFERENCE

Name: Date: Group

Offered:

Position of Evaluator:

School/Organization:

1. How would you rate the quality of the Making A Difference sessions? Poor Excellent 1 2 3 4 5 6 7 8 9 10 2. To what extent did the sessions meet your expectations? Poor Excellent 1 2 3 4 5 6 7 8 9 10 3. How satisfied are you with the amount of information your students received? Poor Excellent 1 2 3 4 5 6 7 8 9 10 4. How satisfied are you with the presenter and their presentation approach/style? Poor Excellent 1 2 3 4 5 6 7 8 9 10 5. To what extent did the sessions meet your needs? Poor Excellent 1 2 3 4 5 6 7 8 9 10 6. Please rate the frequency of bullying incidents at your school/community group in the 4 weeks prior to the Making A Difference Program. (1 represents very few bullying incidents and 10 represents many bullying incidents.) Very Few Many bullying incidents 1 2 3 4 5 6 7 8 9 10 7. If you have observed a reduction in bullying incidents since the commencement of the program, please include any changes in attitude and or /behaviour of your students (please provide examples.)


86

____________________________________________________________________________

____________________________________________________________________________

___________________

8. Please rate the frequency of bullying incidents at your school/community group in the since the Making A Difference Program. (1 represents very few bullying incidents and 10 represents many bullying incidents.) Very Few Many bullying incidents 1 2 3 4 5 6 7 8 9 10 9. Please rate the severity of the bullying problem at your school/community group prior to the Making A Difference Program. (1 represents a very severe problem and 10 represents a bullying-free environment.) Severe Bullying-free 1 2 3 4 5 6 7 8 9 10 10. Please rate the severity of the bullying problem at your school/community group AFTER the Making A Difference Program. (1 represents a very severe problem and 10 represents a bullying-free environment.) Severe Bullying-free 1 2 3 4 5 6 7 8 9 10 11. What is your overall impression of the group?

________________________________________________

12. What would you suggest to improve the presentation of these sessions in the future?

________________________________________________

13. Would you: YES NO Recommend the Program to others? ___ ___

Were the sessions long enough? ___ ___


87

Was the number of sessions appropriate for the intended purpose? ___ ___

(If NO to any of the above, please provide a reason on back of page.) Please add any additional comments and/or suggestions.

____________________________________________________________________________

____________________________________________________________________________

___________________ _____

YOUR ON GOING EFFORTS OF REINFORCEMENT ARE VITAL TO THE CONTINUED SUCCESS OF OUR PROGRAMS!

THANK YOU!

Appendix H – Process Matrix

88


Inputs/Activities/ Outputs

Indicator(s)

Measurement Tool


Person responsible for data collection?


Facilitators Fidelity Did the program facilitators deliver the same program consistently?

Delivery of the Making A Difference Program (8-1 hr. sessions per wk.)

Conversational

review on a monthly basis

Fidelity Checklist (Quantitative) # of “yes”

for questions 1 through 10 weeks 1-8)

Making A Difference Fidelity Check Weeks 1 to 8

-Program Facilitators complete the checklist -At the end of every session (weeks 1-8)


November 2010 through June 2011

Did the program facilitators deliver the same power point presentation for weeks 1 through 8?

# of “yes” for question 3 (weeks 1, 2, 3, 4, 5 & 8 and question 4 weeks 6 & 7)

What activities worked well? What activities did not work so well?


Conversational

review on a monthly basis

Fidelity Checklist (Qualitative) Review

comments section for qualitative responses


-Program Facilitators complete the checklist -At the end of every session (weeks 1-8)


November 2010 through June 2011


89



Indicator(s)

Measurement Tool




Teachers (Co-leaders)

Were teachers satisfied with the quality of the Making A Difference Program?

Teacher Education and co-leading of groups (8- 1 hr. sessions per week)

Coleader Evaluation (Quantitative) # of

ratings 8 or greater on the 1-10 rating scale for questions 1, 2, 4


qualitative responses to question 11 & 13

Coleader Evaluation




Were teachers satisfied with the amount of information students received?

Teacher Education and co-leading of groups (8- 1 hr. sessions per week)

Coleader Evaluation (Quantitative) # of

ratings 8 or greater on the 1-10 rating scale for questions 3 & 5



Coleader Evaluation




Students Did the students feel the program was helpful?


Students Making A Difference Post Survey (Qualitative) Review

qualitative

Making A Difference School Safety Post Survey

-Program Facilitators will provide the evaluations to students and collect on the same day




90



Indicator(s)

Measurement Tool




Conversational review on a monthly basis with other facilitators

responses to questions 16, 17, 18

-Students complete the post survey on the last session (week 8) of the Making A Difference program -On the last day of the program (week 8)


91



Indicator(s)

Measurement Tool




Target population

How many students were reached?


Fidelity Checklist

Calculate the # of participants who completed the student post survey

Review Fidelity checklist attendance


-Program Facilitators will take weekly attendance -Program Facilitators will provide the evaluations to students and collect on the same day -Students complete the post survey on the last session (week 8) of the Making A Difference program -On the last day of the program (week 8)



How many teachers were reached?


Fidelity Checklist Calculate

the # of Teachers who completed the Coleader Evaluations

Review Fidelity Checklists comments section to see if other teachers were recorded as being present


-Program Facilitators will take case notes on the fidelity checklist to record a staffing change involving coleaders or the presence of extra teachers -Program Facilitators will provide the evaluations to classroom teachers and collect on the same day)-Teachers




92



Indicator(s)

Measurement Tool




complete the evaluation -On the last day of the program (week 8)

Appendix J – Teacher Satisfaction Data

93

Would you recommend the program to others?

Frequency Percent Valid Percent

Cumulative

Percent

Valid no 2 4.9 5.4 5.4

yes 35 85.4 94.6 100.0

Total 37 90.2 100.0

Missing 99 4 9.8

Total 41 100.0

Were the sessions long enough?


Cumulative

Percent

Valid no 4 9.8 11.1 11.1

yes 32 78.0 88.9 100.0

Total 36 87.8 100.0

Missing 99 5 12.2

Total 41 100.0

Was the number of sessions appropriate for the intended purpose?


Cumulative

Percent

Valid no 6 14.6 17.1 17.1

yes 29 70.7 82.9 100.0

Total 35 85.4 100.0

Missing 99 6 14.6

Total 41 100.0

Appendix J – Teacher Satisfaction Data

94

How engaged was the classroom teacher


Cumulative

Percent

Valid low engagement 14 34.1 37.8 37.8

high enagement 23 56.1 62.2 100.0

Total 37 90.2 100.0

Missing 99 4 9.8

Total 41 100.0

Only 62% of the classrooms had an engaged teacher.

FINAL OUTCOMES REPORT MEASURING THE EFFECTIVENESS OF · PDF fileFINAL OUTCOMES REPORT ....

Documents

Transcript of FINAL OUTCOMES REPORT MEASURING THE EFFECTIVENESS OF · PDF fileFINAL OUTCOMES REPORT ....