Effects of School-Wide Positive Behavioral Interventions andSupports and Fidelity of Implementation on Problem Behavior

in High Schools

K. B. FlanneryUniversity of Oregon

P. FenningLoyola University

M. McGrath Kato and K. McIntoshUniversity of Oregon

High school is an important time in the educational career of students. It is also a timewhen adolescents face many behavioral, academic, and social-emotional challenges.Current statistics about the behavioral, academic, and social-emotional challengesfaced by adolescents, and the impact on society through incarceration and dropout,have prompted high schools to direct their attention toward keeping students engagedand reducing high-risk behavioral challenges. The purpose of the study was to examinethe effects of School-Wide Positive Behavioral Interventions and Supports (SW-PBIS)on the levels of individual student problem behaviors during a 3-year effectiveness trialwithout random assignment to condition. Participants were 36,653 students in 12 highschools. Eight schools implemented SW-PBIS, and four schools served as comparisonschools. Results of a multilevel latent growth model showed statistically significantdecreases in student office discipline referrals in SW-PBIS schools, with increases incomparison schools, when controlling for enrollment and percent of students receivingfree or reduced price meals. In addition, as fidelity of implementation increased, officediscipline referrals significantly decreased. Results are discussed in terms of effective-ness of a SW-PBIS approach in high schools and considerations to enhance fidelity ofimplementation.

Keywords: positive behavior support, high schools, school-wide intervention

High schools are the final educational stopbefore young adults enter the workforce orpostsecondary institutions. There are manybehavioral, academic, and social-emotionalchallenges faced by adolescents during these

years. According to a survey conducted by theCenters for Disease Control and Prevention(2011), over 32% of adolescents reported thatthey had been in a physical fight, 20% hadbeen bullied at school and over 7% had at-tempted suicide.

In addition to these high-risk behaviorsamong many adolescents, keeping students inhigh school is an additional problem. Recentlyreported National Center on Educational Statis-tics (NCES) data about event dropout rates (i.e.,high school students who dropped out of schoolin a particular academic year), are promising asa lower percentage of enrolled high school stu-dents dropped out during the 2009 academicyear (3.4%) compared with 1972 (6.1%; Chap-man, Laird, & KewalRamani, 2010). Statusdropout rates, defined as individuals betweenthe ages of 16–24 without a high school di-ploma not currently or ever enrolled in high

This article was published Online First November 4, 2013.K.B. Flannery, Educational and Community Supports,

University of Oregon; P. Fenning, School of Education,Loyola University; and M. McGrath Kato and K. McIntosh,Educational and Community Supports, University of Ore-gon.

The development of this article was supported in part bygrants from the Institute of Education Sciences, U.S. De-partment of Education (#R324A070157). Opinions ex-pressed herein are the authors’ and do not reflect necessarilythe position of the U.S. Department of Education, and suchendorsements should not be inferred.

Correspondence concerning this article should be ad-dressed to K. B. Flannery, 1235 University of Oregon,Eugene, Oregon 97403-1235. E-mail: brigidf@uoregon.edu

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

School Psychology Quarterly © 2013 American Psychological Association2014, Vol. 29, No. 2, 111–124 1045-3830/14/$12.00 DOI: 10.1037/spq0000039

111

school, were also reported. In 2008, three mil-lion individuals met the criterion for “statusdropout” in the United States (Chapman et al.,2010).

Keeping students engaged in high school andavoiding dropout is more important than ever,given increased focus on technological and pro-fessional positions and the necessity of educa-tion for gainful employment. One key contrib-utor to school disengagement by high schoolstudents is the likelihood that when behavioralchallenges occur, high school personnel aremore likely to handle problematic behaviorsthrough punitive disciplinary measures (Fen-ning et al., 2012). Suspension and expulsion arethe most commonly used in high schools, de-spite the fact that they are predictive of reducedschool connectedness, increased dropout, andentry to the juvenile justice system. Instead ofrelying on exclusionary responses that arelinked to poor outcomes for students, highschools need discipline practices that improvelong-term student outcomes for students requir-ing behavior support (Centers for Disease Con-trol & Prevention, 2011; Losen & Skiba, 2010).

To decrease problem behaviors and schoolexclusionary practices, it has been recom-mended that schools focus on an approach thatis prevention-oriented in nature (Fenning et al.,2012). One such approach is School-Wide Pos-itive Behavioral Interventions and Supports(SW-PBIS), which is a multitiered system ofsupport, aligned with response to intervention,in which the school focuses on developing apredictable, efficient, and effective school cli-mate; students are taught expected behaviorsand provided support for success on a preven-tion-oriented basis; and data are used to evalu-ate outcomes and make decisions regarding stu-dent needs and school practices (Lewis, Barrett,Sugai, & Horner, 2010). Students receive be-havioral support along a continuum, rangingfrom all students receiving direct instructionand acknowledgment of expected behaviors(universal), to the delivery of supplemental/group supports (secondary) and/or more inten-sive individualized interventions (tertiary) forthose who continue to show behavioral chal-lenges despite strong universal and supplemen-tal behavior systems in place (Committee onSchool Health, 2013).

In elementary and middle school settings,implementation of SW-PBIS has been associ-

ated with positive outcomes such as improvedschool climate, improved achievement, and re-ductions in referrals to the office (Horner,Sugai, & Anderson, 2010), including in ran-domized control trials (Bradshaw, Koth, Thorn-ton, & Leaf, 2009; Horner et al., 2009). Otherresearch studies at elementary and middleschools have demonstrated an impact of SW-PBIS the reduction of office discipline referrals(ODRs), decreased suspensions, and increasedacademic skills (e.g., Algozzine, Wang, & Vio-lette, 2011; Barrett, Bradshaw, & Lewis-Palmer, 2008; Horner et al., 2009; Luiselli, Put-nam, & Sunderland, 2002; Taylor-Green et al.,1997; Warren et al., 2006).

However, in high schools, the evaluation ofoutcomes has been much more limited and re-search efforts have not incorporated experimental orquasi-experimental methodologies. A limitednumber of studies have examined implementa-tion of SW-PBIS across grade levels, includinghigh schools. One study by Muscott, Mann, andLeBrun (2008) examined outcomes related tostatewide achievement data in schools imple-menting SW-PBIS in a northeastern state. Al-though student improvements were documentedin this study, there were no comparators, andtwo of the four high schools included in thestudy dropped out. Case studies have also beenemployed to document the effects of SW-PBISat the high school level. For example, Bohanonet al. (2006) implemented SW-PBIS in a ra-cially and ethnically urban high school with amajority of students qualifying for free or re-duced-price lunch (FRL; 89%) and a sizablepopulation of English Language Learners (ELL;21%). The school team used the first 2 years toorganize their SW-PBIS team, interview staffabout school needs, and organize discipline re-ferral data. During the third year, the team im-plemented SW-PBIS and obtained a 20% reduc-tion in ODRs and a decrease in the proportion ofstudents requiring more intensive support butno data were collected for comparison schools.

Successful implementation of any innovationrequires attention to the context where it isbeing implemented (Fixsen, Naoom, Blase,Friedman, & Wallace, 2005). Although highschools differ from each other, there are somecommon contextual variables that should beconsidered when implementing in any highschool. SW-PBIS may be more difficult to im-plement in high schools because of these unique

112 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

conditions. For example, high school staff maybelieve that high school students should alreadyhave learned socially appropriate behaviors.Therefore, high school personnel may perceiveless of a need to explicitly teach behaviors totheir students. However, the direct teaching ofexpected behaviors may be a necessary elementof effective SW-PBIS implementation in highschools as in other settings. Morrissey, Boha-non, and Fenning (2010) provided a case studyfocused on the processes and outcomes of anurban school that taught and acknowledged be-havioral expectations as part of their SW-PBISeffort. The findings from this case study indi-cated that the overall rate of ODRs decreasedafter the addition of the systematic teaching andacknowledging of appropriate behaviors duringthe implementation of SW-PBIS.

In addition, structural variables specific tohigh schools may present barriers to SW-PBISimplementation. For example, the sheer size ofhigh schools, the number of administrators andfaculty compared with elementary and middleschool settings, the organization of the facultyinto departments, the content focus that highschool teachers may bring to the table (e.g.,English, Science) and the developmental stageof adolescents are some of the high school con-textual variables that may make implementationof system level behavior supports such as SW-PBIS challenging (Bohanon, Flannery, Malloy,& Fenning, 2009).

The contextual variables of high schools needto be considered throughout professional devel-opment and implementation of practices. Flan-nery, Frank, Doren, Kato, and Fenning (2013)examined the implementation of universal SW-PBIS components in eight of the same highschools included in the current study, which intotal served over 15,525 students across a3-year time period. In this study, technical as-sistance was provided to the high schools, witha focus on supporting the school teams in im-plementing standard SW-PBIS components in-tended for all school environments, as well asspecific attention to the common high schoolcontextual variables that might be unique tohigh schools and, therefore, impact implemen-tation. SW-PBIS implementation components,as measured by the School-Wide EvaluationTool (SET; Sugai, Lewis-Palmer, Todd, &Horner, 2001), indicated positive growth inSW-PBIS implementation as measured over

time, but statistically significant gains in SW-PBIS components did not occur until the secondyear of implementation.

In summary, the evidence base for SW-PBISin high schools is still in the early stages. At-tention to the high school context within theSW-PBIS framework supports the potential forpositive outcomes among students in highschools that implement SW-PBIS. The purposeof the current study was to examine the effectsof SW-PBIS on the levels of individual studentproblem behaviors during a 3-year effectivenesstrial without random assignment to condition. Asecondary research question was the extent towhich the level of fidelity of implementation ofSW-PBIS produced changes in problem behav-ior.

Method

Settings and Participants

The sample included 36,653 students in 12high schools, with half of the schools in onestate in the U.S. Pacific Northwest and half inone state in the U.S. Midwest. The schoolsranged in enrollment from 751 to 2,942 stu-dents, with an average student enrollment of1,770 (SD � 673.72). The average percentageof students receiving free or reduced-pricelunch (FRL) was 27% (range 7%–43%; SD �15%). The percentage of minority studentsacross all sites ranged from 13% to 52% (M �30.65%; SD � 13.89%). The average highschool event dropout rate was 1.84 (range .7–4.4; SD � 1.12). Ten school districts were rep-resented in this study, with one district havingthree schools in the study. Of the schools re-porting the number of full-time classroomteachers in National Center of Educational Sta-tistics database (NCES; http://nces.ed.gov/ccd/schoolsearch), the average was 138.11 (n � 9;range � 38–238; SD � 70.89). See Table 1 fordescriptive statistics for intervention and com-parison schools during the baseline year, as wellas school-level referrals throughout the study.To assess any differences between the groups atbaseline, Mann–Whitney U tests (nonparamet-ric tests, due to the small number of schools)were conducted, and no statistically significantdifferences were found, including school-levelODR rates (p � .37).

113EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Measures

Problem behavior. The number of ODRsreceived by each student per year was used tomeasure levels of individual student problembehavior. All ODR data were collected by eachschool’s staff and entered into their school’sdatabase. Researchers obtained the data fromthe school after each school year. Although thevalidity of ODRs as a direct measure of com-plex student behavior has been debated (Kern &Manz, 2004; Nelson, Gonzales, Epstein, &Benner, 2003; Rusby, Taylor, & Foster, 2007),when collected in a standardized manner, ODRsare considered to be valid indicators of the ratesof problem behavior, delinquency, punitive dis-cipline procedures, and perceived school safety(Irvin, Tobin, Sprague, Sugai, & Vincent, 2004;Morrison, Peterson, O’Farrell, & Redding,2004). Individual counts of ODRs have beenshown to correlate strongly with standardizedbehavior rating scales and outcomes of screen-ing measures for behavior disorders (McIntosh,Campbell, Carter, & Zumbo, 2009; Walker,Cheney, Stage, & Blum, 2005).

Fidelity of implementation. Fidelity ofimplementation of SW-PBIS was assessed us-ing the School-Wide Evaluation Tool (SET;Sugai et al., 2001). The SET is an externalevaluation tool that measures the implemen-tation of SW-PBIS and results in a score ofthe percentage of critical features that areimplemented to criterion. The SET consists of28 items (scored on a 3-point scale with 0 �not implemented, 1 � partial implementation,

and 2 � full implementation) that are distrib-uted into seven subscales measuring whetherschools have implemented the essential fea-tures of SW-PBIS: (a) defined behavioral expecta-tions, (b) taught behavioral expectations, (c) estab-lished ongoing system for rewarding behavioralexpectations, (d) achieved consensus on systemfor responding to behavioral violations, (e) en-gaged in ongoing behavioral monitoring anddecision making, (f) maintained effective man-agement practices, and (g) secured district-levelsupport for ongoing implementation. Each im-plementation rating is criterion referenced withregard to the specific practice in question. For amore in-depth description of the SET, pleasesee Horner et al. (2004).

The SET administration includes a schoolvisit, brief interviews with administration,school staff and students, records reviews,and observations. Studies of its psychometricproperties show excellent internal consistency(� � .96), interrater (99%), and test-retest(.97) reliability, moderate to strong concur-rent validity with other measures of SW-PBISfidelity of implementation (r � .75), and sen-sitivity to SW-PBIS training (Horner et al.,2004). The interrater agreement was per-formed on 76% of the SET administrationsduring this study and the reliability was 88.93% (SD � 7.5%). The SET was adminis-tered in each year of the study (Year 1 [base-line], Year 2, and Year 3) in both theimplementation and comparator sites.

Table 1Sample Descriptive Statistics (Baseline Year)

Mean (SD)

Intervention schools(n � 8)

Comparison schools(n � 4)

Enrollment 1886.50 (771.69) 1539.25 (409.38)Percent non-White students 32% (15) 28% (14)Percent free/reduced meals 34% (14) 19% (12)Academic achievement (% meeting or exceeding standards) 65% (4) 70% (7)School-wide evaluation tool baseline score 47% (25) 33% (24)Dropout rate 2.24 (1.17) 1.05 (0.35)Full time classroom teachers 110 (53.02) 152.17 (78.77)ODRs (school-level rate per 100 students per day): Year 1 (baseline) 1.72 (1.82) 0.89 (0.86)ODRs (school-level rate per 100 students per day): Year 2 1.87 (2.31) 0.70 (0.66)ODRs (school-level rate per 100 students per day): Year 3 1.02 (0.48) 1.00 (1.07)

114 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

Procedure

This study had three primary steps: (a) re-cruitment of six sites from two states (fourimplementation sites and two comparator sites);(b) structured professional development in Year1 (before implementation with students) andtraining each subsequent year with universalSW-PBIS content specifically tailored for highschools; and (c) delivery of monthly technicalassistance by project staff based on the school’saction plan. Each of these steps is describedbelow.

Recruitment. After institutional researchboard approval at the two partner universities,the project used a multistep process for select-ing the implementation and comparator schools(see Figure 1). First, the research project leadinvestigator from each state contacted the SW-PBIS coordinators within their state, explained

the expectations of the project and obtainedrecommendations of high schools that theythought would be interested in participating andhad the following components in place: (a) anadministrative interest in implementing SW-PBIS in their high schools, (b) a team of indi-viduals to support implementation of SW-PBIS,and (c) a coach or school psychologist trained inmultitiered supports available to support theteam. Second, National Center on EducationalStatistics (NCES) demographic data were gath-ered about each of the nominated high schoolsthat were within a 2-hr driving time from therespective universities. Data from NCES in-cluded enrollment, number of teachers, percent-age of FRL, ethnicity, school type, and locale.Schools were removed as potential sites if they(a) enrolled fewer than 500 students, (b) werenot a building serving students in Grades 9–12,

Figure 1. Implementation process.

115EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

and/or (c) the locale was “rural.” This processresulted in 18 potential schools in the two states.Third, a member of the research team contactedpotential site administrators to determine thelevel of interest in meeting about participatingin the project. Upon contact, two schoolselected not to participate: one school saw noneed for SW-PBIS as they didn’t see behavioras a concern, and the second school was alreadyinvolved in a SW-PBIS initiative with anothergroup. For the schools expressing interest inparticipation, a meeting was held with eachbuilding administrator and often a district ad-ministrator, coach, or the school psychologist.At this meeting, the project was described, andthe district personnel shared information aboutcurrent work related to SW-PBIS and any cur-rent initiatives that might interfere with focus-ing on SW-PBIS as an initiative. Principalswere asked to provide basic information abouttheir school related to the implementation ofmultitiered systems (e.g., the structure andmeetings of the universal team, the administra-tive support for the effort, presence of buildingor district coach). Two schools withdrew at thispoint: one did not have a coach who couldprovide support and the other was already part-nering with another group on SW-PBIS.

From the pool of 14 remaining schools, thecross-university team identified 12 schools thatmet the following criteria from the Implemen-tation Site Selection Scoring Guide: (a) a teamexists that is representative, has a designatedleader, and focuses on school-wide behaviorand/or academics; (b) administrator philosoph-ically supports SW-PBIS principles, attendsteam meetings and/or is accessible to the teamfacilitator; and (c) universal data are gathered inat least one area and available for review by theteam.

Considerations in assignment of schools totreatment or comparison conditions were: (a)proportionate distribution of schools in eachcondition across states (e.g., equal number oftreatment schools in each state); and (b) schoolcharacteristics such as FRL, minority, and en-rollment across all schools. As there were onlytwo comparators per state, there was not a 1:1match, but comparator schools were selectedthat had similar demographics (e.g., size, pro-portion of student body by race/ethnicity) to theselected implementation sites. After consentsand memoranda of understanding were received

from the districts, a SET was conducted in eachschool, and project staff worked to collect initialschool (e.g., days per year, enrollment) andstudent data (e.g., ODRs, Achievement, Atten-dance) from the previous year. The extant datacollection process took several months for thedistrict to prepare and share with researchersand, therefore, data were not available for as-signment decisions.

Training. The Leadership Team from eachof the implementation sites received approxi-mately 16 hr of initial training. Training contentwas organized in modules and specifically ori-ented toward implementation of SW-PBIS inthe high schools with the use of high schoolexamples, strategies to remove potential highschool specific barriers, and exploration ofunique aspects of the high school context. Inparticular, the training highlighted the systemvariables that may impact implementation in ahigh school, such as school size (e.g., staff,students, amount of data), structure and organi-zation (e.g., departmentalization, typical use ofmeetings, administrative team, distribution ofdata systems across multiple individuals), anddevelopmental level of students (e.g., desire tobe autonomous, value on being part of decisionmaking, continued need for teaching of appro-priate behaviors). The team training included con-tent on: Impact of the High School Context on Fea-tures of SW-PBIS; Strategies to Support Buy-In;Development of Infrastructure: Leadership Team,Communication, and Decision Making; Using Datafor Decision Making; Developing and Teaching Ex-pectations; Developing Acknowledgment and Con-sequence Systems. At least 50% of the teammembers from each school participated in thetraining and in the majority of instances, anadministrator was also present. Each team thendeveloped training for their overall facultybased on the areas the team identified as prior-ities in their buildings.

During subsequent years, an additional 18 hrof team training was provided. This trainingfocused on specific areas of interest identifiedby the teams as important and included sharingof strategies by each of the teams. The contentof these sessions focused on a range of topicsdeemed important by the school teams and admin-istrators, such as accessing and using existing highschool data systems for formative decisionmaking, developing consequence systems thataddress student function, meaningfully involv-

116 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

ing students in implementation, establishingbuy-in from staff, fostering healthy teaming,and planning for sustainability. Cross site fo-rums were held involving all implementationhigh schools sharing resources and engaging inproblem solving around challenges of SW-PBISimplementation and finding creative strategiesto address barriers in a positive manner.

Technical assistance. Each site was as-signed a staff from the university project teamto provide technical assistance (e.g., coaching,modeling, training, referral to resources) to theuniversal (school-based) team and internal fa-cilitator/leader throughout implementation. Thetechnical assistance person worked with theschool teams to establish an annual action plan.To assist in the development of the action plans,sites utilized variety of data including fidelitydata (e.g., SET) and extant student-level data.The action plan was based on the priorities ofeach site and guided the type of external tech-nical assistance provided. The action plan wasreviewed and updated each quarter by the teamfacilitator and the external technical assistanceperson assigned to that school. Though the areasof focus differed across sites due to the individ-ualized needs of each setting, typical items in-cluded on the action plan were tasks related tothe development of clearer norms and proce-dures for the team, how to involve students inSW-PBIS development and implementation,communication strategies for both staff and students,review and clarification of current practices re-lated to student expectations and consequencesfor inappropriate behavior, school-level disci-plinary data and patterns in the most commontypes of disciplinary problems, and develop-ment of acknowledgment systems for staff andstudents.

The external technical assistance providermet about once per month with the SW-PBISleadership team at each site and from this meet-ing the amount and type (e.g., phone, face toface) of technical assistance needed was deter-mined. The amount of technical assistance tothe facilitator of the team and the full teamranged from 20 to 65.5 hr (M � 33.84; SD �16.46) per year. The assistance was provided byproject staff through a variety of mechanisms,such as attending team meetings, phone calls ore-mails to the internal high school team facili-tator, attending school-wide events, the deliveryof a cross-site forum, and providing examples

or resources from other schools. Table 2 pro-vides information about the types of actions theteams focused on during the implementationperiod.

Design and Data Analysis

The study was a prepost intervention-compari-son cluster trial, with three waves of measurement(Year 1 [baseline], Year 2, and Year 3). A cohort-sequential design was used, in which all studentswere included in analyses at each year and max-imum likelihood estimation was used to accountfor the planned missing ODR data. Because gradecohort was not a statistically significant predictorof the outcome, it was not included in final models(Hox, 2010). Two similar analyses were con-ducted. To assess the effects of SW-PBIS condi-tion (i.e., intervention vs. comparison) on problembehavior, the analysis was a multilevel latentgrowth model, with intervention condition pre-dicting baseline scores (level) and change overtime (slope) in individual student ODRs. BecauseODRs are a count variable with a negative skew(i.e., many students had 0 ODRs each year), Pois-son regression was used. Standard errors wereadjusted to account for the nested data structure(i.e., students within schools, with intervention atthe school level). In addition, school-level enroll-ment and the percent of students receiving FRLwere added as covariates, centered at the grandmean, to control for the effects of these variableson both ODRs and fidelity of implementation(Domitrovich et al., 2008). To assess the effects offidelity of implementation, SET scores duringeach year were substituted as time-varying cova-riates in place of condition. All analyses wereconducted using Mplus 6.11 (Muthén & Muthén,1998–2010).

Due to a corrupted school ODR data spreadsheet,ODR data were missing for one comparison schoolin 1 year. These missing data represented 1.4%of the dependent variables. Missing data werehandled using the maximum likelihood estima-tor in Mplus because it produces less biasedparameters than listwise or casewise deletion(Baraldi & Enders, 2010).

Results

Figure 2 shows the school-level rates ofODRs per 100 students per day for interventionand comparison schools for each year of the

117EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

study. As seen, the mean rate was considerablyhigher for intervention schools at Year 1 (base-line), but the mean rate was reduced to that ofcomparison schools in Year 3. In terms of effectsizes (Cohen’s d; Cohen, 1988), there was amedium difference at baseline (d � 0.57),

which decreased to a negligible difference atYear 3 (d � 0.03). Figure 3 shows the modelassessing the effects of SW-PBIS on problembehavior. After controlling for FRL and enroll-ment, there was a statistically significant asso-ciation between SW-PBIS and level (i.e., inter-

Table 2Actions Taken by High School Teams

Area of focus Specific actions

Team infrastructure andfunctioning

• Reviewed representation and make adjustments to ensure team participation andrepresentation needed to accomplish the tasks

• Established team norms and scheduled meeting times• Supported facilitator in developing skills needed to lead a group• Established a clear linkage to the administrative team and their decisions• Ensured all team members and staff have foundational knowledge about SW-PBIS• Updated new administrators as they join the building administration• Established subcommittees focused on communication, acknowledgements, and

data• Reviewed and aligned multiple multi-tiered initiatives in building so can be

effective and more efficient• Integrated SW-PBIS into existing multi-tiered initiatives• Developed student teams to focus on specific issues (i.e., bullying, “Use Another

Word”)Buy-in • Implemented strategies to achieve buy-in of philosophy, foundational features, and

practices associated with SW-PBIS• Targeted students, administrators, school staff, security officers, office-staff, related

service staff, parents, and community members in the processExpectations • Developed and implement lessons and strategies to teach the expectations

(involving students) using data to guide priorities• Implemented the use of “boosters” to revisit the expectations throughout the year

at relevant times (e.g., respect before prom, perseverance after midterm)• Engaged in staff professional development, developed strategies for teaching

expectations (e.g., video, skits, student writing assignments) and supported indelivery of lesson plans

Data for decision making • Organized and prepare data in visual format (e.g., tables, graphs) so usable byteam on a monthly basis to make decisions

• Supported person who presents data to team with process to effectively describeand obtain decisions from team

• Worked with district staff to develop output from data systems that is usable to theteams as formative assessment measure

• Integrated other relevant data systems (e.g., attendance, tardy, truancy, creditaccrual data)

• Shared data with larger faculty and staff on an ongoing basis• Used data to report progress of SW-PBIS system and practice components

Behavior referral system • Established a clear referral and consequence system and taught system to staff,students, and parents

• Identify and share what constitutes a classroom versus office-managed referral• Clarify existing policies/processes with staff and students (e.g., tardy, suspension,

detention, appropriate clothing)Acknowledgement system • Identify and implement staff and student acknowledgement systems; review and

refine if necessary• Involve students in the design of the acknowledgment system

Communication with staff andstudents

• Develop strategies and timing for communicating with staff and students aboutstatus and implementation of SW-PBIS in the school

• Develop mechanisms to obtain feedback from staff and students about the SW-PBIS efforts (e.g., surveys, information boxes, small group professionaldevelopment)

118 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

cept) of problem behavior (b � 1.34, p � .001),indicating statistically significantly higher lev-els of problem behavior (individual studentcounts of ODRs, as opposed to the school levelsof ODRs presented in Table 1) in the implemen-tation schools, particularly at baseline. How-ever, there was a statistically significant effectof SW-PBIS on slope (i.e., change) for problembehavior (b � �0.61, p � .001), showing thatstudent problem behavior in the implementationschools significantly decreased over time.

It was also of interest to assess the relationbetween SW-PBIS fidelity of implementationand change in problem behavior. Figure 4shows the model assessing the effects of imple-mentation fidelity on problem behavior. Aftercontrolling for FRL and enrollment, there wasno statistically significant effect of baseline fi-delity of implementation on baseline problembehavior (b � 1.29, p � .52). However, therewas a statistically significant effect of imple-mentation fidelity on problem behavior in Year

Figure 2. Average Office Discipline Referral rates per 100 students per day by condition.

Figure 3. Model for effects of SW-PBIS on individual problem behavior. Note: � p � .05.�� p � .01. ��� p � .001. Beta weights in figure are x-standardized. Error terms, fixed loadings,and covariances removed for figure clarity. Enroll � school enrollment at baseline. FRL �percent of students receiving free or reduced lunch at baseline. ODRs � Office DisciplineReferral counts per individual student.

119EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

2 (b � �3.05, p � .03) and Year 3 (b � �3.18,p � .01), showing that as SET scores in-creased, individual student problem behaviordecreased.

Discussion

This study examined the effects of imple-menting SW-PBIS on levels of individual stu-dent problem behavior in high schools. Resultsof a multilevel latent growth model showed thatalthough rates of problem behavior were higherin intervention schools during the baseline year,there was a statistically significant decrease inproblem behavior for students in schools imple-menting SW-PBIS over the course of the study,whereas there was a steady increase in problembehavior in comparison schools. Follow-upanalyses examining the effect of SW-PBIS im-plementation fidelity showed a statistically sig-nificant inverse relation between strength ofSW-PBIS implementation fidelity and studentproblem behavior. These analyses show that notonly did SW-PBIS significantly reduce problembehavior for students in high schools, but thedegree of reduction was significantly related tothe degree to which the critical features of SW-PBIS, as measured by the SET, were delivered.

Although this study was not experimental innature, it is one of the first known studies toinclude comparator schools in the evaluation ofthe effects of SW-PBIS at the high school level.These findings are somewhat limited, as the rateof baseline ODRs were considerably at the out-set of the study. Yet the finding of decreasedODRs over time in the implementation highschools and a concomitant increase in ODRswithin the comparison schools is noteworthy.As seen in our implementation schools, initialbaseline rates of behavior may be expected tospike once school teams focus on their data andsystematically attend to observing, recording,and organizing behavioral infractions in thebuilding. Technical assistance efforts employedthroughout the study included assisting the im-plementation high schools with systematicallydefining, organizing, and displaying ODRs in away that was useful for system-level universaldecision making by the team (e.g., action plan-ning, setting behavioral priorities in their build-ing, evaluating the results of their behaviorpractices).

Data systems in high schools have histori-cally been established for required state report-ing functions rather than serving a formativeassessment function to assist with universal sys-

Figure 4. Model for effects of SW-PBIS fidelity of implementation on individual problembehavior. Note: � p � .05. �� p � .01. ��� p � .001. Beta weights in figure are x-standardized.Error terms, fixed loadings, and covariances removed for figure clarity. Enroll � schoolenrollment at baseline. FRL � percent of students receiving free or reduced lunch at baseline.ODRs � Office Discipline Referral counts per individual student. SET � School-wideEvaluation Tool.

120 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

tem level efforts, such as SW-PBIS. For exam-ple, teams were often challenged by not havingregular access to data that could be manipulatedin a user-friendly way. Unlike academic efforts,in which curriculum-based measurement toolsare readily available as formative assessmentmeasures (Shinn, 2013), the access to user-friendly school-wide behavioral databases fordecision making was a barrier for teams fromthe outset of the study. High school teams re-quired a great deal of assistance to organizetheir data and strategies that allowed them toanalyze and summarize the variety of data highschools are required to collect (e.g., attendance,tardy, truancy data, credit accrual). Future workwith high schools should address how to sup-port high school systems with establishing be-havioral databases supportive of universal be-havioral supports.

ODRs have a number of weaknesses, such as afocus on externalized records of behavior (Morri-son, Furlong, & Morrison, 1994), but they arecollected in virtually every high school and areuseful in assessing the school climate and behav-ioral needs of the building. The reductions ofODRs in this study, though not experimentallytied to SW-PBIS, are an important metric becausediscipline referrals to the office are associated withother unwanted outcomes, such as school dropoutand entry to the juvenile justice system (Losen &Skiba, 2010; Wald & Losen, 2003). More work iscertainly needed to document the impact of SW-PBIS using a broader array of data that is feasibleto collect and deemed important for high schoolstudent outcomes, including school climate andinternalizing behaviors not captured throughODRs, as well as variables such as dropout rates,credit accrual, and entry to postsecondary institu-tions.

Another important finding was that fidelity ofSW-PBIS implementation, as based on the SETfindings, mattered in terms of desired outcomesfor students, specifically through reductions inODR rates. Schools that had higher SET scoresexperienced greater reductions in ODRs thanschools with weaker implementation, suggestingthat high schools that more closely align withSW-PBIS components have better outcomes.Therefore, working with high schools in adheringto the major SW-PBIS systems, data, and prac-tices might pay dividends in terms of studentoutcomes. Focusing on system-level and founda-tional practices, such as achieving large-scale staff

and student buy-in, sustaining administrative sup-port, facilitating healthy teaming, aligning SW-PBIS with other multitiered systems of supports/initiatives, and establishing system-level datapractices were critical steps prior to the implemen-tation of practices. Given that system-level andfoundational factors take time, we found that en-couraging school teams to celebrate their struc-tural planning efforts was very important. Schoolteams needed to feel that they were “making prog-ress” in system-level structures, which is hardwork in the absence of any implementation effortsand actual outcomes with students. Achieving ad-equate fidelity of implementation is often chal-lenging, particularly when taking the necessarytime to establish a solid foundation of buy-in, datasystems, and teaming and then applying system-level supports within high schools that, by theirvery nature, have a number of challenging con-textual variables described earlier (Bohanon et al.,2006; Bohanon et al., 2009).

The unique characteristics of high schoolswere attended to during formal professional de-velopment efforts and throughout ongoing tech-nical assistance. Although adhering to the stan-dard components such as the direct teaching andacknowledging of behavior is important toachieving implementation fidelity of SW-PBISat any level, high schools required a somewhatdifferent focus or format compared with ele-mentary and middle schools. For example, thedevelopmental age of the students was ad-dressed through schools identifying meaningfulstrategies for involving them. For example,some schools included students in the produc-tion of videos that were used as teaching toolsrather than relying only on more traditionallydelivered teacher-led lesson plans, which iscommon among implementation of SW-PBIS inlower grades. It was also important to considerthe developmental level of adolescents whenconsidering ways to acknowledge them. Forexample, publicly announcing students’ nameson a loud speaker for appropriate behavior maynot be rewarding to most high school students.Finding other ways to acknowledge studentsand staff, such as close parking spaces, promtickets, restaurant gift cards, passes to the frontof the lunch line, and so forth were more highlyregarded by students in the participating highschools. For more examples on practical highschool strategies see the monograph developedby the SW-PBIS Technical Assistance Center

121EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

(Flannery & Sugai, 2010).The current findingssuggest that adapting standard practices andsystems to the high school context is importantto achieve fidelity of implementation and sus-tain it over time. This study documents positivefindings for high schools similar to randomizedcontrolled trials for SW-PBIS at the elementarylevel (Horner et al., 2010). The implementationefforts in the current study integrated profes-sional development and technical assistancecommon to “standard” implementation of SW-PBIS with the understanding of the impact thathigh school culture and structural variables mayhave on the implementation of initiatives.

The structural variables found in high schoolsmay account for the longer time needed beforereductions in ODRs were realized, as has beenarticulated in case study research with similarpopulations (Bohanon et al., 2006). In the firstyear of this study, ODRs were actually higher inthe implementation schools in relation to thecomparison schools and then decreased mark-edly over time. Given the larger size of highschools, the needed buy-in and support of manymore faculty, administrators, students, and com-munity members in high schools relative toother educational environments, it might be thecase that these structural features contribute tothe necessity of longer implementation beforepositive student level results are seen. As hasbeen articulated, school psychologists and otherconsultants who provide technical assistance tohigh schools should plan for ways to supportteams in the foundational period in which im-portant structures are established, including thefollowing: (a) developing effective teamingstructures, (b) finding strategies to work withhigh school staff members who might not be-lieve that behaviors need to be taught to highschool students, (c) aligning and integratingSW-PBIS practices with other initiatives, and(d) establishing and systematizing formativedata collection systems that utilize ODRs andother relevant sources of data, such as creditsaccrued, attendance, tardy and truancy data.

Limitations

A number of important limitations may haveled to the obtained results. First, the small num-ber of schools in the study limited the varianceat the school level. Although the large numberof students in the sample led to adequate statis-

tical power, results of future studies with moreschools could be interpreted with more confi-dence. Second, the assignment of schools tocondition was not random, although there wereefforts to select similar intervention and com-parison schools. Because assignment was notrandom, causal attributions cannot be made.Third, most schools in the study, includingcomparison schools, were implementing at leastsome critical features of SW-PBIS during thebaseline year. As mentioned earlier, anotherlimitation was the omission of ODRs as a se-lection criterion during the school recruitmentprocess. It is recommended that researchers col-lect ODR rates as part of the site selectionprocess. Using these data in assignment to con-dition may result in a stronger test of the inter-vention and more generalizable outcomes.Given the reality of the need for technical as-sistance in organizing data that can be readilyaccessed and meaningfully used by schoolteams to make decisions, this component couldbe considered a required step for “readiness” ofSW-PBIS implementation at the high schoollevel. Finally, it should be noted that the use ofODRs to measure student problem behaviormay be less precise and prone to lower reliabil-ity than more objective measures (such as directobservation), potentially because of individualvariations in their use among school personnel,varying tolerance levels for minor problem be-haviors, or bias. In addition, ODRs are limitedto external records of behavior and a number ofsystem variables and do not capture more inter-nalizing and more subtle types of behavior(McIntosh et al., 2009; Morrison et al., 1994).

Future Research

Future research should investigate the effectsof SW-PBIS with more high schools and forlonger periods of time than 2 years of interven-tion. It is possible that effects from SW-PBIStraining may take longer to have an effect thanin elementary schools. Full implementation ofSW-PBIS is known to take a number of years.This period of implementation is commonly 3to 4 years in elementary and middle schools(Sugai, Horner, & McIntosh, 2008), but it hasbeen estimated to be closer to 5 to 8 years inhigh schools (Bohanon et al., 2006; Flannery,Sugai, & Anderson, 2009). Another importantcomponent that needs to be examined is

122 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

whether SW-PBIS in high schools impacts eth-nic disproportionality in discipline, which hasbeen well-documented over many years, partic-ularly for African American males, as well asother variables associated with exclusionarydiscipline, such as school dropout and entry tothe juvenile justice system among ethnic minor-ity youth (Skiba et al., 2011; Wald & Losen,2003). In addition, future studies can examinethe effects of SW-PBIS on other outcomes inhigh schools, such as academic achievement,credit accrual, and graduation rates.

References

Algozzine, B., Wang, C., & Violette, A. S. (2011).Reexamining the relationship between academicachievement and social behavior. Journal of Pos-itive Behavior Interventions, 13, 3–16. doi:10.1177/1098300709359084

Baraldi, A. N., & Enders, C. K. (2010). An introduc-tion to modern missing data analyses. Journal ofSchool Psychology, 48, 5–37. doi:10.1016/j.jsp.2009.10.001

Barrett, S., Bradshaw, C., & Lewis-Palmer, T.(2008). Maryland statewide PBIS initiative: Sys-tems, evaluation, and next steps. Journal of Posi-tive Behavior Interventions, 10, 105–114. doi:10.1177/1098300707312541

Bohanon, H., Fenning, P., Carney, K. L., Minnis-Kim, M. J., Anderson-Harriss, S., Moroz, K. B.,. . . Sailor, W. (2006). Schoolwide application ofpositive behavior support in an urban highschool: A case study. Journal of Positive Behav-ior Interventions, 8, 131–145. doi:10.1177/10983007060080030201

Bohanon, H., Flannery, K. B., Malloy, J., & Fenning,P. (2009). Utilizing positive behavior supports inhigh school settings to improve school comple-tion rates for students with high incidence con-ditions. Exceptionality, 17, 30–44. doi:10.1080/09362830802590193

Bradshaw, C. P., Koth, C., Thornton, L., & Leaf, P.(2009). Altering school climate through school-wide positive behavioral interventions and sup-ports: Findings from a group-randomized effec-tiveness trial. Prevention Science, 10, 100–115.doi:10.1007/s11121-008-0114-9

Centers for Disease Control and Prevention. (2011).Youth risk behavior survey. Atlanta, GA: U.S. De-partment of Health and Human Services. Retrievedfrom www.cdc.gov/yrbs

Chapman, C., Laird, J., & KewalRamani, A. (2010).Trends in high school dropout and completionrates in the United States: 1972–2008 (NCES2011–2012). Washington, DC: National Center for

Education Statistics, Institute of Educational Sci-ences, U.S. Department of Education. Retrievedfrom http://nces.Ed.gov/pubsearch

Cohen, J. (1988). Statistical power analysis for the be-havioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Committee on School Health. (2003). Out of schoolsuspension and expulsion. Pediatrics, 112, 1206–1209. doi:10.1542/peds.112.5.1206

Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M.,Hoagwood, K., Buckley, J. A., Olin, S., . . .Ialongo, N. S. (2008). Maximizing the implemen-tation quality of evidence-based preventive inter-ventions in schools: A conceptual framework. Ad-vances in School Mental Health Promotion, 1,6–28. doi:10.1080/1754730X.2008.9715730

Fenning, P., Pulaski, S., Gomez, M., Morello, M.,Maciel, L., Maroney, E., . . . Maltese, R. (2012).Call to action: A Critical need for designing alter-natives to suspension and expulsion. Journal ofSchool Violence, 11, 105–117. doi:10.1080/15388220.2011.646643

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman,R. M., & Wallace, F. (2005). Implementation re-search: A synthesis of the literature. Tampa, FL:University of South Florida, Louis de la ParteFlorida Mental Health Institute, The National Im-plementation Research Network (FMHI Publica-tion #231).

Flannery, K. B., Frank, J. L., Doren, B., Kato, M. M., &Fenning, P. (2013). Implementing schoolwide positivebehavior support in high school settings: Analysis ofeight high schools. The High School Journal, 96,267–282. doi:10.1353/hsj.2013.0015

Flannery, K. B., & Sugai, G. (2010). School-widePBIS implementation in high schools: Currentpractice and future directions. Eugene, OR: Uni-versity of Oregon, Educational and CommunitySupports. Retrieved from http://pbis.org

Flannery, K. B., Sugai, G., & Anderson, C. M.(2009). School-wide positive behavior support inhigh school: Early lessons learned. Journal of Pos-itive Behavior Interventions, 11, 177–185. doi:10.1177/1098300708316257

Horner, R. H., Sugai, G., & Anderson, C. M. (2010).Examining the evidence base for school-wide pos-itive behavior support. Focus on Exceptional Chil-dren, 42, 1–14.

Horner, R. H., Sugai, G., Smolkowski, K., Eber, L.,Nakasato, J., Todd, A., & Esperanza, J. (2009). Arandomized, wait-list controlled effectiveness trialassessing school-wide positive behavior supportin elementary schools. Journal of Positive Behav-ior Interventions, 11, 133–144. doi:10.1177/1098300709332067

Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin,L. K., Sugai, G., & Boland, J. B. (2004). TheSchool-Wide Evaluation Tool (SET): A research in-strument for assessing school-wide positive behavior

123EFFECTS OF SW-PBIS IN HIGH SCHOOLS

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.

support. Journal of Positive Behavior Interventions,6, 3–12. doi:10.1177/10983007040060010201

Hox, J. J. (2010). Multilevel analysis: Techniquesand applications. New York, NY: Routledge.

Irvin, L. K., Tobin, T. J., Sprague, J. R., Sugai, G., &Vincent, C. G. (2004). Validity of office disciplinereferral measures as indices of school-wide behavioralstatus and effects of school-wide behavioral interven-tions. Journal of Positive Behavior Interventions, 6,131–147. doi:10.1177/10983007040060030201

Kern, L., & Manz, P. (2004). A look at currentvalidity issues of school-wide behavior support.Behavioral Disorders, 30, 47–59.

Lewis, T. J., Barrett, S., Sugai, G., & Horner, R. H.(2010). Blueprint for schoolwide positive behaviorsupport training and professional development.Retrieved from www.pbis.org

Losen, D. J., & Skiba, R. (2010). Suspended educa-tion: Urban middle schools in crisis. Retrievedfrom http://civilrightsproject.ucla.edu/research/k-12-education/school-discipline/suspended-education-urban-middle-schools-in-crisis

Luiselli, J., Putnam, R., & Sunderland, M. (2002).Longitudinal evaluation of behavior support inter-ventions in public middle schools. Journal of Pos-itive Behavior Interventions, 4, 182–190. doi:10.1177/10983007020040030701

McIntosh, K., Campbell, A. L., Carter, D. R., &Zumbo, B. D. (2009). Concurrent validity of officediscipline referrals and cut points used in school-wide positive behavior support. Behavioral Disor-ders, 34, 100–113.

Morrison, G., Furlong, M. J., & Morrison, R. (1994).School violence to school safety: Reframing theissue for school psychologists. School PsychologyReview, 23, 236–256.

Morrison, G. M., Peterson, R., O’Farrell, S., & Red-ding, M. (2004). Using office referral records inschool violence research: Possibilities and limita-tions. Journal of School Violence, 3, 39–61. doi:10.1300/J202v03n02_04

Morrissey, K. L., Bohanon, H., & Fenning, P. (2010).Teaching and acknowledging expected behaviorsin an urban high school. Teaching ExceptionalChildren, 42, 27–35.

Muscott, H., Mann, E., & LeBrun, M. (2008). Posi-tive behavioral interventions and supports in NewHampshire: Effects of large-scale implementationof schoolwide positive behavior support on studentdiscipline and academic achievement. Journal ofPositive Behavior Interventions, 10, 190–205. doi:10.1177/1098300708316258

Muthén, L. K., & Muthén, B. O. (1998–2010). Mplususer’s guide (6th ed.). Los Angeles, CA: Author.

Nelson, J. R., Gonzales, J. E., Epstein, M. H., &Benner, G. J. (2003). Administrative discipline

contacts: A review of the literature. BehavioralDisorders, 28, 249–281.

Rusby, J. C., Taylor, T. K., & Foster, E. M. (2007).A descriptive study of school discipline referrals infirst grade. Psychology in the Schools, 44, 333–350. doi:10.1002/pits.20226

Shinn, M. R. (2013). Identifying and validating aca-demic problems in a multi-tiered system of ser-vices and supports model in a time of shiftingparadigms. In R. Brown-Chidsey & K. J. Andren(Eds.), Assessment for intervention: A problemsolving approach (pp. 199–228). New York, NY:Guilford Press.

Skiba, R. J., Horner, R. H., Chung, C., Rausch,M. K., May, S. L., & Tobin, T. (2011). Race is notneutral: A national investigation of African Amer-ican and Latino disproportionality in school disci-pline. School Psychology Review, 40, 85–107.

Sugai, G., Horner, R. H., & McIntosh, K. (2008).Best practices in developing a broad-scale systemof support for school-wide positive behavior sup-port. In A. Thomas & J. P. Grimes (Eds.), Bestpractices in school psychology (Vol. 3, pp. 765–780). Bethesda, MD: National Association ofSchool Psychologists.

Sugai, G., Lewis-Palmer, T. L., Todd, A. W., & Horner,R. H. (2001). School-wide Evaluation Tool (SET).Eugene, OR: Educational and Community Supports.Retrieved from http://www.pbis.org

Taylor-Greene, S., Brown, D., Nelson, L., Longton,J., Gassman, T., Cohen, J.,. . . Hall, S. (1997).School-wide behavioral support: Starting the yearoff right. Journal of Behavioral Education, 7, 99–112. doi:10.1023/A:1022849722465

Wald, J., & Losen, D. J. (2003). Editor’s notes. In J.Wald & D. J. Losen (Eds.), New directions foryouth development: Deconstructing the school-to-prison pipeline (pp. 1–2). San Francisco, CA:Jossey-Bass.

Walker, B., Cheney, D., Stage, S. A., & Blum, C.(2005). Schoolwide screening and Positive Behav-ior Supports: Identifying and supporting studentsat risk for school failure. Journal of Positive Be-havior Interventions, 7, 194–204. doi:10.1177/10983007050070040101

Warren, J. S., Bohanon-Edmonson, H. M., Turnbull,A. P., Sailor, W., Wickham, D., Griggs, P., &Beech, S. E. (2006). School-wide positive behav-ior support: Addressing behavior problems thatimpeded student learning. Educational PsychologyReview, 18, 187–198. doi:10.1007/s10648-006-9008-1

Received May 2, 2013Revision received August 22, 2013

Accepted August 26, 2013 �

124 FLANNERY, FENNING, KATO, AND MCINTOSH

Thi

sdo

cum

ent

isco

pyri

ghte

dby

the

Am

eric

anPs

ycho

logi

cal

Ass

ocia

tion

oron

eof

itsal

lied

publ

ishe

rs.

Thi

sar

ticle

isin

tend

edso

lely

for

the

pers

onal

use

ofth

ein

divi

dual

user

and

isno

tto

bedi

ssem

inat

edbr

oadl

y.