7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
1/44
RESEARCH REPORT SEPTEMBER 2013
Teacher Evaluation in Practice
Implementing Chicagos REACH Students
Susan E. Sporte, W. David Stevens, Kaleen Healey, Jennie Jiang, and Holly Hart
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
2/44
1 Executive Summary
3 Introduction
Chapter 1
9 The Classroom Observation Process
Chapter 2
15 The Use o Student Growth inEvaluation
Chapter 3
21 Training and Communication
Acknowledgements
The authors grateully acknowledge the support o the Joyce Foundation and its continuing interest in this
important line o work. This analysis based on one districts experience with a dramatically dierent teacher evalu-
ation system could not have happened without Joyces steadast support and long term commitment. In addition
we would like to acknowledge the support o Chicago Public Schools and the Chicago Teachers Union. We have
gained particular insights rom Matt Lyons, Paulette Poncelet, Sheila Cashman, Elizabeth Press, Didi Schwartz,
Amanda Smith, Susan Kajiwara-Ansai, and Meghan Zeran rom CPS central oce and rom Carol Care o the
CTU as well as other teachers rom the CPS-CTU Joint Committee who have provided valuable eedback as the
work has progressed.
In addition we thank the teachers and administrators o Chicago Public Schools who shared their thoughts
through multiple surveys, and those who provided their valuable time in one-on-one interviews. This report
was made possible through their participation. We also learned rom those individuals serving as Instructional
Eectiveness Specialists and thank them or the extra time they spent sharing with us.
We are also indebted to the Research Assistants who helped in countless ways to make this report possible:
Patrick Wu, Elc Estrera, Jen Cowhy, Catherine Alvarez-McCurdy, Josie Glore, and Gabrielle Friedman. Their con-
tributions have been invaluable. We thank members o the University o Chicago Consortium on School Researchs
Steering Committee who commented on an earlier drat o this reportespecially Peter Godard, Karen Lewis, and
Luis Soria. It is always humbling to have our work critiqued by those with on-the-ground experience and we thank
them or sharing that experience with us. Their thoughts helped shape this nal product. Finally, we owe a deep
debt o gratitude to members o the UChicago CCSR sta who worked with us as thought partners and readers
through numerous iterations to arrive at this nal report: The communications sta, Emily Krone and Bronwyn
McDaniel, who kept us ocused and oered excellent eedback; ellow researchers Elaine Allensworth, Jenny
Nagaoka, Melissa Roderick, Penny Sebring, Marisa de la Torre, and Stuart Luppescu; and our superb nal technical
readers Lauren Sartain and Paul Moore. Our deep discussions with all o them have added to this report as well as
our understanding o the promise and challenge o changing the teacher evaluation system in Chicago.
Chapter 4
25 How Principals Manage Classroom
Observation Workloads
Chapter 5
29 Questions to Consider
33 Reerences
35 Appendices
37 Endnotes
TABLE OF CONTENTS
This report was produced by UChicago CCSRs publications
and communications sta: Emily Krone, Director or Outreach
and Communication; Bronwyn McDaniel, Communications and
Research Manager; and Jessica Puller, Communications Specialist.
Graphic Design: Je Hall Design
Photography: Cynthia Howe and David Schalliol
Editing: Ann Lindner
09.2013/pd/[email protected]
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
3/44
Executive Summary
Executive Summary
This report ocuses on the perceptions and experiences o teachers and
administrators during the rst year o REACH implementation, which was
in many ways a particularly demanding year. These experiences can be
helpul to CPS and to other districts across the country as they work to
restructure and transorm teacher evaluation.
Historically, teacher evaluation in Chicago has allen
short on two crucial ronts: It has not provided adminis-
trators with measures t hat dierentiated among strong
and weak teachersin act, 93 percent o teachers were
rated as Excellent or Superiorand it has not provided
teachers with useul eedback they could use to improve
their instruction.1
Chicago is not uniqueteacher evaluation systems
across the country have experienced the exact same
problems.2Recent national policy has emphasized
overhauling these systems to include multiple measures
o teacher perormance, such as student outcomes, and
structuring the evaluations so they are useul rom both
talent management and teacher proessional develop-
ment perspectives. Principals and teachers need an
evaluation system that provides teachers with speciic,
practice-oriented eedback they can use to improve their
instruction and school leaders need to be able to identiy
strong and weak teachers. Required to act by a new state
law and building o lessons learned rom an earlier pilot
o an evidence-based observation tool,3 Chicago Public
Schools (CPS) rolled out its new teacher evaluation
systemRecognizing Educators Advancing Chicagos
Students (RE ACH Students)in the 2012-13 school yea r.
The REACH system seeks to provide a measure o
individual teacher eectiveness that ca n simultane-
ously support instructional improvement. It incorpo-
rates teacher perormance ratings based on multiple
classroom observations together with student growth
measured on two dierent types o assessments.
Wh ile the pract ice o usi ng cl assroom observations
as an evaluation tool is not completely new, REACH
requires teachers and administrators to conceptualize
classroom observations more broadly as being part o
instructional improvement eorts as well as eva luation;
evaluating teachers based on student test score growth
has never happened beore in the district.
REACH implementation was a massive undertak-
ing. It required a large-scale investment o time and
energy rom teachers, administrators, CPS central oice
sta, and the teachers union. District context played an
important role and provided additional challenges as the
district was introducing other major initiatives at the
same time as REACH. Furthermore, the school year
began with the irst teacher strike in CPS in over 25
years. Teacher evaluation was one o several contentious
points in the protracted negotiation, and the speciic
issue o using student growth on assessments to evaluate
teachers received considerable coverage in the media.
This study uses data collected rom a ll 2012
through spring 2013, including:
Two surveys o all 1,195 principals and assistant
principals, administered in December 2012 and
Apri l/May 2013, re spec tively
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
4/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
2
Two surveys o teachers, one administered in
January 2013 to a sample o 2,000 classroom
teachers and one administered in March 2 013
to all teachers in the district
Interviews with a random sample o 31 classroom
teachers and six principals rom six schools,
conducted in spring 2013
Interviews with nine central oice sta members
(Instructional Eectiveness Specialists), conducted
in November 2012
Summary o Main Findings:
Teachers and administrators nd the observation
process useul or improving instruction
Overwhelming majorities o teachers and admin-
istrators believe the observation process supports
teacher growth, identiies areas o strength and
weakness, and has improved t he qu ality o
proessional conversations between them.
Most administrators eel conident in their ability
to gather evidence and assign ratings; a large
majority o teachers believe their evaluator is air
and unbiased and able to assess their instr uction.
Some teachers expressed concern that classroom
observation ratings are too subjective to be usedin high-stakes evaluations, while others eel appre-
hensive about revealing instr uctional weaknesses
or ear o being penalized on their evaluations.
Teachers are hesitant about the use o student
growth on assessments to evaluate their
classroom perormance
Over hal o teachers surveyed believe REACH relies
too heavily on student growth.
Special education teachers are particularly criticaland nd the assessments to be inappropriate
measures o their students learning and their
instruction.
Communication with teachers is an area or
improvement; administrators want support
on coaching and providing useul eedback
The requency and quality o training and
communication received by teachers varies widely.
Teachers are conused about how student growth
actors into their inal rating. Both teachers and
administrators need clarity about score ca lculations
and how they will be used or personnel decisions.
Most administrators list coaching and providing
useul eedback as high priorities or their own
proessional development.
REACH places demands on administrator
time and capacity
Administrators reported spending about six hours
per ormal observation cycle, including the observa-
tion, pre- and post-observation conerences, and
data management. Based on the amount o time
administrators reported spending on observations,
and the average number o observations perormed,
the ty pical elementary school administrator spent
approximately 120 hoursor two ull weekssolely
on observations that were part o the teacher evalu-
ation system. The typical high school administrator
spent approximately three ul l weeks.
Administrators are expected to train teachers about
the system, conduct classroom observations, hold
meaningul conversations with teachers about t heir
instruction, and complete required paperwork while
balancing their other job responsibilities.
This report is the irst in a series o studies
on Chicagos REACH teacher evaluation system.
Subsequent work will investigate the consistency in
observation ratings, the multiple measures o student
growth, and the relationships among these variables.
As the init iati ve continue s to unold, uture work w ill
also examine changes in these measures over time.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
5/44
Introduction
Introduction
In the all o 2012, Chicago Public Schools (CPS) instituted a sweeping
reorm o its teacher evaluation system with the introduction o REACH
Students. REACH Students replaces CPSs ormer 1970sera checklist
policy by incorporating a detailed classroom observation process and
student growth measures into teachers eectiveness scores (i.e., ormal
or summative evaluation ratings).4
With thi s pol icy, Ch icago joins other states a nd
districts across the country in developing new systems
to evaluate teacher perormance. More than 40 states
now incorporate student test scores or other achieve-
ment measures into their teacher evaluations.5 Over
the next ew years, several la rge urban districts (e.g.,
Los Angeles, Philadelphia, and New York) will be pilot-
ing or implementing similar new teacher evaluation
systems required by their states.
This report provides an initial look at the irst-year
implementation o REACH (Recognizing Educators
Adva ncing Chic ago) St udents (herea ter reerre d to as
REACH). Recent reports on teacher evaluation have
highlighted the problems that systems like Chicagos
attempt to correct, but there is still much to learn about
districts implementation experiences and their early
successes and challenges. We begin by describing the
REACH evaluation system and the speciic questions
that guided our study.
Purpose and Design o REACHRecent eorts to revamp teacher evaluation systems
relect the education ields increasing shit in ocus
rom schools to individual teachers.6 A growing number
o studies are examini ng how student learning is re-
lated to teacher eectiveness. This work shows student
achievement gains va ry signiicantly across teachers.
Furthermore, teacher eectiveness accounts or more
var iation in stud ent out comes tha n any other school
actor.7 Policymakers have responded to these research
indings: ederal policy under the U.S. Department o
Educations Race to the Top grant competition encour-
ages states to identiy strong and weak teachers by in-
corporating multiple measures o teacher perormance
in state evaluation requirements.8 Combined, develop-
ments in education research and policy have put teacher
eectiveness ront-and-center o eorts to improve
students educational outcomes.
The possibility o receiving a ederal Race to the Top
grant prompted the Illinois State Board o Education
to pursue key goals or providing students with access
to high-quality teacher and leaders, and it incentivized
Illinois legislators to pass the Perormance Evaluation
Reorm Act (PERA) in 2010. PERA requires every dis-
trict in Illinois to adopt new teacher evaluation systems
that assess both teacher practice and st udent growth. 9
The teacher practice measures required by PERA must
include multiple ormal classroom observations, as
well as suppor t or teac her i mprovement. For student
growth, the law deines various quali ying assessment
types a nd combinations o assessments that must be
used. Teacher perormance and st udent growth ratings
must then be combined to create a single, summative
rating o teacher perormance.
To comply with PERA requirements and to build
o a generally successul pilot o an ev idence-based
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
6/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
4
observation rubric (see CPSS Experiment with Teacher
Evaluation: EITP, p. 8), CPS rolled out its new teacher
evaluation systemREACHin the 2012-13 school year.
The main components o REACH in 2012-13 include:
Multiple classroom observations: Non-tenured
teachers must be observed our times per year, and
observations must last or at least 45 minutes and
include a pre- and post-observation conerence.
REACH requires administrators to provide eedback
to teachers ater each observation.
An explicit observation rubric: REACH utilizes
a modiied version o the Charlotte Danielson
Framework or Teaching.10 In this rubric teachers are
rated on our areas, or domains, o teaching practice:
Planning and Preparation, Classroom Environment,
Instruction, and Proessional Responsibilities. Each
o the domains is urther broken down into 4-5
components in which expectations or each level
o perormance are described in detail.
Trained evaluators: REACH requires all administra-
tors to be certiied by completing a series o training
modules and passing two a ssessments. It urther
employs trained specialists t hat work with adminis-
trators on calibration and assigning evidence-based
ratings aligned with the r ubric.
Student growth measures: REACH utilizes two di-
erent measures o student growth (Perormance
Tasks and either value-added or expected gains).
Althou gh R EACH is i ntended to provide a more accu-
rate measurement o teacher practice, CPS has been clear
that the system should also be a vehicle or proessional
growth. The CPS observation rubric (herea ter reerred
to as the Fra mework) provides a common language
about what constitutes eective teaching and a struc-
ture or having conversations ocused on supporting
instructional improvement (see Appendix B). Recent
research on such process-based observations systems
suggests that they can lead to improved student learn-
ing.11 Furthermore, while test score data are intended to
provide an additional measure o t eacher eectiveness,
they are a lso intended to inorm teachers choices about
appropriate instructional content or their students.
REACH implementation was a massive undertaking.
It required a large-scale investment o time and energy
rom teachers and administrators alikein the orm o
training or administrators to be certiied as observ-
ers, more requent and time-intensive observations and
conerences or both teachers and administrators, andoverall training on a new and complex system. By the
end o this year, the observation process had resulted in
over 36,000 observations or about 6,00 0 non-tenured
teachers and 13,000 tenured teachers. REACH also
required the district to create a whole new set o assess-
ments since many teachers do not teach in grade levels
or subject areas that are captured on typical standard-
ized assessments. In order to link students and teachers
to provide accurate student growth inormation, the
CPS central oice had to redesign the way data onteachers and students are collected.
TABLE 1
CPS School and Personnel Statistics (2012-13)
Schools* 578
Elementary Schools 472
High Schools 106
Non-Tenured Teachers 5,743
Tenured Teachers 15,109
Administrators** 1,195
Source: CPS Stats and Facts, Administrative records
* Does not include charter or contract schools
** Only includes principals and assistant principals
The 2012-13 school year was particularly diicult
time to launch such a large-scale and complex teacher
evaluation system: The school year began with the irst
teacher strike in more than two decades; the CEO o
CPS resigned in October, ushering in the third leader-
ship change in our years; all schools had a longer day
and year; and CPS began transitioning to the Common
Core State Standards or teaching and learning. On top
o all o this, debates about school closings, enrollment
declines, and budget shortalls began in the all. A series
o heavily attended and emotional public hearings were
held throughout the year, and a controversial decision
was made in t he spring to c lose 49 school s.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
7/44
Introduction
Guiding QuestionsThe increased attention to teacher evaluation rom
policymakers and practitioners has been accompanied
by increased attention rom researchers seeking to
evaluate implementation o these new systems. Many
studies have ocused on technical aspects, such asthe reliability o t he measurement tools.12 Another
important, but smaller, body o work has exa mined
the use o new teacher evaluation systems in schools
and districts.13 Building on this early research, this
report provides inormation on the irst year o REACH
implementation, answering questions about teachers
and administrators perceptions o the system a nd
their experiences with the new system. The speciic
questions and issues explored in this report include:
QUESTION 1: What are the benets and
drawbacks o observation systems designed
or both teacher development and evaluation?
One o the beneits o using classroom observations
in evaluation systems is that they have the potential
to meet schools dual needs o supporting proes-
sional growth and dierentiating teacher practice.14
Observations can create structures or providing
teachers with timely and individualized eedback on
their classroom practice. This inormation can guide
coaching and proessional development activities, as
well as help t eachers de velop goals or improvement. I n
addition, observation ratings provide adm inistrators
with st and ard ized and deen sible evidence or ma king
personnel decisions.
Yet, using observations or both purposes may also
create a number o tensions. One study suggests that
some teachers may be less likely to seek instructional
support rom administrators i exposing their weak-
nesses could result in a poor evaluation.15 Furthermore,
teachers may not respond positively to encouragement
rom administrators ater receiving low ratings or
disciplinary actions rom them.16 Finally, classroom
evaluators who are responsible or supporting teacher
growth and ormally assessing eectiveness may intro-
duce bias into the accountability process.17 In short, i
not implemented well, the beneits o using classroom
observations may devolve into dueling purposes with
each cancelling the beneits o the other.
UChicago CCSRs study o CPSs earlier pilot pro-
gram, which was ca lled the Excellence in Teaching
Pilot, ound that teachers and principals thought their
discussions about instruction were more relective and
objective using the Danielson Framework than the CPS
checklist.18
Observations conducted under the pilot,however, did not count toward teachers oicial evalu-
ation score. In Chapter 1, we ask: How air and useul
do teachers and administrators ind R EACH classroom
observations as a means o improving instruction?
Does using school administrators as both coaches
and evaluators raise any concerns or challenges?
QUESTION 2: How do teachers view the use o
student growth on standardized assessments
in their evaluation?
The incorporation o student growth measures into
teachers evaluations has been a contentious issue,
both in Chicago a nd nationally. While supporters
maintain teachers should be held accountable or stu-
dent learning, critics contend that metrics designed to
assess student progress are poor measures o teacher
perormance.19 Additionally, opponents ear that
adding stakes to student assessments increases the
likelihood that teachers will nar row their curriculum or
teach to the test so as to avoid a negative evaluation.
Despite these issues, states and districts have moved
orward with including student growth measures in
teachers evaluations.
Addre ssin g teacher and admini str ator skept icism o
student growth measures is critical or leveraging the
ull potential o the system to improve instruction. In
Chapter 2, we ask: To what extent do teachers perceive
student growth measures can provide an accurate as-
sessment o their perormance? How, i at all, are teach-
ers using the assessment data produced by REACH?
QUESTION 3: What are the successes and chal-
lenges related to training and communication?
Wh ile RE ACH add resses many o the lim itations o t he
previous teacher checklist system, thoughtul design is
not enough to guara ntee success.20 Beore REACH can
improve teacher evaluation or instructional practice,
it irst has to move rom a written policy document to
a system embedded in the work o teachers and school
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
8/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
6
administrators. Implementation is critical to achieving
intended outcomes.21
The task o implementing a teacher evaluation sys-
tem o this scale and complexity should not be under-
estimated. REACH involves over 20,000 teachers and
other school sta and 1,200 admin istrators in nearly600 schools. Principals and assistant principals had to
be certiied and trained on using the new observation
rubric. Teachers had to be inormed about the goals o
the new system, trained on how to engage in the new
observation process, and taught how their summative
evaluation score would be calculated. Observations and
the pre- and post-observation conerences had to be
scheduled and completed.
Understanding the experiences o teachers and
administrators as they implemented such a complexand time-intensive systemin addition to all their
other responsibilitiesis a critical irst step toward
understanding any potential eects that REACH
might have. I teachers and administrators are not
inormed o REACHs goals and do not understand its
var ious element s, they m ay not implement the poli cy
as intended. Insuicient training and resources are
reasons or implementation ailure. 22 In Chapter 3,
we a sk: How knowledgeable w ere t eachers a nd ad min-
istrators about REACH? How did they describe their
training ex periences? What aspects o implementation
did participants identiy as needing improvement?
QUESTION 4: How do principals understand and
describe their capacity to manage classroom
observation workloads?
In the la st decade, principals have been increasingly
called upon to be instr uctional leaders in their schools,
especially through supporting eective instructional
practices. 23 Given this emphasis on principals as in-
structional leaders, many assume that it is the principal
who s hould be r esponsible or conduc ting obs erv ation s
and evaluating teacher practice.
It is not clear, however, whether principals have the
time and capacity to manage the observation workload
created by new evaluation systems. To increase the
reliability o ratings, most systems call or teachers to
be observed multiple times a year. Each observation
typically involves scheduling and conducting the obser-vation, w riti ng up evidence a nd enteri ng it into a dat a-
base, having pre- and post-observation discussions with
teachers, and coaching teachers on areas or improve-
ment. The entire process or a single teacher can take
several hours. While assistant principals in CPS also
became certiied evaluators, it still ell to the principals
to ensure that all o the observations and pre- and post-
conerences required by REACH were conducted.
Previous studies conducted by UChicago CCSR
researchers have highlighted some o the capacity is-sues created by the introduction o new teacher
evaluation systems. For example, workload demands
contributed to lower engagement in the new system
or some principals, while others reported giving less
attention to tenured teachers in order to complete all
o their required eva luations.24 In Chapter 4, we ask:
How much time did administrators spend on classroom
observations during the irst year o REACH? How do
they eel about the demands the new REACH system
places on them?
In This ReportChapters 1 and 2 o this report describe the observation
and student growth elements o REACH and pro-
vide par ticipa nts perce ptions ab out t he va lue o thi s
initiative as both an evaluation and development tool.
Chapters 3 and 4 describe participants experiences
with implement ation, oc using on commu nication,
training, and time demands. Final ly, in Chapter 5, we
present some questions to consider as implementation
continues. Additional reports in th is series will inves-
tigate observation ratings, student growth ratings, and
the relationship between them.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
9/44
Introduction
What Goes Into a Teachers Evaluation Score?
A teachers REACH summative evaluation score is
comprised o a teacher practice score and up to two
measures o student growth. The teacher practice
component consists o classroom observations com-
pleted by a certied administrator utilizing the CPS
Framework or Teaching, a modied version o the
Danielson Framework or Teaching. Student growth
measures are discussed in detail in Chapter 2. In
2012-13 only non-tenured teachers were to receive a
summative evaluation score.
Teacher Practice: CPS Framework for Teaching
Student Growth: REACH Performance Tasks
Student Growth: Value-Added
75%
10%
15%
Elementary Teachers in Tested Subjects/Grades
(Receive individual value-added)
75%
15%
10%
Elementary Teachers in Untested Subjects/Grades
(Receive schoolwide value-added in literacy)
Source: Chicago Public Schools
90%
10%
High School Teachers in Core Subject Areas
100%
High School Teachers in Non-Core Subject Areas
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
10/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
8
Research Activities
To answer our questions we used multiple sources o inormation, including surveys and interviews. Surveys pro-
vide a broad picture o participants perceptions; interviews provide deeper insights into participants experiences.
Winter 2012-13 Surveys: We surveyed all 1,195 prin-
cipals and assistant principals in December 2012,
receiving 733 responses (a 61 percent response
rate). We surveyed a random sample o 1,000
non-tenured teachers and 1,000 tenured teachers
in January 2013.A We received 901 responses (a 45
percent response rate). The entire content o this
survey administration was related to REACH.
Spring 2013 Surveys: We included survey items as
part o CPSs annual My Voice, My School survey.
This survey was administered to all teachers in
March 2013 and had a response rate o 81 percent.
Then, we surveyed all principals and assistant
principals in April/May 2013, receiving 687 respons-es (a 57 percent response rate). Some questions
were the same as in the winter 2012 survey to
gauge changes in perception; others were dierent
because the initiative was more mature. Survey
content was shared with other topics.
Spring 2013 Principal and Teacher Interviews:
We randomly selected three high schools and
ve elementary schools or our interview sample.
We then randomly selected teachers rom within
those schools to interview. We were able to
interview six principals and 31 classroom teachers
rom six schools.
Fall 2012 CPS Central Oce Staf
(Instructional Efectiveness Specialists)
Interviews: We interviewed nine specialists
(about hal o the sta in this position) in
November 2012. These specialists werecharged with providing technical assistance
to administrators in conducting classroom
observations.
CPSs Experiment with Teacher Evaluation: EITP
Between 2008 and 2010 CPS implemented the
Excellence in Teaching Pilot (EITP), a pilot teacher
evaluation program that used the Charlotte Danielson
Framework or Teaching to guide the evaluation o
classroom instruction. EITP provided an alternative
system to the teacher evaluation checklist CPS had
used or 30 years. Over the two-year period, a total
o 100 elementary schools participated in the pilot.
CPSs current REACH evaluation system
resembles EITP in many ways. Like the pilot, trained
administrator evaluators observe teachers classroom
instruction using a modied version o the Danielson
Framework. Some observations are unannounced
and others are planned in advance and include a pre-
and post-observation conerence. The new system,
however, diers rom EITP on several importantdimensions. First, although the pilot had no stakes
attached, REACH is the ocial evaluation system or
non-tenured teachers in its rst year, and will expand
to include stakes or all teachers over time. Second,
while administrator training or the smaller-scale pilot
was done in-person, training or the new system was
provided statewide via an online platorm. And nally,
the pilot provided measures o perormance based
only on observations, while the new system includes a
student growth component or all teachers regardless
o the grade or subject they teach.
UChicago CCSRs two-year study o the pilot ound
most principals and teachers were supportive o EITP
and ound it benecial or their practice. Specically,
principals and teachers reported using the Danielson
Framework and evidence rom classroom observations
made their conversations about instruction more
objective and refective. In addition, the study ound
principals ratings o teachers were both valid and
reliable.B The pilot also uncovered some challenges.
For example, many principals lacked the instructional
coaching skills required to have deep discussions about
teaching practice. Where principals were less procient
at conerencing with teachers, teachers were less posi-tive about the new system and more doubtul o their
principals ability to use the Framework accurately or
rate them airly. A later ollow-up study o EITP ound
that the pilot had a positive eect on both reading
and math scores. Higher-achieving schools and those
serving ewer low-income students were the primary
beneciaries.C
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
11/44
Chapter 1 | The Classroom Observation Process
The Classroom ObservationProcess
CHAPTER 1
The main element o the REACH evaluation system is
the observation process used to rate teacher practice.
The process is centered around the CPS Framework or
Teaching (the Framework), a classroom observation
rubric based on the Cha rlotte Danielson Framework
(see Appendix B). RE ACH also est abli shes a set o pro-
cedures or how evaluators should conduct classroom
observations, collect evidence about what was observed,
and discuss the evidence and ratings with teachers
(see Table 2).The teacher practice component is intended to serve
two unctions. Drawing on its roots in the Danielson
Framework, the classroom observation process is
structured to provide teachers with in ormation they
can use to improve their teaching practices. It includes
a pre- and post-observation conerence to create a
orum or evaluators to provide constructive eedback
to teachers on their practice and oer support or
improvement. In addition, the teacher practice compo-
nent is intended to provide school administrators with
a means to evaluate the instructional eectiveness o
TABLE 2
What does the formal observation process include in 2012-13?
Source: Modified from REACH Students Teacher Practice
Note: In 2012-13 administrators were required to conduct at least four formal observations for each non-tenured teacher and at least one formal observation for each
tenured teacher.
Pre-Observation Conference
A brief 15-20 minute
conference with a focus
on Domain 1 (Planning
and Preparation)
The teacher and
administrator decide
which lesson will be
evaluated
Post-Observation Conference
The teacher and
administrator discuss the
classroom observation
The teacher's self-reflection
is evaluated for Component
4A (Reflecting on Teaching
and Learning)
Ends with suggestions for
improving teacher practice
Classroom Observation
The administrator observes
teacher for about 45 minutes
Observation primarily
focuses on the components
in Domain 2 (Classroom
Environment) and Domain 3
(Instruction)
The administrator gathers
evidence and assigns ratings
teachers in their building. Ratings across classroom
observations are combined with test score gains to
give each teacher an o icial evaluation score.
In this chapter we examine teachers and adminis-
trators perceptions o the teacher practice component.
Our indings draw on both sur vey and interview data.
Survey data show the extent to which teachers and
principals across the district have positive or negative
view s ab out t he ob ser vations pr oces s. O vera ll, both
groups ind the process to be a useul means o helpingteachers improve their instructional practice. Teachers
appreciate the eedback they receive rom their evalu-
ators and believe the rating process is transparent.
Admini str ators thi nk the observation proc ess wi ll lead
to improvements in teaching and st udent learning.
Interview data provide insight into how the observa-
tion process supports instructional improvement. In
addition, it highlights teachers descriptions o how the
coordination o the evaluation process can undermine
the value o t he observations as an improvement tool.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
12/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
10
The Observation ProcessSupports Proessional GrowthAdmini str ator s and te acher s ex press ed positive v iews
o the teacher practice components potential to sup-
port teacher growth and proessional development. On
the survey, 76 percent o teachers said the evaluationprocess at their school encourages their proessional
growth. Similarly, 76 percent o admin istrators
reported believing that the observation process would
result in instructional improvement at their school,
and 82 percent reported noticeable improvements in
hal or more o the teachers they had observed over
the school year (see Figure 1).
In interviews, teachers identiied three ways in
whic h the obser vation comp onent suppor ts t eacher
learning. First, they remarked that the Framework
rubric sets clear expectations about quality instruc-
tion. As one teacher succinctly put it: The observation
rubric describes what really good teaching looks like. It
gives me a clear descr iption o what teaching look s like
at each level.Responses on surveys indicate that ma ny
teachers and administrators agree w ith this sentiment:
75 percent o teachers and 91 percent o administra-
tors reported that the Framework provides a common
deinition o high-quality and eective teaching. Clear
descriptions o quality instr uction help teachers tran-
scend their own individual opinions about teaching and
begin to compare their practice to others. One teacherexplained:
You get into your own practices and orm
your habits and methods. But because
everyone is working within the REACH
system, you can start to see where you are
in the system. Everyone is breathing the
system language. I they all are refecting
the same language, you have to think about
others and all the other teachers.
Because it creates explicit and shared expectations
o quality inst ruction, teachers and administrators
commented that the rubric also provides clear g uidance
about what teachers need to address in order to improve
their practice:
I always thought there needed to be higher
standards in teaching, and I think the
observation rubric has made the standards
higher. [Beore] it was up to the principals
discretion o how he or she elt. [Now] its
clearer about what that means, how to grow,
how to improve. CPS Teacher
[In post-conerences] instead o just saying,
You got a 3 here and a 2 here, we can say,
What is the dierence between a basic and
procient [rating]? I didnt see this, I didnt
see this. And it was a really clear thing,
Start doing that, or, Stop doing something
else. CPS Principal
Admini str ator s were vir tua lly una nimous on th is
point: 96 percent responded on the winter survey that
the Framework helps them identiy areas where teach-
ers can improve.
Second, teachers reported that the teacher prac-
tice component has potential to improve instruction
Pe
rcentofAdministrators
70
60
50
100
90
80
40
20
10
0
30
Have incorporated
your feedback into
their teaching?
(n=622)
Have made noticeable
improvments over
this year?
(n=621)
Source: Spring Administrator Survey, May 2013
Note: Percentages may not add up to 100 due to rounding.
All Most About Half A Few or None
FIGURE 1
Most administrators report at least half of their
teachers have incorporated feedback and improved
Of the teachers you have observed this year, how many...
10%
63%
19%
8%
5%
48%
29%
17%
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
13/44
Chapter 1 | The Classroom Observation Process
because it creates opportunities to discuss teaching
with ad min istrators a nd colleagu es. I n pa rticular, t he
pre- and post-conerences were a way o getting needed
eedback and support:
I love being able to rene what I do andtalk about it with somebody. So the idea
is that I get to sit down every month or so
and say this isnt really working or me and
my administrator will nd something that
can help me. That is really benecial.
CPS Teacher
I think that it is nice to have someone in
the classroom requently to really see
how youre doing and what youre doing
and give you eedback in a way that is not
really an attack. Its more like a positive,
constructive criticism on dierent aspects
o teaching. CPS Teacher
These comments highlight that teachers value
eedback on their instruction. They also show
that conversations with administrators tend to be
respectul a nd supportive. In act, only 6 percent
o teachers on the winter survey said eedback was
delivered in a hurt ul manner. Across the district, 82
percent o teachers indicated they have proessional
conversations with their administrators ocused
on instruction, 89 percent said their evaluator
supports their growth, and 76 percent reported
that their evaluators eedback was useul. Among
administrators, 94 percent thought the Framework
has improved the quality o their conversations with
teachers about instruction.
Finally, teachers also noted the conversations
helped them intentionally re lect on their own class-
room practice. I think its good to see what you did
and how you can improve,one teacher said. I cant
see mysel teach, and I love to hear how I can improve.
By creating opportunities to examine their own prac-
tice, the observation process helps teachers identiy
their strengths and weaknesses, as well as prioritize
areas on which to ocus their improvement eorts.
For some, the relection habit carries outside o the
ormal observation structure to their teaching more
generally: [The Framework] causes us to be more c on-
scious o our planning and the words coming out o our
mouth. It causes us to really look at what we are doing in
our classrooms.
Admini str ator s ag reed: 92 perc ent o principal andassistant principal survey respondents thought the
Framework encourages teachers in their school to
relect on their instructional practice (see Figure 2).
On the winter survey, 81 percent o teachers said it helps
them identiy a reas where their teaching is strong, and
82 percent said it helps them identiy areas where they
can improve.
Most Teachers Believe Administrator
Ratings were Accurate and Fair
Teachers were generally positive about the accuracy
o the ratings they received rom school administra-
tors. On the spring survey, 87 percent o teachers said
their evaluator was air and unbiased, and 88 percent
said they were able to assess their instruction accu-
rately (see Figure 3). On the winter sur vey, 72 percent
o teachers said their ratings were about the same or
higher than they thought they should have been.
One reason teachers were positive about their
ratings is they believe the speciicity o the Framework
helps makes ratings more concrete:
I like that they [the Framework] actually
speciy what it is that we are being
evaluated on, versus the old system where
your principal essentially gave you a rating
and some comments about what youve
been doing. With this [system], youre either
doing this or youre not. I youre not, then
youre not meeting [the standards]. I you
are, then youre procient.
Anot her re ason teachers eel rati ngs tend to b e
generally objective is because administrators have to
collect and present evidence about what they specii-
cally saw during observations. [The Framework] holds
a lot o accountability,one teacher said. Not only or
the teachers but also or the administrator; they have to
prove e verything t hey ve ound.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
14/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
12
FIGURE 2
Nearly all administrators report the CPS Framework is useful for instructional improvement
The CPS Framework...
PercentofAdministrators
2%
5%
53%
40%
70
60
50
100
90
80
40
20
10
0
30
Is a useful tool for
identifying teachereectivenessin this school
(n=622)
Is a useful tool for
providing targetedsupport for teachers
(n=623)
Encourages
teachers in thisschool to reflect ontheir instructional
practice
(n=622)
Has provided a
definition ofeective teaching
in this school
(n=623)
Has improved the
quality of myconversations with
teachers in this
school about
instruction
(n=620)
2%
5%
50%
44%
2%
7%
58%
33%
2%
6%
54%
38%
2%
5%
54%
38%
Strongly Agree Agree Disagree Strongly Disagree
Source: Spring Administrator Survey, May 2013
Note: Percentages may not add up to 100 due to rounding.
FIGURE 3
Most teachers believe their evaluator has the capacity to assess instruction fairly and accurately
My evaluator...
PercentofTeachers
5%
8%
24%
63%
70
60
50
100
90
80
40
20
10
0
30
Is fair andunbiased
(n=13,600)
Is able toaccurately assess
teachers instruction
(n=13,622)
Knows my strengthsand weaknesses
(n=13,624)
Knows whatsgoing on in my
classsroom
(n=13,627)
6%
13%
30%
51%
5%
10%
30%
54%
3%
9%
29%
59%
54%
To a Great Extent Some A Little Not At All
Source: Spring Teacher Survey MVMS, April 2013
Note: Percentages may not add up to 100 due to rounding.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
15/44
Chapter 1 | The Classroom Observation Process
While havi ng positive p erceptions o their rating,
some teachers thought using multiple raters could
improve the reliability o the ratings and how they are
used in personnel decisions. One concern is that being
observed by only one evaluator may lead to inaccurate
ratings. Recent research seems to validate this view,inding that multiple observers produce more reliable
ratings. 25I think it would be s omething to think about
having each observation done by a second person,one
teacher stated. Yes, ratings are evidence based, but
[evaluators] do ma ke interpretations based on previous
knowledge o you. So I think that ties into how they view
what you are doing and i you are doing it well.
In addition, some teachers worry administrators
may use observation ratings to remove or deny tenure
to sta they do not like. Thus, the ratings one receivesmay have little to do with what actually happens dur-
ing an observation. I just eel like it shouldnt be the
administration who is doing the observation on you,one
teacher said. Because the bottom line is that i they dont
want to keep you in the school, they are not going to. They
are going to give you a bad observation rating.Some
cities have addressed this problem by incorporating
governance structures that support personnel systems.
These systems ty pically include the use o expert men-
tor teachers as evaluators and coaches, as well as review
structures or personnel decisions that involve teachers
and administrators in the decision-making process.26
Administrators Dual RoleCan Undermine ProessionalLearning BenetsREACHs reliance on administrators both to ocially
evaluate teacher practice and to provide instructional
coaching may undermine the learning potential o the
observation process. Because observation ratings have
such a big impact on summative evaluation scores, teach-
ers are highly motivated to demonstrate their proessional
competence when they are observed. Since the admin-
istrator giving ocial ratings is simultaneously provid-
ing instructional support, however, teachers are orced
to weigh the costs and benets o using the observation
process as an opportunity to share their instructional
weaknesses. For some teachers, the risk o receiving a
poor rating is too great. As one teacher explained:
Because there is such an emphasis placed
on assessing the quality o teachers, there is
no incentive or teachers to admit insecurity
or talk about areas in which he or she strug-
gles. I elt like I had to mask the things that I
didnt do as well and try to explain why theydidnt go well because, at the end o the day,
Im being rated. So there is more o an incen-
tive to present mysel avorably than to have
an honest discussion about instruction.
Several teachers across our interviews schools
described instances when they perceived that attempts
to get support or addressing weaknesses led to nega-
tive consequences on their evaluation. For example, one
teacher said he was very honest at the start o the yearabout his practice by highlighting or his evaluator things
in my daily teaching that I need to strive to x.Ater being
inormally warned that his evaluator would pay more
attention to those areas, he elt as though the eva luator:
ended up putting a laser ocus on the
things that I do want to x, but are hard to
x. Instead o being rewarded or being
sel-aware and honest about improvements,
I eel like I m actually being penalized.
Anot her teacher recount ed goi ng to her pr incipa l
or help regarding classroom management issues in
one class. Instead o receiving support, she elt the
request led to her receiving a low observation rating:
My principal joked around and said hell do my
next observation in that classroom [in which I
was struggling]. It was a joke, and then he
actually did it. When I said I really dont know
i it is appropriate or me to be judged based
on that classroom, when there are so many
other classrooms and grades that I teach
where Ive already been observed, I was scold-
ed and told that I am essentially saying that I
cant do my jobIt makes me eel like I cant
even come to my own administrator or help,
because that inormation was essentially used
against me in the observation process.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
16/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
14
These two comments highlight the potential risks
involved in asking an evaluator or support. I teach-
ers present a realistic view o their teaching, they may
be rated as less skilled compared to others who put on
a perormanceduring a scheduled observation. It is
important to keep in mind that we do not have evidence
about how widespread instances like the ones above
are. Nonetheless, these cautions show how the learning
opportunities created by the observation process could,
in the long run, be undermined when the evaluators
giving ratings are also primary instructional coaches.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
17/44
Chapter 2 | The Use o Student Growth in Evaluation
The Use o Student Growthin Evaluation
CHAPTER 2
Prior to REACH, teachers in Chicago were not held or-
mally accountable or the perorma nce o their students.
The use o student growth to measu re teacher peror-
mance breaks new ground. A teachers student growth
score summarizes t he change in his or her students
standardized test scores between two time periods:
under REACH, the beginning and end o a school year.27
PERA requires that student growth be a signicant
actor in teachers evaluations, though CPS and the
CTU agreed to phase in t his requirement so that studentgrowth accounted or no more tha n 25 percent o a teach-
ers evaluation score in the initial yea r. The weight given
to student growth in a teachers nal evaluation varies
according to the subject and grade level o the teacher
(see What Goes Into a Teachers Evaluation Score?on
p. 7). Student growth is ca lculated di erently depending
on the assessment that is used. These assessments also
var y by g rade and subject , but they can i nclude:
A gain score on district-developed Perormance
Tasks, which are w ritten or hands-on assessments
speciically designed or the grade and subject o the
course and are most oten scored by the teacher
A value-added score on the NWEA MAP, an adaptive,
computer-based test administered to students in
grades 3-8 in reading and math
An expected gains score on the subject area
EXPLORE, PLAN, or ACT ( EPAS) a ssessments
administered to students in grades 9-11 in English,
reading, math, and science28
A measure o average schoolwide literacy growth
rom either the NW EA MAP or the EPAS
The Student Growth Component o REACH box
on page 20 urther describes the measures o st udent
growth used in RE ACH.
As s et orth by PE RA , st udent g row th i s inc orpo -
rated strictly or evaluation purposes. However, CPS
has been clear that they expect REACH to positively
aect teacher development and student learning. I the
student growth component is to be useul beyond teach-
ers evaluations, it must provide teachers with inorma-
tion that can inorm their instruction. A student growth
score alone does not provide teachers with inormation
that is timely or detailed enough to guide improve-
ments in their instructional practice; it is one number
that summarizes changes in test scores across a group
o students over a given period o time. In contrast,
students perormance on the individual assessmentsused to calculate student growth might inorm teach-
ers instruction by providing them with inormation on
their students skills or level o understanding.
In this chapter, we describe teachers responses to
the use o student growth in t heir evaluations, as well a s
how useul teachers ound the assessments or their in-
struction. We ind apprehension among teachers about
the incorporation o student growth metrics into their
evaluation. Teachers were generally positive about the
potential instructional value o the assessments used
to measure st udent growth, though the perceived use-
ulness varied considerably by the assessment.
Teachers Are ApprehensiveAbout the Use o StudentGrowth in Their EvaluationGiven that student growth is a new addition to teach-
ers evaluation, it is not surprising that many teachers
expressed concerns over its use in measuring teacher
perormance and in personnel decisions, or that many
were misi nor med or con us ed about how student
growth actors into their evaluation. Additionally,
some teachers raised concerns about the potential
or bias when applying the student growth measures
across dierent classroom contexts.
On our survey, 57 percent o teachers said that they
believe or strongly believe that the REACH system
relies too heavily on standardized tests (see Figure 4).
Anot her 30 percent sa id th at they somew hat b elieve
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
18/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
16
this, while only 13 percent o teachers said that they do
not believe that REACH relies too heavily on standard-
ized tests. We asked teachers an open-ended question
about what they ound most problematic about the
REACH system. Nearly one-third o the 552 teachers
who r esponded t o this quest ion ident ii ed the studentgrowth component and the assessments used to mea-
sure student growth, making t hese the most requently
cited problematic aspects o REACH. 29 While some o
these teachers maintained that test scores should never
be used in teachers evaluations, others identiied more
speciic concerns. These concerns included the narrow
representation o student learning that is measured by
standardized tests, t he numerous inluences on student
perormance that are outside o a teachers control,
and an increase in t he already heavy testing burden onteachers and students.30
Teachers responses to our interview and an open-
ended survey item revealed that many o them were
misinormed or unclear on how much student growth
contributes to their summative evaluation. For exam-
ple, one teacher wrote, I am concerned about my eort
as a teacher completely relying on the test scores o my
students.In act, student growth does not account or
more than 25 percent o any teachers evaluation this
year, and student growth w ill not account or more than
30 percent once REACH is ully implemented; there-
ore, no teachers evaluation will completely rely on test
scores. Some teachers urther att ributed the incorpora-tion o student growth to the district, rather than to the
state law. As we show in the next chapter, most teachers
reported receiving inormation about REACH rom
their school administration. Yet only 45 percent o
principals a nd assistant principals reported having a
strong or very strong understanding o how student
growth actors into a teachers summative rating, so
it is not surprising that teachers are also unclear.
Several teachers expressed concerns that measures
o student growth are unair to teachers in more chal-lenging schools because student growth, and thereore
a teachers evaluation score, is related to the supports
that students may or may not receive outside o the
classroom. One teacher explained this concern:
Im not going to want to work in a [strug-
gling] school i my evaluation is tied to test
scores, because there are things that I cant
control. I cant stop gang violence. I cant
stop poverty. I cant stop the parents who
dont care i their kids go to school. I think
the part that I nd unair is that so much o
what goes on in these kids lives is aecting
their academics, and those are things that a
teacher cannot possibly control.
Related to the issue o airness, many teachers
expressed apprehension over how the student growth
measures would be used by the district in part icular
that they would be used to ire teachers or to institute
merit pay. For example, one teacher explained that she
had grave concernsthat her students perormance
could negatively impact her job security, in part because
there are so many other actors outside o the classroom
that inluence student growth.
Two groups o teachersspecial education
teachers and non-core subject teacherswere
particularly critical o the student growth component.
Special education teachers raised concerns that
PercentofTeachers
70
60
50
100
90
80
40
20
10
0
30
Strongly Believe Believe
Somewhat Believe Do Not Believe
(n=731)
13%
30%
24%
33%
Source: Winter Teacher Survey, January 2013
Note: Percentages may not add up to 100 due to rounding.
FIGURE 4
Most teachers believe or strongly believe REACH
relies too heavily on standardized tests
Please indicate the extent to which you believe that
REACH, overall, relies too heavily on standardized tests.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
19/44
Chapter 2 | The Use o Student Growth in Evaluation
the REACH Perormance Tasks, NWE A MAP, and
EPAS assessments were inappropriate measures o
their instruction and o their students learning. One
special education teacher explained: The grade level
REACH Pe rorma nce Tasks we re nearly impossible or
my special education students, and it will be diicult toshow improvement or many students who are our and
ive grade level s behind.This teachers concern was
echoed by many special education teachers who believed
that holding their studentsand, thereore, their own
evaluationto the same standard as regular education
students and teachers was unair. Many teachers were
unclear on what accommodations could be provided
or their special education students as they took the
assessments, which were the same assessments that
were g iven to reg ula r educat ion st udents.
31
Some non-core subject teachers (e.g., art, music, and
physical education) were troubled by the incorporation
o schoolwide literacy growth into their eva luation.
These teachers disliked being held accountable or the
work o other teachers a nd or a cont ent a rea that they
were not ne cess ar ily prepa red t o te ach. For exa mple, a
high school art teacher explained his eelings about the
schoolwide literacy measure:
The comment has been made that I will get
judged on reading scores because we are all
teachers o literacy, and theres a part o me
that agrees with that. But...there is no part o
my certication or training that says I need
to learn how to teach a student how to read.
Teachers Found Beginning-o-YearREACH Perormance Tasks UseulThe REACH Perormance Tasks were developed by
teachers and district specialists as Type III assessments
(see The Student Growth Component of REACHon
p. 20). As dened by PER A, Type III assessments ar e
rigorous, aligned to the courses curriculum, and mea-
sure student learning in that course. Perormance Tasks
are a written or hands-on demonstration o mastery,
or progress towards mastery, o a particular skill or
standard,32 which make them very dierent rom
traditional multiple-choice assessments. The primar y
purpose o the RE ACH Perormance Tasks is to provide
a measure o student understanding at the beginning
and end o the school year so that a growth score can
be calculated and incorporated into teachers evalua-
tions. In the best ca se, however, the beginn ing-o-year
Perormance Tasks would also provide teachers with
inormation that is useul or their instruction, suchas inormation about their students skills or about
the districts expect ations or what content should be
covered in their class.
Amon g teacher s who admini ster ed a beg inn ing- o-
year REACH Perormance Task, 70 percent reported
that it was somewhat or very useul or their instruction
(see Figure 5). In inter view s, teachers r epor ted using
the Perormance Tasks as an indication o what mate-
rial they needed to cover. Moreover, teachers seemed
to appreciate the more comprehensive set o skills thatstudents could demonstrate on the Perormance Tasks:
72 percent agreed that the tasks provided inormation
that is not measured on traditional multiple-choice
assessments.
Wh ile the Peror mance Ta sks prov ided t eachers
with in sight into what materia l they ne eded to c over,
ew teachers used the Perormance Tasks as measures
o student understanding. Two-thirds o teachers
(67 percent) agreed that the Perormance Tasks were
rigorous assessments o st udent learning, but they may
have been too rigorous; nearly the same proportion
(66 percent) indicated that the tasks were too chal-
lenging or beginning-o-year assessments. Because
the Perormance Tasks oten covered material that
the students had not yet been exposed to, they did not
provide a measure o students understanding o that
material. Rather than test students prior knowledge,
the Perormance Tasks assessed students on content
that they had not been taught. One teacher explained
why t he le vel o cha llenge is par ticu lar ly a problem or
a beg inning-o-year assessment:
[my students] were really upset by it. Not
only because it was something they had
never seen beore, but they didnt know
me, so it was kind o like they didnt know
me and I was giving them something and
challenging them in a way that it was unair
or them and it made me eel really bad.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
20/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
18
This teacher explained that once she has gotten to
know her students, she can motivate them to persevere
through a chal lenging assessment. But without the time
to build those relationships, it was diicult or her to
help her students overcome their rustration.
There was airly widespread conusion amongteachers about the administration o the Perormance
Tasks. Just 41 percent o teachers who had adminis-
tered a beg inning-o-year Perormance Task indicated
that they were clear on how the tasks should be scored.
Over one-third (36 percent) o teachers indicated they
did not have adequate time to score the tasks, and
one-third (35 percent) indicated they had diiculty
recording the scores on the districts internal site. Just
under hal (43 percent) o teachers indicated that they
were not at all cle ar on what ac commo dations c ouldbe made or students with IEPs who were taking a
Perormance Task.
Apart rom t heir instruc tional value, severa l
teachers raised concerns about how easy it would be to
game the scoring o the Perormance Tasks. Teachers
score their own students Perormance Tasks at both
the beginning and end o the year. In an interview and
on an open-ended survey item, teachers noted that i
they wanted to max imize their student growth score,
they could simply give all students a low score on the
beginning-o-year task and a higher score at the end o
the year. While we have no measures o how requent-
lyi at allthis practice occurred, it has the potential
to undermine how teachers and administrators per-
ceive the accuracy o the eva luation ratings.
NWEA MAP Provided Timelyand Useul DataThe NWE A MAP is a series o computer-based,
adaptive assessments administered to CPS students
in grades 3-8 at the beginning and end o the school
year.33 Seventy-eight percent o teachers who admin-
istered a beginning-o-year NW EA MAP assessment
ound it somewhat or very useul or their instruction
(see Figure 5) and a simila r prop ortion (75 percent)
agreed that the NW EA MAP helped them to target
their instruction to meet students individual needs.
PercentofTeachers
70
60
50
100
90
80
40
20
10
0
30
Performance
Tasks
(n=13,253)
EPAS
(n=3,225)
NWEA MAP
(n=6,316)
Source: Spring Teacher Survey MVMS, April 2013
Note: Responses only include teachers who conducted the assessment(s) in the
fall of 2012. Only high school teachers in core subjects administered EPAS. Only
elementary school teachers in grades 3-8 reading and math administered NWEA
MAP. All elementary school teachers and a subset of high school teachers in core
subjects administered Performance Tasks. Percentages may not add up to 100
due to rounding.
Very Useful Somewhat Useful
A Little Useful Not Useful
FIGURE 5
Teachers report assessments vary in instructional
usefulness
How useful is the following for your instruction?
14%
17%
32%
38%
8%
14%
34%
44%
22%
29%
34%
16%
The rigor o the NWEA MA P assessments and the
timeliness with which teachers receive their students
results may help to explain the NWE A MAPs instruc-
tional value. Eighty-ive percent o teachers who had
administered the beginning-o-year NW EA MAP
ound it to be a rigorous assessment o student learning.
Addit ionally, t he major ity o teachers a greed t hat the
results provided by the NW EA MAP are both timely (92
percent) and easy to understand and use (72 percent).
Wh ile the computeriz ed natu re o the NW EA M AP
assessments likely contributed to their instr uctional
useulness, it also created problems or some teach-
ers. Over two-thirds o teachers who had administered
the beginning-o-year NWE A MAP reported that they
experienced technical diiculties, such as issues with
computer hardware or internet access. One teacher
explained how technical problems can aect student
perormance:
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
21/44
Chapter 2 | The Use o Student Growth in Evaluation
At our school, our technology isnt up-to-
date. The computers themselves are about
nine or 10 years old.When everybody was
taking the test at once, that was an issue
because our routers couldnt handle the
amount o trac. So the internet would goout. I think that really skewed our test results
because students, especially on reading,
would have to read the story and then go
back to it and then they were stuck and they
would have to go back. The students wont
reread what they read, so they might orget
a part, and then theyre asked questions.
How to use the NWE A MAP with particular popula-
tions o students was again a concern or teachers: 30percent were not at all clear on what accommodations
were acceptable or special edu cation st udents and 39
percent were not at all clear on whether ELL students
should take the NWE A MAP.
EPAS Results Not Timely or DetailedHigh school teachers were less positive about the value
o the beginning-o-year EPAS assessments or their
instruction than elementary teachers were about the
NWE A MAP a ssessments. EPAS assessments are given
in grades 9-11, as par t o ACTs testing system. While 71
percent o teachers who had administered a beginning-
o-year EPAS assessment agreed that the test was a rig-
orous assessment o student learning, only 50 percent
o those teachers reported that it was somewhat useul
or very useul or their instruction (see Figure 5).
One issue limiting the instructional value o the
beginning-o-year EPAS assessments is the timeliness
with wh ich teachers r ecei ve their resu lts. Unli ke t he
computer-based NWE A MAP that provides teachers
with result s im medi ately ollowi ng the a ssessment ,
teachers do not receive their students EPAS scores or
several months. Just 50 percent o teachers who admin-
istered a beginning-o-year EPAS assessment indicated
that they had received their students results in a timely
manner. In our interviews, we heard why this delay is
problematic or teachers:
We didnt get the results back until basically
almost January, so its kind o like the data
is dead...[it] refected what they knew three
months ago. I I had gotten an item analysis,
that would have been more helpul. But I just
got a raw score so I know that they scored a14.I know I want to improve that score, but
I dont know why theyre getting that score.
Moreover, as the teacher above explained, the results
o the EPAS exams are not detailed enough to guide
teachers instruction.34 Teachers receive their students
subject scores, but not an item analysis. Just 44 percent
o teachers indicated that the beginning-o-year EPAS
helped them to pinpoint individual students strengths
and weaknesses.Since high schools had administered EPAS as paper
and pencil exams or a number o years beore the
implementation o REACH, their administration caused
less widespread conusion among teachers than the
NWE A MAP or REACH Perormance Tasks. However,
the issue o how to use the tests with special education
and ELL students remained: 25 percent o teachers
indicated that they were not at all clear on what accom-
modations were acceptable or students with IEPs and
35 percent were unclear on whether their ELL students
should take the assessment.
As thi s cha pter and the one t hat precedes it show, the
implementation o the observation process and student
growth component o REACH required substantial
eort rom teachers and administrators. In the next
chapter, we explore the challenges and successes o
training and communication to support this eort.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
22/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
20
The Student Growth Component o REACH
Illinois Perormance Evaluation Reorm Act (PERA)
denes three dierent assessment types:
Type I assessments can be the typical multiple-
choice standardized assessment that measures
a certain group or subset o students in the same
manner with the same potential assessment items,
is scored by a non-district entity, and is adminis-
tered either state-wide or beyond Illinois.
Type II assessments can be any assessment
developed or adopted and approved or use by the
school district and used on a district-wide basis
by all teachers in a given grade or subject area.
Type III assessments are rigorous, aligned to
the courses curriculum, and are determined by
the teacher and qualied evaluator to measure
student learning in that course.D
PERA stipulates that all teachers must be evalu-
ated using at least one Type I or Type II assessment
and at least one Type III assessment.E To meet this
requirement, CPS has identied two dierent types o
student assessments to be used as part o REACH:
REACH Perormance TasksAs its Type III assessment, CPS utilizes REACH
Perormance Tasks, which were administered in
the all and the spring and are intended to measure
change in student mastery over one or two skills
or standards. The REACH Perormance Tasks were
developed by over 150 teachers organized into teams
aided by content area specialists rom central oce.
These teams developed over 90 Perormance Tasks
that covered all elementary teachers, including those
teaching in areas such as art, music, physical educa-
tion, and library studies that are not traditionally
covered by standardized tests and a subset o teach-
ers in high school core courses. Each Perormance
Task all/spring pair took approximately 40 hours to
drat, revise, and pilot.
Value-Added and Expected Gains Measures
For its Type I assessment in elementary schools, CPS
has chosen to compute teachers value-added score
on the math and reading NWEA MAP. A value-added
score rom the all to spring administrations o the
NWEA MAP will be computed or teachers who teach
grades three through eight reading or math. All other
elementary school teachers will receive a schoolwide
literacy growth score.
For its Type I assessment or high school teachers,
CPS is exploring using the EPAS suite o tests
(EXPLORE, PLAN, and ACT) to measure expected
student gains. In 2012-13, the EPAS assessments
were administered without stakes. EXPLORE was
administered twice to ninth-graders, PLAN twice to
tenth -graders, and ACT twice to eleventh-graders.
While these scores will not count towards teachers
evaluation this year, the data will be used to develop
an expected gains metric or possible use in the
2013-14 school year.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
23/44
Chapter 3 | Training and Communication
Training and Communication
CHAPTER 3
As with the implement ation o a ny major polic y in itia-
tive, REACH required ex tensive communication and
training eorts at both the district and school levels. In
Chicago, almost 1,200 administrators a nd over 20,000
teachers needed to be inormed about and trained on
the new system in this irst year o implementation.
In this chapter, we describe teacher and administrator
experiences with trai ning and communication. We draw
upon our surveys and interviews o teachers and admin-
istrators to understand how well-inormed and preparedparticipants elt in this rst year o i mplementation, and
to explore what area s they elt still needed improvement.
We nd that , while admi nistrators re ceived ex tensive
training, training and inormation or teachers varied
widely both acro ss and within schools. Fi nally, teachers
and administrators alike expressed a need or transpar-
ency not only about how nal summative scores would be
calculated but also about how teacher evaluation would
ultimately be utiliz ed in personnel decisions.
Administrators Felt Preparedto Conduct Observations andAssign RatingsMore than 80 percent o administrators reported
their proiciency as strong or very strong in recording
and aligning evidence and determining observation
ratings. Administrators received extensive training
in these a reas. Prior to conducting any observations,
administrators had to complete an online certiication
process that included video-based scoring practice and
an assessment o their rating accuracy. On average,
administrators reported spending over 30 hours on
this certiication process. Administrators who did
not pass the a ssessment portion o this certiication
ater two attempts were required to attend additional
in-person training. Adm inistrators who did not pass
the assessment portion ater our attempts did not
conduct observations. As o November 2012, almost90 percent o CPS administrators had been certi ied.35
Beyond certiication, the district required admin-
istrators to attend our hal-day, oundational REACH
proessional development sessions throughout the year.
These sessions included content on the teacher practice
component, evidence alignment, rating calibration, and
evidence-based conversations. On our administrator
survey, about 70 percent o administrators said they
ound district-provided proessional development on
REACH helpul or very helpul.
In interviews, administrators reported conducting
joint obser vations wit h another pr actitioner and dis -
cussing the evidence and ratings they assigned was most
relevant to helping them eel prepared. Administrators
oten relied on their networks Instructional Eec-
tiveness Specialist or this on-the-job training and were
positive about the individualized coaching they received
(see Instructional Effectiveness Specialists box). Over
Instructional Eectiveness Specialists
Specialists conduct joint classroom observations
and calibration sessions with administrators in
their schools. Specialist ratings are not directly
incorporated into any teachers evaluation rating.
Instead their purpose is to work with administrators
on their ability to assign unbiased ratings based only
on the evidence they collected during observations.
In 2012-13 there were 18 specialists, approximately
one or every school network (although some
were not hired by the time school started in the all).
Individual specialists backgrounds, experiences and
capacity varied, but all were certied and trained
to ensure evaluator quality and inter-rater reliability
as well as identiy evaluators needs in conducting
observations.
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
24/44
UCHICAGO CCSR Research Report | Teacher Evaluation in Practice
22
80 percent o administrators ound their conversations
with their netw ork speci alist help ul or ver y help ul
in the areas o evidence collection, alignment, and
assignment o ratings. In interviews, principals stated
they ound conducting joint observations with their
specialist helpul because they could ocus on theirindividual needs as observers.
Principals also relied heavily on their own admin-
istrative teams or training and ca libration. In our
interviews, principals reported spending signiicant
time training, jointly observing, comparing evidence,
and discussing ratings with their assistant principals at
the beginning o the year. One principal and assistant
principal team even planned to conduct all o their
observations together because they elt it would
maximize t heir learning and ensure calibration:
We did them together because we wanted
to coordinate and do what we called, max
calibrate. We wanted to make sure we were
on the same page on what we classied
this is 2A, and then this is 3Bso we had
those discussions, which took a long time.
We just thought that would help us to grow
and evolve.
Principals Want AdditionalTraining on Coaching TeachersWhen asked what resource s or tra ini ng they needed or
observations and conerences, administrators most re-
quently identiied help with acilitating teacher growth
and development. Ninety-our percent reported proes-
sional development on providing useul eedback and 97
percent reported coaching teachers as a high or medium
training priority (see Figure 6).
In interviews, principals identiied speciic issues
with provid ing eedb ack t o te acher s ba sed on obser va-
tions. One principal described having diiculty priori-
tizing areas or improvement with struggling teachers:
Theres 15 things they need to get better at, and so all
15 o them are im porta nt, whe re do I begin?Anot her
administrator talked about struggling to ind the
best way to reach each individual teacher:
An area I still struggle with is... when some-
bodys doing something wonderully, how
to be a good thought partner with making it
better... Just knowing that people want di-
erent things, some people want more direct,
like, This is what you should do, and some
people that turns o immediately. So its just
nding, with each individual, what is going
to be the right piece to reach him or her.
A ew princ ipal s des cribed str ugg ling w ith provi ding
useul and thoughtul eedback to teachers in the ti me
they had allotted: How do you do it well, and have
really thoughtul conversationsI just dont know how
you would get any r eal inormat ion and push somebody
in 25 minutes?Another principal elt that the post-
conerence only allowed her time enough to report to
the teacher what I thought she needed to work onand
not to provide speciic steps on how to improve.
PercentofAdministrators
70
60
50
100
90
80
40
20
10
0
30
Providingprofessional
development
to teachers toaddress specific
needs
(n=612)
Coachingteachers to
improve their
practice
(n=609)
Providinguseful
feedback
to teachers
(n=612)
Source: Spring Administrator Survey, May 2013
Note: Percentages may not add up to 100 due to rounding.
High Priority Medium Priority
Low Priority Not a Priority
FIGURE 6
Most administrators report coaching and feedback
as high priorities for their own training
How high a priority are the following topics for your ownprofessional development?
70%
25%
4%
1%
4%
75%
19%
1%
79%
18%
2%
1%
7/29/2019 Teacher Evaluation in Practice: Implementing Chicago's REACH Students
25/44
Chapter 3 | Training and Communication
Specialists also said their administrators were un-
easy about conerencing with t heir teachers. Specialists
stated some administrators were unsure o the purpose
o the pre-observation conerence, while other adminis-
trations told them they were eeling uneasy about
leading conerences where they have to tell teachersthey received lower ratings than in t he past.
Training and Communication orTeachers Varied WidelyFor teachers, the requency and quality o REACH
trainin
Top Related