Increasing effective and highly effective teachers State Considerations

• Increasing effective and highly effective teachers

State Considerations

in Designing and Implementing

Evaluation Systems that Include

Teachers of Students with Disabilities

1

Lynn Holdheide, Vanderbilt UniversityHeather Buzick, Ph. D., Educational Testing Service

Samantha Warburton, Massachusetts Dept of Education

2

Multiple sources of evidence inform the summative performance rating

2

Massachusetts Department of Elementary and Secondary EducationRevised 10/15/2011

State Considerations in Designing and Implementing Evaluation Systems that include

Teachers of Students with Disabilities

Lynn HoldheideVanderbilt University, National Comprehensive Center for Teacher Quality

Office of Special Education Programs Project Director’s Conference

Tuesday, July 24, 2012

10:30 a.m. – 12:00 p.m.

4

Learning Targets

• Participants will have an increased awareness of:• considerations states should contemplate when

designing teacher evaluation systems;

• potential solutions to mitigate perceived challenges;

• practical state approaches in teacher evaluation design; and,

• available research to guide decisions and needed research to validate state and district efforts.

5

Federal and State Theory of Action

6

The Goal of Teacher Evaluation

Increasing effective and highly effective teachers• number and/or percentage

• retention and equitable distribution

Method for determining and identifying effective and highly effective teachers• must include multiple measures

• Effectiveness evaluated, in significant part, on the basis of student growth

• supplemental measures may include, e.g. multiple observation based instruments

7

Policy RequirementsPolicy Requirements

8

Considerations for States when Evaluating Teachers of Students with Disabilities

Did not state special education teachersRationale: More than special education teachers are responsible for the academic progress of students with disabilities.

Few new evaluation models differentiate the processRationale: Perceived fairness, improved implementation fidelity, decreased costs.

Intent to help teachers improveRationale: evaluation process accurately measures growth and reinforces the use of high-leverage instructional principles

9

4 State Considerations when Evaluating Teachers of Students with Disabilities

1) Ensuring needs of students with disabilities and their teachers are considered at the beginning of the design process.

Central to ensuring that the evaluation process leads to quality feedback regarding teacher performance.

Design is universal and not retro-fitted after the fact.

10


2) Measuring growth of students with disabilities

Electing to exclude the scores of students with disabilities within value-added modeling or other measures of student learning could greatly limit teacher accountability. When growth is not accurately measured for students with disabilities or performance expectations are not aligned with possible learning outcomes, teachers may be less likely to accept students with disabilities in their classrooms.

11

A Forum of State Special Education and Teacher Effectiveness Experts and Researchers

• To identify the challenges in using the growth of students with disabilities to evaluate educators

• To develop considerations for states when designing systems that include the academic growth of students with disabilities

• To identify needed areas of research http://www.tqsource.org/pdfs/

TQ_Forum_SummaryUsing_Student_Growth.pdf

12

Measuring Student Growth

• Teachers want to be confident that the measures used are a fair and accurate representation of both student growth and their contribution to that growth.

12

13

Tested Subjects

• Challenges in Using Growth Models for Special Educators & SWD A research-derived value-added model for special

educators does not exist Student learning trajectory Students assessed with accommodations Small student samples commonly associated with

special education caseloads Student mobility Test Scaling

13

14

Considerations for States

• Use multiple measures and consider weights to reflect the amount of evidence in support of validity and accuracy for value-added scores

Based on transparent judgment initially; then empirically based

• Support accessible assessments that offer precise measurement along the entire score scale (e.g., multistage adaptive assessment, universal design)

• Create a standardized system to accurately assign, monitor, and record the use of testing accommodations

• Adopt a roster validation system; use full roster method to give 100 percent credit to all teachers in a coteaching situation

15

Measuring Teachers’ Contributions to Student Learning Growth: A Summary of Current Models

Model Description

Student learning objectives

Teachers assess students at beginning of year and set objectives and then assess again at end of year; principal or designee works with teachers to determine success

Subject and grade alike team models

Teachers meet in grade-specific and/or subject-specific teams to consider and agree on appropriate measures that they will all use to determine their individual contributions to student learning growth

Pretests and posttests model

Educators identify or create pretests and posttests for every grade and subject

Schoolwide value added

Teachers in tested subjects and grades receive their own value-added score; all other teachers get the schoolwide average

16

Potential Challenges for Students With Disabilities in the SLO Process

• Students with disabilities could be overlooked in the SLO process. Therefore, the growth (or lack thereof) could go unnoticed.

• Increased need for teacher capacity to collect, interpret, and monitor student performance data against standards-aligned, rigorous goals.

• The comparability of measurement and student growth is compromised because the process may not be standardized or objective.

17

Sampling of State Considerations

• Ensure that growth for all students, including students with disabilities, is accounted for in the SLO process.

• Encourage collaboration between general and special education teachers to construct SLOs to ensure alignment with the established standards AND to accommodate the specific learning needs.

• Encourage that SLOs can be tiered so that student targets can be differentiated according to the present levels of student performance.

18

Rhode Island Student Learning Objectives

• There is a requirement that all students are covered under an SLO: General education teachers are responsible for the

progress and mastery of all students on their rosters, including students with disabilities.

Teachers are encouraged to set tiered goals so that targets are differentiated.

General education and special education teachers are encouraged to work collaboratively

19

Rhode Island Student Learning Objectives

• Partnered with special education teachers in early adopter districts and local institutions of higher education to draft sample SLOs.

• Example SLOs for students with disabilities are located at http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/SLO.aspx.

• Guidance document that provides recommendations on SLO development across context http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/Docs/SPED_FAQ_revised.pd

http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/SLO.aspx

http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/SLO.aspx

http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/Docs/SPED_FAQ_revised.pd

http://www.ride.ri.gov/EducatorQuality/EducatorEvaluation/Docs/SPED_FAQ_revised.pd

20

Groupwide Value-Added Challenges

• Teachers may be held accountable for students they have never taught or had the opportunity to influence.

• Accountability for the growth (or lack thereof) of students with disabilities may not be captured or monitored if students with disabilities are not included in the value-added scores.

• Teachers may not be as motivated to improve student mastery of state standards if there is no direct accountability for their content areas.

21


3) The appropriateness of existing measures of

instructional practice Evidence-based instructional practices for students with

disabilities- direct/explicit instruction, scientifically based reading instruction,

learning strategy instruction

Specific roles and responsibilities of special educators- Consultant, expert, and/or teacher

- IEP facilitation, collaboration, secondary transition, social and behavioral interventions, compliance with legal mandates

Specific curricular needs- Expanded Core Curriculum (Post School Outcomes)- Access

22


4) The distinct considerations for teachers (both general and special education) serving in a coteaching capacity.

Should effective co-teaching practices be a factor in teacher evaluation?

Would using the general observation rubric suffice?

How should student growth be attributed in an co-teaching classroom?

23

Lynn HoldheideSenior TA [email protected] Thomas Jefferson Street NWWashington, DC 20007-3835Phone: 877-322-8700 or 202-223-6690Website: www.tqsource.org

mailto:[email protected]

Copyright © 2012 Educational Testing Service. All rights reserved.

State Considerations in Designing and Implementing

Evaluation Systems that include Teachers of Students

with Disabilities

Heather BuzickEducational Testing Service

OSEP Project Directors' ConferenceJuly, 2012

Copyright © 2012 Educational Testing Service. All rights reserved.Copyright © 2012 Educational Testing Service. All rights reserved.

Overview

• Measurement challenges, current research, suggestions for practice– Student academic growth as an indicator of

teacher effectiveness (value-added)– Teacher observation protocols– Multiple teachers

• Research ideas

25


Concerns

• Teachers are not held accountable for the education of all students if indicators from students with disabilities are not included in teacher evaluation systems

• Current evaluation systems may not capture differentiated instruction

• Challenges can create disincentives to accept particular students into the classroom

• Perceived unfairness to teachers with high numbers of students with disabilities

26


Two broad areas that can impact the meaning of value-added

scores• Measurement challenges

– Various threats to the validity of inferences about student academic growth that is attributed to teachers

• Complex instructional context– Can impact evaluation of both general education

teachers and special education teachers

27


Measurement challenges (I)

• Testing accommodation use– Inconsistent use across years– Particularly for those associated with a score boost (e.g.,

read aloud, extended time)

• Extreme low performance on linear state assessments– Difficult to get a good measure of growth– Systematic and predictable

28


Measurement challenges (II)

• Small samples or missing data– Some special education classrooms– Lower match rates for students with

disabilities due to mobility, absence on test dates, etc.

29


Instructional context

• Shared responsibility between general education and special education teachers

• Time that students spend in the regular classroom learning content

• The performance of all students in a classroom may be impacted – positively or negatively – by the presence of a co-teacher, extra funding support for special services, peer behaviors, or other factors not directly related to an individual teacher

30


Evidence and research

Analyses with state general assessment data to:

• Identify systematic characteristics of student data that may impact meaning and validity of inferences about teachers

• Research robustness of value-added models to systematic characteristics– Document factors that do not threaten validity– Find solutions for those that do

31


Classroom contextExample from one state database, general assessment

reading and math scores and associated teacher:

• 59% of teachers in grades 3-8 in the sample had at least one student with a disability in the classroom

• General education teachers had 3 to 4 students with disabilities on average or 16% of students in classroom

• 5 students on average for special education teachers (13% of teachers)

32


Average Mathematics Gain Scores(difference in annual scale scores)

Grade: 4 5 6 7 8 Studentsper grade

% of teachers

Mix of general ed. and special ed. 20 17 16 16 9 ~25,000 ~40%

No students with disabilities 20 18 15 16 9 ~40,000 ~47%

33



% of teachers



Students with disabilities only 11 19 15 14 13 ~2,000 ~13%



% of teachers



Students with disabilities only 11 19 15 14 13 ~2,000 ~13%


Accommodations

• 56% of teachers in the sample had at least one student who used an accommodation on the general assessment– Half of those teachers has at least one student with inconsistent

accommodation use across years– An average of 3 students per teacher used accommodations

inconsistently

• Across three states, up to ¼ of students who received accommodations on the general assessment did so inconsistently across years – Related to grade

34

Copyright © 2012 Educational Testing Service. All rights reserved.Copyright © 2012 Educational Testing Service. All rights reserved.35

Number of students

Accommodations/Grade: 4 5 6 7 8

None 61,405 59,947 58,976 60,308 60,142

Consistent 2,368 2,448 2,142 2,506 2,332

Add read aloud 369 332 311 250 300

Add read aloud, extra time

516 522 468 419 368

Remove read aloud 237 306 287 387 406

Remove read aloud, extra time 202 241 267 294 229

Average Gain Scores in Reading


None 13 19 18 3 15

Consistent 7 19 8 8 19

Add read aloud 23 36 13 17 24


27 35 21 13 29

Remove read aloud -1 9 4 2 11

Remove read aloud, extra time -4 12 0 -2 8

Prior Year Reading Scores - Difference from Non-Accommodated Group


None - - - - -

Consistent -28 -35 -35 -46 -47

Add read aloud -40 -48 -39 -49 -44


-42 -48 -47 -52 -57

Remove read aloud -19 -27 -29 -39 -39

Remove read aloud, extra time -21 -30 -30 -39 -40

Average Gain Scores in Reading


None 13 19 18 3 15

Consistent 7 19 8 8 19

Add read aloud 23 36 13 17 24


27 35 21 13 29

Remove read aloud -1 9 4 2 11

Remove read aloud, extra time -4 12 0 -2 8


Considerations for states

• Identify any systematic and predictable factors that may impact teacher ratings based on student outcomes (annually, across state)– Consider including factors in value-added model– Document and communicate factors that do not

threaten the validity of inferences about teacher effectiveness

– Identify special cases where validity is in question• For special cases

– Work with teachers to understand the quality of their individual value-added score given their particular classroom context

– Adjust weights on value-added scores and other measures

36


Observation Protocols

• Single general protocol (e.g., Danielson’s Framework for Teaching)– May not sufficiently outline expectations for instruction

provided to students with disabilities– May not create incentives for teachers to adopt effective

practices for teaching students with disabilities– Definition of effective teaching does not always map on

to definitions of effective teaching put forward by special education community

• Some teachers may never able to be rated in top category

37


Reliability of observers’ scores

• Can observers reliably differentiate between teachers who do and do not make use of effective instructional practices for students with disabilities?

• Rater background and familiarity with educating students with disabilities can contribute variability to observers’ scores

38


Suggestions - Observations

• One option for districts would be to adopt observation protocols designed specifically for use with students with disabilities– Classroom Climate Scale, as developed by McIntosh, Vaughn, Schumm, Haager,

& Lee (2002)– Individualizing Student Instruction protocol (Connor et al., 2009), which

examines how teachers’ tailor their instruction to students’ individual needs

• This option is costly and unlikely to be adopted by states or districts

Connor, C. M., Morrison, F. J., Fishman, B. J., Ponitz, C. C., Glasney, S., Underwood, P. S., et al. (2009). The ISI Classroom Observation System: Examining the literacy instruction provided to individual students. Educational Researcher, 38, 85–99.

McIntosh, R., Vaughn, S., Schumm, J. S., Haager, D., & Lee, O. (1993). Observations of students with learning disabilities in general education classrooms. Exceptional Children, 60, 249-261.

39


Suggestions - Observations

• A more feasible option is to supplement an existing observation protocol with a subset of items specific to teaching students with disabilities– Consider incorporating domains from one of the existing protocols

specific to students with disabilities– Alternatively, existing response categories on observation protocols could be

adapted to more appropriately reflect teachers’ interactions with students with disabilities

– One viable short-term solution would be to develop “scoring support documents” to assist observers in the scoring process, with an emphasis on the kinds of evidence-based practices that have proven to be effective for teaching students with disabilities

40


Suggestions – Observer ratings

• To improve observer familiarity with instruction for students with disabilities, districts could ensure that observers have some training or background specific to special student populations

• During rater training – include one video of special education instruction

41


Considerations formultiple teachers

• A roster validation system can increase both the face validity of value-added scores as well as the accuracy of estimates (Hock & Isenberg, 2011) – Example: The Houston Independent School

District uses a system where teachers can regularly log in and verify the accuracy of their rosters

42


Considerations formultiple teachers

• Full roster method– Both the general education and special education

teachers receives 100 percent responsibility of their shared students

– Helps to ensure that students with disabilities are not viewed as the sole responsibility of the special education teacher (Hock & Isenberg, 2011)

Hock, H., & Isenberg, E. (2011). Methods for accounting for co-teaching in value-added models. Princeton, NJ: Mathematica Policy Research. Retrieved from http://www.aefpweb.org/sites/default/files/webform/Hock-Isenberg%20Co-Teaching%20in%20VAMs.pdf

43


Research ideas

• Research on value-added models:– Routine validation research that includes scores from

students with disabilities– Sensitivity studies with specific variables relevant to

students with disabilities (i.e., accommodation use, entry/exit from special education)

– Explore the correspondence between value-added scores and other indicators of teacher effectiveness by subgroup

44


Research ideas

• Research on observation protocols:– Provide guidance on how to modify rubrics to include

items or response categories specific to students with disabilities

– Conduct research on validity and reliability for modified rubrics or specific observation protocols for students with disabilities

– Evaluate observer performance specific to those who educate students with disabilities

45

State Considerations in Designing and Implementing Evaluation Systems that Include

Teachers of Students with DisabilitiesOSEP Program Directors’ ConferenceJuly 24, 2012

Samantha Warburton

Educator Evaluation Project Lead

When policy and practice must move faster than research and

development, where do you begin?

Massachusetts philosophy:

Don’t let perfection become the enemy of good: the work is too important to delay

Understand this is just the beginning: we will be able to do this work with increasing sophistication each year

Phase-in implementation: take advantage of emerging research, resources, and feedback from the field

Massachusetts Department of Elementary and Secondary Education

4747

One State’s Perspective…

48

June 2011 – MA Board of Education passed new regulations

September 2011 – Implementation began in 34 “Level 4” schools, 11 “Early Adopter” districts, and 4 Special Education Collaboratives

January 2012 – MA Dept of Elementary and Secondary Education (ESE) published the Massachusetts Model System for Educator Evaluation

September 2012 – Implementation begins in all RTTT districts

September 2013 – All districts implement educator evaluation

September 2013 – Districts begin phase-in of Rating of Impact on Student Learning


48

MA Educator Evaluation Framework: Status and Timeline

Educators earn two separate ratings

49


* Educator Plans above apply only to experienced educators; all new educators are placed on a “Developing Educator Plan.”

50

5 Step Evaluation Cycle Every educator

is an active participant in an evaluation

Process promotes collaboration and continuous learning

Process applies to all educators


Questions for Policy Makers:

• Attribution: “When crediting teachers for student learning, how should the individual contributions of teachers acting in a coteaching or consultant role be determined?

• Assessments: “How can the contributions to student achievement be accurately measured for teachers instructing special populations for which alternative standards and/or assessments used?”

• Educator differentiation: “Are the key features of teacher effectiveness for specialized personnel, such as special education teachers different… and should those unique features lead to additional or different content on observation protocols, student growth assessments, or alternative instruments?”

• Evaluator training: “When rating special education teachers…using an observation protocol or alternative instrument, what special training, if any, do evaluators need?”

Holdheide, L.R., Goe, L., & Reschly, D.J.. (2010) Challenges in Evaluating Special Education Teachers and English Language Learner Specialists. National Comprehensive Center for Teacher Quality.


51

Who IS the evaluator?

Are variations in contributions

measurable? How should we use the MCAS

Alternate Assessment?

How do we differentiate

without creating “two systems”?

Implementation Strategies

1. Approach to internal SEA collaboration

2. Approach to external field engagement

3. Early choices in system design

4. Alignment across educator effectiveness continuum


52

1. Internal SEA Collaboration

Educator Evaluation team member sponsored by Office of Special Education Policy and Planning

Strategies for ongoing collaboration: Monthly meetings with Director of Special Education Participation in Ed Eval Leadership Steering Committee (LSC)

and work group developing guidance on Impact Rating Structured communication about relevant activities and issues

Presentations on Ed Eval framework to Special Ed office staff

Presentations to LSC and educator effectiveness office on considerations for and MA approach to evaluating teachers of students with disabilities

Internal work group focused on evaluation for teachers of students with disabilities


53

2. External Field Engagement

It takes time for these questions to emerge: intentionally created conditions to ensure engagement and feedback from the beginning

Strategies for ensuring engagement and feedback: Early implementation in volunteer Special Ed Collaboratives Presentations/workshops for district Special Ed Directors

Included districts with Level 4 schools implementing in 2011-12

Leveraging resources: Communications: state associations

Advisory and feedback: early implementers and local advocacy groups

Info on promising practices and strategies in other states:

state and national support networks and researchers


54

3. Early Design ChoicesCreated a Model Rubric for all teachers; developing additional resource to support its use

Strategies for design: Solicited review of draft teacher rubric by internal work group

and the field (Special Ed teachers & administrators) Majority of recommendations for SpEd teachers reflected best practices

for all students; consequently raised the bar for all teachers

Identified need for resource that will: Provide detail on knowledge, skills, and responsibilities unique to work

with students with disabilities; potential sources of evidence

Support evaluators in using rubric for both special education and general education teachers (esp. principals with limited SpEd expertise)

Support teachers in self-assessing and goal setting


55

56

Example (DRAFT)Work began by SpEd Administrators;

currently under development by group of SpEd teachers

* May provide more differentiation by severity of disability and/or responsibilities associated with ROLE of teacher

4. Alignment Across Continuum

Educator evaluation doesn’t exist in a vacuum: envision aligning preparation, induction, licensure, and ongoing professional development

Strategies for alignment: Role for higher education and preparation programs

Evolution from “Why are you telling us this??” to “Please tell us more!”

Specific examples: Guidelines on Induction and Mentoring

Addressing alignment between engaging in PD activities for evaluation with requirements for PD activities toward relicensure


57

58

How do I learn more?Visit the ESE educator evaluation website:

www.doe.mass.edu/edeval

Contact ESE with questions and suggestions:

[email protected]

Study the MCAS Growth Model:

www.doe.mass.edu/mcas/growth/

Link to MTA’s one-page overview of the regulations:http://massteacher.org/advocating/Evaluation.aspx

(click on “chart”)


58

http://www.doe.mass.edu/edeval

mailto:[email protected]

http://www.doe.mass.edu/mcas/growth/

http://massteacher.org/advocating/Evaluation.aspx

Additional Information on the Massachusetts Educator Evaluation Framework

60

Multiple sources of evidence inform the summative performance rating

60


6161

Putting the two ratings together


Combination of summative rating and impact on student learning determines the type and

duration of Educator Plan

• Developing Educator Plan: Duration is one school year or less, developed by the educator and evaluator for:– Administrators in the first three years in a position in a district– Teachers without Professional Teacher status– Educators in a new assignment (at the discretion of the

evaluator)

• Self-Directed Growth Plan: Developed by the educator for:– Experienced educators rated proficient or exemplary with

moderate or high rating for impact on student learning (two-year plan)

– Experienced educators rated proficient or exemplary with low rating for impact on student learning, with goals focused on the inconsistency and supervisor review (one-year plan)

62


Combination of summative rating and impact on student learning determines the type and

duration of Educator Plan

• Directed Growth Plan: Duration is one school year or less developed by the educator and the evaluator for:– Educators who receive a Summative Performance Rating of

Needs Improvement

• Improvement Plan: Duration is at least 30 calendar days and no more than one school year, developed by the evaluator for:– Educators who receive a Summative Performance Rating of

Unsatisfactory, with goals specific to improving the educator’s unsatisfactory performance

63


64

ESE’s Model Systemfor Districts to Adopt or Adapt

District-Level Planning and Implementation Guide

School-Level Planning and Implementation Guide

Guide to Rubrics and Model Rubrics for: Superintendent, School-Level Administrator, Teacher, and Specialized Instructional Support Personnel Additional guidance on role-specific indicators

Model Contract Language

Protocol for superintendent evaluation

Protocol principal evaluation

65

Later additions to the Implementation Guide

Guidelines for Developing and using multiple measures of

student learning, growth and achievement Determining low, moderate and high impact on

student learning

Examples and Resources on: Multiple measures of student learning Determining educator impact Ways to collect and use feedback from students

& staff

ESE plans to collect and disseminate promising practices at the local level

Increasing effective and highly effective teachers State Considerations

Documents

Transcript of Increasing effective and highly effective teachers State Considerations