MEASURING TEACHING PRACTICE

Tony Milanowski & Allan Odden SMHC District Reform Network

March 2009

Session Overview

1. Our Vision

2. Summary of Our Review of Seven Assessment Approaches

3. Your Assessment Needs & Issues

4. How Can We Move Forward?

Our Vision

District Vision of Effective Instruction

Embodied in Teaching Competency Model

Methods to Measure Teaching Competency

Anchors HRM programs to acquire, develop, motivate, & retain teacher human capital

Teaching Assessment Links Competencies to HC Management Systems

But with value-added, why bother?

• 60-69% of teachers do not teach subjects with testing that allows value-added estimation

• There are issues with the validity of value-added estimates which are often not recognized– High sampling error hard to distinguish any

but top and bottom teachers – Low stability of estimates across years

(Goldhaber & Hanson, 2008; McCaffery et al, 2008; Koedel & Betts, 2007)

But with value-added, why bother?

• Behavioral science research suggests that results-only feedback is not that effective for performance improvement– What do you tell teachers to do in order to

improve their value-added?

– Like teaching golf or tennis using only scores

• How do you know your strategies are being applied?

Effective Measurement Requirements

• Capture Your Vision of Instruction – Reflect key drivers of student achievement

– Reflect district strategies for improving student achievement

• Reliability & Validity • Efficiency• Acceptance by Teachers & Administrators• Adaptability to Varied HCM Uses

Adapting to Multiple HRM Uses

• Doesn’t mean that the same assessment process is applied for every use

• But the same competency model should underlie all the uses

• Different methods of data collection about the competencies could be used (more later)

Seven Assessment Approaches

• Classroom Assessment Scoring System (CLASS) • Formative Assessment System Continuum of Teacher

Development - New Teacher Center at University of California Santa Cruz

• Framework for Teaching & Framework adaption by the Cincinnati Public Schools

• National Board for Professional Teaching Standards Certification Assessments

• Teacher Advancement Program (National Institute for Excellence in Teaching)

• Performance Assessment for California Teachers (PACT)• PRAXIS III© (Educational Testing Service)

Preliminary Conclusion 1• We have several potentially appropriate competency

models to choose from – Underlying competency models have substantial similarities

• Engagement• HOTS• Differentiation of instruction• Use of formative assessment

– But specific differences in emphasis are worth noting • PACT & National Board assessments are most concerned with content &

pedagogical content knowledge• The Formative Assessment System-NTC is concerned with engagement

dimensions• CLASS focuses on social interactions• In Cincinnati, the emphasis is on standards

Preliminary Conclusion 2

• Any model will need to be adapted by incorporating key elements of district strategy (Cincinnati, TAP) – Student standards– Curriculum alignment– Grade or subject teaming– Use of particular literacy or math approach

Preliminary Conclusion 3

• Convergence on what is needed for reliable assessment – Clear standards & multi-level behavior-based

rubrics– Assessor training – Accountability for accuracy

• The technology to reliably assess teaching exists– NB, PACT, CLASS, Cincinnati all can produce

reliable ratings

Preliminary Conclusion 4

• There is promising technology to improve the efficiency of teaching assessments– Videos & written reflections as used in NB &

PACT can allow assessment outside of the school year

– MyTeachingPartner based on CLASS shows how efficient formative assessment can work

Preliminary Conclusion 5

• Some promising evidence on relationship to student achievement for: – PACT predecessor (Connecticut BEST)– National Board – CLASS Pre-K– Cincinnati Framework adaptation

Preliminary Conclusion 6

• One assessment method is not going to fit all HRM purposes – Hard to get good information on content &

pedagogical content knowledge from an observation

– Need to also consider walk throughs– Would need to develop observational

protocols for NB or PACT models for some uses

– The point is to base different data collection methods on the same competency model

Assessment Processes & Purposes

Selection InductionTenure

DecisionPerf. Mgmt

PDCompen-

sation

Formal Observations X X X

InformalObservations X X X

Walk Throughs X X X

Performance Assessment X X X X

Your Assessment Needs & Issues

1. What does your district do to assess teaching?

2. Are you satisfied with your current practice?

3. What areas need improvement?

4. How do you know you are measuring the right stuff?

5. What are the barriers to teacher & administrator acceptance?

6. How important are state teaching standards & licensing/relicensing requirements?

How Can We Move Forward?

• Our Tentative Roadmap

• Does this make sense to you?

How Can We Move Forward?

• Using roadmap, work collaboratively with 3-4 districts to develop their existing teaching assessment approach or adapt another approach.

• Districts would then pilot over the next two school years for teacher development, evaluation and/or tenure decision making.

• We would help conduct a formative evaluation of the implementation and effectiveness of the approach, including an assessment of the relationship between practice measurements and student achievement.