Teacher Evaluation for Teachers in

Non-Tested Grades and Subjects

A Pennsylvania PerspectiveCreated by O David Deitz for the PMEA Leadership Conference 7/16/12

(there are still a lot of questions to be asked and answered!)

?

House Bill 1901

• (B) FOR PROFESSIONAL EMPLOYES AND TEMPORARY PROFESSIONAL• EMPLOYES WHO SERVE AS CLASSROOM TEACHERS, THE FOLLOWING

SHALL APPLY:• (1) BEGINNING IN THE 2013-201 4 SCHOOL YEAR, THE EVALUATIO N• OF THE EFFECTIVENESS OF PROFESSIONAL EMPLOYES AND TEMPORARY• PROFESSIONAL EMPLOYES SERVING AS CLASSROOM TEACHERS SHALL

GIVE• DUE CONSIDERATION TO THE FOLLOWING:• (I) CLASSROOM OBSERVATION AND PRACTICE MODELS THAT ARE• RELATED TO STUDENT ACHIEVEMENT IN EACH OF THE FOLLOWING AREAS:• (A) PLANNING AND PREPARATION.• (B) CLASSROOM ENVIRONMENT.• (C) INSTRUCTION.• (D) PROFESSIONAL RESPONSIBILITIES.• (II) STUDENT PERFORMANCE, WHICH SHALL COMPRISE FIFTY PER• CENTUM (50%) OF THE OVERALL RATING OF THE PROFESSIONAL EMPLOYE• OR TEMPORARY PROFESSIONAL EMPLOYE SERVING AS A CLASSROOM

TEACHER• AND SHALL BE BASED UPON MULTIPLE MEASURES OF

• STUDENT ACHIEVEMENT.

Race to the Top

Goal:An effective

teacher in every classroom.

Building Level Data, 15%

Teacher Specific Data, 15%

Elective Data, 20%

Observation/ Evidence, 50%

Teacher Effectiveness System in Act 82 (HB 1901)Observation/Evidence

Danielson Framework Do-mainsPlanning and PreparationClassroom EnvironmentInstructionProfessional Responsibili-ties

Building Level DataPSSA AchievementPVAAS GrowthGraduation RatePromotion RateAttendance RateAP Course ParticipationSAT/PSAT

Teacher Specific DataPSSA AchievementPVAAS GrowthIEP GrowthLocally Developed District Rubrics

Elective Data/SLOsDistrict Designed Measures and Examina-tionsNationally Recognized Standardized TestsIndustry Certification ExaminationsStudent Projects Pursuant to Local Re-quirementsStudent Portfolios Pursuant to Local Re-quirements

5

Charlotte Danielson’s Framework for Teaching

Four Domains 22 Components 50%

Observation/Evidence

Danielson Framework Domains1. Planning and

Preparation2. Classroom Environment3. Instruction4. Professional

Responsibilities

Building Level Data

Building Level DataPSSA AchievementPVAAS GrowthGraduation RatePromotion RateAttendance RateAP Course ParticipationSAT/PSAT

•Building Level Data• PSSA Achievement• PVAAS Growth• Graduation Rate• Promotion Rate• Attendance Rate• AP Course Participation• SAT/PSAT

15 %

Teacher Specific/Elective DataTeacher Specific Data PSSA Achievement PVAAS Growth IEP Growth Locally Developed District Rubrics Elective Data/SLOs District Designed Measures and Examinations Nationally Recognized Standardized Tests Industry Certification Examinations Student Projects Pursuant to Local Requirements Student Portfolios Pursuant to Local Requirements

15%

20%

35%For NTGS: Elective

What are some ways to develop evidence of student achievement in non-tested

grades and subjects?

Student Learning Objectives

Other Assessments

Measures of Collective Performance

Three Approaches to the Challenge

Measures of Collective Performance

School-wide growth measures (like the PA Report Card)

Standardized assessments used to measure the performance of groups of

teachers (like program evaluation)

Other Assessments

Standardized end of course assessments, formative-benchmark-unit assessments

Teacher-developed assessments of student learning or growth

Student Learning ObjectivesContent- and grade/course-specific measurable

learning objectives that can be used to document student learning over a defined

period of time

Can be written to measure growth or mastery

• NY, IN, MA, GA, UT, RI, NH, New Haven CT, Austin TX

Student Learning Objectives

• FL, DE, Pittsburgh PAOther Assessments

• KY, Pittsburgh PAMeasures of

Collective Performance

What are other states/schools using for non-tested grades and subjects?

What model might be most appropriate for PA to develop?

Student Learning Objective Models

Models of Student

Assessment

Measurement of Student

Performance & Teacher

Effectiveness

What is PA planning to do?Student Learning Objectives with

Model Assessments

Why this approach?The “other assessment” approach causes

narrowing of curriculum and is not financially sustainable.

The “measures of collective performance” are already included in the formula (required by Race to the Top) and do not offer NTGS teachers an opportunity to be evaluated on what they actually teach.

SLOs that don’t describe the assessment protocol tend to lack a true learning focus.

The Approach

• Provide a method for teachers to write SLOs.• Provide a method for teachers to design

assessments appropriate to the SLO.• Provide piloted and peer-reviewed models on the

PDE SAS website.• Provide professional development appropriate to

SLO development.The Methodology: AN SLO TEMPLATE

Looking at the components of PA’s SLO Template…..

• Teacher Information• Course Information• Learning Objective• Data and Targets Used to Establish the SLO• Assessment/Performance Task• Administration of the Assessment/Performance Task• Data Collection• Implementation Timeline• Teacher Effectiveness Measure

Teacher Information

Teacher Name

School Name

District Name

…one piece at a time.

Course InformationCourse Title

Content Area

Grade Level(s)

Brief Course Description

Number of students per class/session

Number of minutes per class/session

Frequency of classes/sessions

Total number of classes/sessions

LO (Learning Objective)

(SLO) Student Learning Objective

SLO Rationale Why was this objective chosen? How do you know it is a worthy area of focus? What are the transfer goals? Based upon this objective, how will students be able to independently demonstrate

their learning?

Curriculum Framework Connection What are the big ideas, essential questions, concepts and competencies (as identified

by PA’s Curriculum Framework) to be measured?

Big IdeasEssential QuestionsConceptsCompetenciesStandards Alignment Based upon the big ideas, essential questions, concepts and competencies selected,

identify the content standards that are aligned to this objective.

Data and Targets Used to Establish the SLO

Awareness of student level of preparedness What, if any, are the course or grade prerequisites? What is shown by the current (baseline) data? Which students may need targeted objectives (e.g.

students with IEPs, diverse learners, etc.)?

Targets What is the expected outcome (goal) by the end of the

designated instructional period (i.e., year, semester, course length, etc.)?

Using student baseline data, how many differentiated targets/goals are needed?

Are the targets ambitious yet realistic?

Assessment/Performance Task Assessment/Performance Task Name

Description of the Assessment/Performance Task Describe the assessment/performance task that will be used to measure the learning

objective (LO). Consider any products or artifacts that will be gathered as part of the data collection

process. Identify the developer of the assessments/performance task used (e.g., teacher-made,

district-developed, commercial, etc.).

Assessment/Performance Task Objectives Rationale Describe how the assessment/performance task authentically reflects the learning

objective (LO). How is this assessment/performance task a reliable and valid measure of student

mastery or growth aligned with expected outcomes and goals? How does the assessment/performance task measure the ability of students to

independently apply what they have learned? How is the assessment/performance task free of bias? How do the assessment/performance task outcomes inform instruction? How does the assessment/performance task align with grade, school and district level

curricular goals?

Growth or MasteryDoes this assessment/performance task align with the student achievement goals for the LO?

Growth (change in student achievement across two or more points in time) Mastery (attainment of a defined level of achievement) Growth and Mastery

Administration of the Assessment/Performance Task

Frequency of Assessment/Performance Task Administration How often and when is this assessment/performance task administered? If measuring growth, are multiple assessment windows in place?

Resources Required What unique or specific equipment, technologies, or resources are needed to

complete this assessment/performance task?

Adaptations for Diverse Learners and/or Students with Disabilities What adaptations built into the assessment/performance task are needed

assist diverse learners and/or students with disabilities?

Personnel Can this assessment/performance task be administered by an equivalent

peer? (If not, please explain.) Does a district policy exist with regard to assessment/performance task

administration?

Data Collection

Measuring Student Progress/Preparation How was baseline data collected? (If baseline data was not collected, please explain.) Can baseline data be compared with the results of this assessment/performance task?

Data Collection In what format will data be collected (e.g., database, graphed, portfolio, etc.)? Is a pre-post test being used? (If so, please describe.) How frequently will data be collected?

Rubrics and Scoring Scales How will individual student growth or mastery be determined (defined and scored) using this

assessment/performance task? Include the specific rubric/scoring scale that will be used. Does the rubric and/or scoring scale correlate with the assessment/performance task?

Data Interpretation How can the assessment/performance task results be interpreted in the same way across

equivalent peers? Is there a reliable and valid scoring and interpretive process (i.e., state developed, district-based,

commercial, standardized, etc.) that is associated with the assessment/performance task? If so, please describe.

Implementation Timeline

Date SLO is due to principal

Date(s) for Assessment and Data Collection

Dates to complete Data Interpretation

Date to present Teacher Effectiveness Measure

Teacher Effectiveness Measure

Classroom Objective How will the aggregated scores of individual student assessment results be used to define teacher effectiveness?Failing: few students achieve content mastery or growth

Needs Improvement: less than a significant number of students achieve content mastery or growth

Proficient: A significant number of students achieve content mastery or growth

Distinguished: An exceptional number of students achieve content mastery or growth

Targeted ObjectiveHow will the mastery or growth of targeted student populations be described and used to define teacher effectiveness?

Failing: Did not meet goal, little to no student mastery or growth

Needs Improvement: Did not fully meet goal but showed some student mastery or growth

Proficient: Met goal or otherwise demonstrated significant student mastery or growth

Distinguished: Surpassed goal or otherwise demonstrated significant student mastery or growth

Targeted Population: Mastery and/or growth goal:

Teacher Effectiveness RatingsWhat were the results of the assessments/tasks and how do they relate to the classroom and targeted objectives?Classroom Objective Failing Needs Improvement Proficient Distinguished

Notes/Explanation

Targeted Objective Failing Needs Improvement Proficient Distinguished

Notes/Explanation

SLO Process DimensionSLO –Assessment Developers-ExpertiseWho was involved in establishing the SLO and what is

their expertise?

Exemplars of Student Work

Let’s see how we can apply this template to a music teaching

setting.(Since choir is what I know the best,

I’ll do a choir SLO. With a little reworking, this SLO would be

applicable to other performance situations.)

Teacher Information

Teacher Name O David Deitz

School Name Teacher Effectiveness Demonstration

District Name PA Dept of Education

Course InformationCourse TitleChoirContent AreaMusicGrade Level(s)6-8, 9-12

Brief Course Description

Secondary level music course that provides opportunity to develop and apply the skills, techniques and knowledge appropriate to choral singing. An integral part of the course is the on-going preparation for public concert performances; in and out of school.

Number of students per class/session

30-70

Number of minutes per class/session

40

Frequency of classes/sessions

3 times per week

Total number of classes/sessions

108

LO (Learning Objective)(SLO) Student Learning ObjectiveStudents will develop independent performance skills appropriate to positive musical contribution to a choral performing ensemble.

SLO Rationale Why was this objective chosen? How do you know it is a worthy area of focus? What are the transfer goals? Based upon this objective, how will students be able to

independently demonstrate their learning?

The quality and musicality of a choir is dependent on each individual as a contributor to the collective whole. Individual assessment informs each students of his/her progress and contribution as an independent musician and (hopefully) as a life-long participant in choral singing.

Curriculum Framework Connection What are the big ideas, essential questions, concepts and

competencies (as identified by PA’s Curriculum Framework) to be measured?

Big Ideas The skills, techniques, elements and principles of the arts can be learned, studied, refined and practiced.

Essential Questions

Why is it important to be able to create, recreate and perform music independently?

Concepts

While much of the school-based musical experience happens within a group, it is also important for people to be able to create, recreate, rehearse and perform music independently.

CompetenciesDemonstrate the ability to independently create, recreate, rehearse and perform musical works and explain why this is important.

Standards Alignment Based upon the big ideas, essential questions, concepts and competencies

selected, identify the content standards that are aligned to this objective.

Academic Standards for the Arts and Humanities9.1 Production, Performance and Exhibition of Dance, Music, Theatre and Visual Arts A. Know and use the elements and principles of each art form to create works in the arts and humanities Music elements: • duration • intensity • pitch • timbre Music principles: • composition • form • genre • harmony • rhythm • texture B. Recognize, know, use and demonstrate a variety of appropriate arts elements and principles to produce, review and revise original works in the arts. Music: • sing • play an instrument • read and notate music • compose and arrange •improvise C. Integrate and apply advanced vocabulary to the arts forms

Data and Targets Used to Establish the SLOAwareness of student level of preparedness What, if any, are the course or grade prerequisites? What is shown by the current (baseline) data? Which students may need targeted objectives (e.g. students with

IEPs, diverse learners, etc.)?

There are no prerequisites for this class. No audition is required, but early screening to assign students to a choral singing part can provide the teacher with some baseline information. Students may have previously been in a choral ensemble class, providing a previous year summative assessment. In addition to information provided by IEPs, students with singing difficulties (pitch matching, speech problems, rhythm problems, and vocal production problems) may need objectives tailored to the nature of the difficulty. Four students with pitch match/tonality problems are identified for this SLO.

Targets What is the expected outcome (goal) by the end of the

designated instructional period (i.e., year, semester, course length, etc.)?

Using student baseline data, how many differentiated targets/goals are needed?

Are the targets ambitious yet realistic?

Each student will demonstrate an ability to sing his/her choral part independently in an unaccompanied setting, exhibiting mastery and/or growth in such components as tone quality, expression and style, phrasing, pitch, intonation, rhythm, diction and articulation.

Assessment/Performance Task Assessment/Performance Task NameChoral Music-Individual Student PerformanceDescription of the Assessment/Performance Task Describe the assessment/performance task that will be used to measure the

learning objective (LO). Consider any products or artifacts that will be gathered as part of the data

collection process. Identify the developer of the assessments/performance task used (e.g., teacher-

made, district-developed, commercial, etc.).

Each member of the high school choir will individually perform a teacher-chosen unaccompanied excerpt from the concert music (repertoire). These will be recorded by having students sing either into a microphone or within the visual and audio range of a video camera. Music selected for the second assessment will be of an equivalent grade level, as adjudicated by professional music organizations.

Assessment/Performance Task Objectives Rationale Describe how the assessment/performance task authentically reflects the

learning objective (LO). How is this assessment/performance task a reliable and valid measure of

student mastery or growth aligned with expected outcomes and goals? How does the assessment/performance task measure the ability of students

to independently apply what they have learned? How is the assessment/performance task free of bias? How do the assessment/performance task outcomes inform instruction? How does the assessment/performance task align with grade, school and

district level curricular goals?

Individual assessments are the only way of ascertaining student progress within the context of the larger performance ensemble. Assessments of this type can both be used as an on-going process through formative assessments (making the necessary pedagogical adjustments in instruction and learning, for both the teacher and the student), as well as summative assessments in the final stages of concert preparation.

Growth or MasteryDoes this assessment/performance task align with the student achievement goals for the LO?

Growth (change in student achievement across two or more points in time) Mastery (attainment of a defined level of achievement) Growth and Mastery

Administration of the Assessment/Performance TaskFrequency of Assessment/Performance Task Administration How often and when is this assessment/performance task administered? If measuring growth, are multiple assessment windows in place?

Students will be assessed at the end of the second quarter and again at the end of the fourth quarter.Resources Required What unique or specific equipment, technologies, or resources are needed to

complete this assessment/performance task?

Current choir repertoire, consisting of an eclectic representation of music from various styles, levels of difficulty, and written in various harmonic approaches (homophonic, polyphonic, syllabic, etc.).

Each student will be given an assessment rubric included in their folder, in order to be familiar with the standards and various levels of proficiency.

Basic recording device in order to record student assessments.

Adaptations for Diverse Learners and/or Students with Disabilities What adaptations built into the assessment/performance task are

needed assist diverse learners and/or students with disabilities?

Students may be diagnosed/identified as possessing emotional or mental disorders such as ADHD, ADD, & MR; requiring instructional and assessment adaptations, according to the requirements of IDEA and the student’s IEP. Adaptations may include factors such as: the student’s comfort level in singing alone; student ability to learn, comprehend or retain subject matter; or to perform certain passages with rhythmic or melodic accuracy when singing alone.

Personnel Can this assessment/performance task be administered by an

equivalent peer? (If not, please explain.) Does a district policy exist with regard to assessment/performance task

administration?This assessment could be administered by an equivalent peer; it can also be administered by the student.

Data CollectionMeasuring Student Progress/Preparation How was baseline data collected? (If baseline data was not collected,

please explain.) Can baseline data be compared with the results of this

assessment/performance task?

Assessment from the 2nd quarter will establish baseline data for the 4th quarter.Data Collection In what format will data be collected (e.g., database, graphed, portfolio,

etc.)? Is a pre-post test being used? (If so, please describe.) How frequently will data be collected?

Data will be collected from listening to the recordings and notating the level of student performance on the attached rubric. This process will be administered twice; the first will be considered a pre-test.

Rubrics and Scoring Scales How will individual student growth or mastery be determined

(defined and scored) using this assessment/performance task? Include the specific rubric/scoring scale that will be used.

Does the rubric and/or scoring scale correlate with the assessment/performance task?

Students will be assessed using the following rubric:

COMPONENT 4-Advanced 3-Proficient 2-Basic 1-Below Basic A. Tone Quality Tone is

consistently focused, clear, and centered throughout the range of the voice.

Tone is focused, clear and centered through the normal singing range. Extremes in range sometimes cause tone to be less controlled. Tone quality typically does not distract from the performance.

Tone is often focused, clear and centered, but sometimes the tone is uncontrolled in the normal singing range. Extremes in range are usually uncontrolled. Occasionally the tone quality distracts from overall performance.

The tone is often not focused, clear or centered regardless of the range, significantly detracting from the overall performance.

B. Expression and Style

Performs with a creative nuance and style in response to the score and limited coaching.

Typically performs with nuance and style that is indicated in the score or which is suggested by instructor or peer.

Sometimes performs with nuance and style that is indicated in the score or which is suggested by instructor or peer.

Rarely demonstrates expression and style. Just plays the notes.

C. Phrasing Phrasing is always consistent and sensitive to the style of music being sung.

Phrasing is usually consistent and sensitive to the style of music being sung.

Phrasing is usually consistent and occasionally sensitive to the style of music being sung .

Phrasing is rarely consistent and/or rarely sensitive to musical style.

D. Pitch Virtually no errors. Pitch is very accurate.

An occasional isolated error, but most of the time pitch is accurate and secure.

Some accurate pitches, but there are frequent and/or repeated errors.

Very few accurate or secure pitches.

E. Intonation Exemplary intonation. Practically of pitches are sung in tune.

Slight imperfections, but most pitches are sung in tune, maintaining tonal center (key) throughout.

Regular intonation difficulties, lacking consistent tonal center.

Significant intonation difficulties throughout, resulting in poor tonal center or recognizable melody.

F. Rhythm The beat is secure and the rhythms are accurate for the pitches being sung.

The beat is secure and the rhythms are mostly accurate. There are a few duration errors, but these do not detract from the overall performance.

The beat is somewhat erratic. Some rhythms are accurate. Frequent or repeated duration errors. Rhythm problems occasionally distract from the overall performance.

The beat is usually erratic and rhythms are seldom accurate distracting significantly from the overall performance.

G. Diction Student articulates clearly, and the text of the music is understandable.

Student articulates the words somewhat clearly, and the text can be understood most of the time.

Student sometimes articulates the words, but the text is often not discernible.

Student rarely articulates the words and the text is not discernible.

H. Articulation Manner of articulation is stylistically clear, agile and unified, according to the written musical excerpt, adding a significant degree of clarity and contrast to the music.

Manner of articulation is mostly clear, according to the written musical excerpt. Minor and infrequent inconsistencies do not significantly distract form the musical clarity and contrast.

Manner of articulation is inconsistent and/or awkward and/or incorrect, according to the written musical excerpt. Adding some clarity and contrast.

Manner of articulation is not very evident, according to the written excerpt, resulting in very little to no clarity and contrast.

Data Interpretation How will the assessment/performance task results be interpreted in the same

way across equivalent peers? Is there a reliable and valid scoring and interpretive process (i.e., state

developed, district-based, commercial, standardized, etc.) that is associated with the assessment/performance task? If so, please describe.

An equivalent peer could adjudicate student performance using this rubric by listening to the recordings. This process has a history of reliability, as found through the CT state common assessments. (ctcurriculum.org)

Implementation Timeline

Date SLO is due to principal Prior to the start of the school year.

Date(s) for Assessment and Data Collection

1st assessment: second week in January2nd assessment: 3rd week in May

Dates to complete Data Interpretation 1st week in June

Date to present Teacher Effectiveness Measure June 5, 2013

Teacher Effectiveness MeasureClassroom Objective How will the aggregated scores of individual student assessment results be used to define teacher effectiveness?Failing: few students achieve content mastery or growth

Needs Improvement: less than a significant number of students achieve content mastery or growth

Proficient: A significant number of students achieve content mastery or growth

Distinguished: An exceptional number of students achieve content mastery or growth

Less than 50% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth).

50% -74% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth).

70 % -84% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth).

85% -100% of students will achieve proficient or advanced levels in 6 out of 8 components (mastery) or will achieve improved scoring in 3 out of 8 components, with no decrease in the other 5 components (growth).

Targeted ObjectiveHow will the mastery or growth of targeted student populations be described and used to define teacher effectiveness?

Failing: Did not meet goal, little to no student mastery or growth

Needs Improvement: Did not fully meet goal but showed some student mastery or growth

Proficient: Met goal or otherwise demonstrated significant student mastery or growth

Distinguished: Surpassed goal otherwise demonstrated significant student mastery or growth

Targeted Population: Four students are identified as having pitch match/tonality problems, as determined by history of participation and/or initial screening via audition process.Mastery and/or growth goal: Students with pitch match problems/tonality problems will be assessed with vocal or piano support, on the same voice part, with an expectation that pitch match occur a minimum of 90% of the time.

Teacher Effectiveness RatingsWhat were the results of the assessments/tasks and how do they relate to the classroom and targeted objectives?

Classroom Objective Failing Needs Improvement Proficient Distinguished

Notes/Explanation Literature selected was not comparable between the two assessments.

Targeted Objective Failing Needs Improvement Proficient Distinguished

Notes/Explanation

SLO Process Dimension

SLO –Assessment Developers-Expertise Who was involved in establishing the SLO and what is their

expertise?

SLO/Student Assessment Development Team June 2012 Sheryl Monkelien, Chair, Music Education - Mansfield University Margaret Bauer, Executive Director – PMEA Dennis Emert, Instrumental Music – Fox Chapel SD – Pittsburgh Gerald Savage, Chair, Vocal Department, CAPA – Pittsburgh Ross Whiting, Fine Arts & Humanities Department – Bok Tech HS – Philadelphia Jamie Kasper, Associate Director – Arts Education Collaboration – Pittsburgh.Exemplars of Student Work

Exemplars can be found at ctcurriculum.org.

What is it that the PA SLO process hopes to accomplish?

To meet federal Race to the Top RequirementsTo meet state legislated teacher evaluation

requirementsTo utilize some processes that teachers

already have in placeTo inform principals as to appropriate

processes in content areas with which they may not be familiar

To provide teachers in non-tested grades and subjects an opportunity to be evaluated in their content area

To inform and instruct teachers in non-tested grades and subjects about standards-based education design: 1) Curriculum Frameworks 2) Standards 3) Backward Design4) Transfer of Learning5) PA Standards Aligned System Website

To have an effective teacher in every classroom.

Thanks for all that you do to bring the arts to the

students and communities in Pennsylvania!

OK, I guess this won’t affect my golf game too much!

• Want to develop or pilot an SLO?• Name/email• School/District• Principal Name/email• Specific Discipline in Music• Develop-Pilot-Both

PhaseIII

TeacherEffectiveness

Pilot

Questions? oddeitz@comcast.net