HOLISTIC SCORING:

HOW WE FIND EVIDENCE

GUY COX,, SUELLYN HENKE, K

AREN HOAGLIN,STACEY LEVIN, JASON

MORITZ, SUSANA MURILLO, M

ARY PETRO & KYLE SHANTON

WHO WE ARE

https://www.youtube.com/watch?v=usQem53fDH8

A BIT MORE BACKGROUND

http://www.albion.edu/academics/departments/education

http://www.facebook.com/Albion.College.Teacher.Education

Approximately 1300 total enrollment; 80 in TEP

TEAC accredited through October 1, 2016

A BIT MORE BACKGROUND (cont’d)

Elementary certifications: English language arts, integrated science, mathematics and social studies

Secondary certifications: biology, chemistry, earth/space science (geology), English, French, German, history, mathematics, physics, political science, psychology, social studies (major only) and Spanish

K-12 certifications: French, German, Spanish and music

WHY HOLISTIC SCORING?

● Teacher Education Accreditation Council (TEAC) accreditation weakness

● change from Teacher Education Accreditation Council (TEAC) national standards to Council for the Accreditation of Educator Preparation (CAEP) standards

● need to identify what would constitute evidence and a tool for assessing the evidence

● financial constraints● requires collaboration

HISTORY OF HOLISTIC SCORING

● Gray, J. (1982). Properties of writing tasks: A study of alternative procedures for holistic writing assessment. final report. http://eric.ed.gov/?id=ED230576

● Hunter, D. M. (1996). The use of holistic versus analytic scoring for large-scale assessment of writing.http://eric.ed.gov/?id=EJ543798

WHAT IS HOLISTIC SCORING?

Holistic scoring is a process for assessing and grading the summative quality … based on professional and personal knowledge, the particular process and event, the purposes at hand, and the meanings shared by the individuals involved. An overall impression includes everything...; this cannot be broken into component parts, per se. A holistic system for scoring is legitimate and reasonable because it requires the scorers to cast a comprehensive view .... The process is synergistic because it requires iterative thinking by each participant, and invokes collaboration. It is a kind of gestalt; the whole is greater than the sum of its parts. (after Dr. Judy Nichols Mitchell)

WHAT IS HOLISTIC SCORING (cont’d)?

Advantages—Focuses on what writers can do, and actually did, rather than on what they didn’t do;

Links function, content, and form-the acts of composing and producing;

Serves as an evaluation measure and/or needs assessment; and,

Is authentic to the demands of real writing tasks.

WHAT IS HOLISTIC SCORING (cont’d)?

Preparation—Compose/select prompt

Collect materials

Select anchor set

Select scoring scale

Training—Describe system

Explain prompt &anchor trial

Tally/discuss scores

Develop/confirm criteria

Application—Read and score all materials

Sort/tabulate all scores

Discuss and react

WHAT IS HOLISTIC SCORING (cont’d)?

WHAT IS HOLISTIC SCORING (cont’d)?

Reference for four-point scoring scale—Three categories (unsatisfactory/low; basic/middle; and proficient/high);Distinguished category eliminated due to DFFT/EI conceptual framework for this category;Round down on split scores (e.g., 0+1 or 2+3);Non-adjacent and non-equivalent scores (e.g., 1, 3; 2, 4) require third scoring; and,Across three scores, count two lowest scores.

See also Danielson Framework for Teaching/Evaluation Instrument.

WHAT IS HOLISTIC SCORING (cont’d)?

Scoring Scale—0 (←unsatisfactory) 1, 2 (←basic) 3, 4 (←proficient)

0, 1 (←unsatisfactory) 2, 3, 4, 5 (←basic) 6, 7, 8 (←proficient)

WHAT IS HOLISTIC SCORING (cont’d)?

SAMPLE ANCHOR SET SCORING TABLE 4 3 2 1 0

A B

C

WHAT IS HOLISTIC SCORING (cont’d)?

WHAT DID WE ACHIEVE?

● TEP Curriculum Map

● Danielson Framework for Teaching Evaluation Instrument

● CAEP Standard 5

TEP Curriculum Maphttps://www.dropbox.com/s/q4q5lazfnktzoon/Converted%20TEP%20%26%20Danielson%20FFT%20Evaluation%20Instrument_correlation_v2.1.xlsx?dl=0

CAEP Standard 5:Provider Quality Assurance & Continuous Improvement

The provider maintains a quality assurance system comprised of valid data from multiple measures, including evidence of candidates’ and completers’ positive impact on P-12 student learning and development. The provider supports continuous improvement that is sustained and evidence-based, and that evaluates the effectiveness of its completers. The provider uses the results of inquiry and data collection to establish priorities, enhance program elements and capacity, and test innovations to improve completers’ impact on P-12 student learning and development.

CAEP Standard 5:Provider Quality Assurance & Continuous Improvement

(cont’d)Quality and Strategic Evaluation5.1 The provider’s quality assurance system is comprised of multiple measures that can monitor candidate progress, completer achievements, and provider operational effectiveness. Evidence demonstrates that the provider satisfies all CAEP standards.

5.2 The provider’s quality assurance system relies on relevant, verifiable, representative, cumulative and actionable measures, and produces empirical evidence that interpretations of data are valid and consistent.

CAEP Standard 5:Provider Quality Assurance & Continuous Improvement

(cont’d)

Continuous Improvement5.3. The provider regularly and systematically assesses performance against its goals and relevant standards, tracks results over time, tests innovations and the effects of selection criteria on subsequent progress and completion, and uses results to improve program elements and processes...

WHAT DID WE ACHIEVE (cont’d)?

● Field Experience & Student Teaching Feedback Forms

● Lesson Plan Form

● reflection (e.g., Time I Heard Self)

FIELD EXPERIENCE FEEDBACK FORM

Old Formcom/s/nv8yc7s72ob86p2/ASSESSMENT%20for%20NEW%20PROGRAM%20updated%20Summer%202007.doc?dl=0

New Formhttps://www.dropbox.com/s/jr1adctce07a5oa/field%20experience_final%20evaluation__tier%201%5BR%5D.docx?dl=0

STUDENT TEACHING FEEDBACK FORM

Old Formhttps://www.dropbox.com/s/hd297ic7jqc2gk7/final-feedback%20form%20for%20mentor%20teachers.docx?dl=0

New Formhttps://www.dropbox.com/s/82mlhomshig2si7/field%20experience_final%20evaluation%5Bmentor_teachers%5D__tier%204.docx?dl=0

WHAT PROBLEMS DID WE HAVE?

● Mental shift from scoring on separate attributes to scoring artifact as a whole.

● Review one artifact in relation to one DFFT element; even if two might be applicable, cumbersome to score.

● Explain the context of the artifact—course, assignment, and objectives of assignment.

● As we progressed in our understanding of the DFFT, we re-assigned artifacts to different elements.

● Can’t do it all—small program, so we can score artifacts from all students without sampling; we can’t score all artifacts from all students. Choose for coverage of the framework.

WHAT’S NEXT?

● Add more “tiers” to our process.

● Deepen our understanding of the DFFT to improve the validity of artifacts-to-elements match.

● Expand to actual pre-service teacher performance.