Post on 20-Jun-2020
The Power of Reliable Rubrics Promoting Equitable and Effective Assessment
Practices through Collaboratively Designed Rubrics.
Whitney Bortz, Ph.D. whitneyelaine8@gmail.com
Radford University, CEHD Nov 2015
Objectives of the Session
• Experience how rubrics can improve the assessment process
• Encounter a model for collaboratively writing tools, introducing tools to faculty and testing reliability using a data management system
• Consider how rubric assessment can be improved in our own institutions
Questions
• How many of you work directly with teacher preparation programs at your institution?
• How many of you play a key role in preparing for CAEP or SPA (Specialized Professional Association) accreditation?
• How many of your teacher preparation programs regularly use detailed rubrics to validate common assessments?
• What about outside of teacher preparation?
Context
• School of Teacher Education and Leadership (STEL) at Radford University
• Formerly accredited by NCATE (National Council for the Accreditation of Teacher Education)
• Now working toward accreditation from Council for Accreditation of Educator Preparation (CAEP)
• 13 Licensure areas • CAEP requires common assessments across all initial
licensure programs that demonstrate candidate performance on the Interstate New Teachers Assessment and Support Consortium (InTASC) standards
The Challenge
• STEL has 13 initial licensure programs • 4 – 6 Advanced programs • CAEP requires that each assessment uses a detailed
rubric/scoring guide • Rubric should have 3 – 5 rating levels with clear,
distinguishable descriptions of performance in each level on each indicator
• Creating such rubrics requires much time and effort (Reddy & Andrade, 2010)
• Should be valid, reliable and fair (Andrade, 2005) (Also, required by CAEP)
Valid and Reliable Rubrics
• Validity – Does the assessment tool measure what it intends to measure?
• Reliability – Will the use of the assessment tool produce consistent results across instances?
• Why? Judgments are inherently subjective (Turbow, 2015)
Rate the painting below on a scale of 1 (bad) to 4 (great).
Reflection
• Did the rubric help you confidently rate the painting? Explain.
• How might the rubric have been helpful
to the student artist?
Benefits of Rubrics
• Positively influence the learning process (Jonsson and Svingby, 2007)
• Play a key role in formative assessment (Sadler, 1989) • Assist in the feedback process, specifically (Hattie &
Timperley, 2007; Shute, 2008) • Can increase the quality of student performance
(Petkov and Petkova, 2006) • Can also help identify programmatic areas for
improvement (Song, 2006; Andrade, 2005; Powell, 2001)
• Others?
Formation of the Rubric Writing Team
• Director of STEL, Director of Field Experience, Director of Assessment, Six Faculty Members in Teacher Preparation
• Interdisciplinary • Disciplinary Experts • Hand-selected
Planning and preparation
– Internal grant funding – Established goals and tasks – Collaboratively wrote grant proposal – Established roles and expectations – Training
What makes a good rubric?
According to CAEP (Chepko, 2014)… • Appropriate - Alignment to standards • Definable - clear, agreed-upon meaning • Observable – quality of performance can be
perceived • Distinct from one another – each level defines
distinct levels of candidate performance • Complete – all criteria together describes the
whole of the learning outcome
Rubric Writing Tips
• Start from the middle and work out (Chepko, 2014; Tomei, 2014)
• Changes in levels – Additive – each levels adds more advanced behaviors – Qualitative – the quality of the behavior changes in each
level • The lowest level should still contain a description of
what the rater could expect to see rather than simply the “absence” of something.
• Align the rubric and the assignment guide, if applicable
Common Issues
• Double- or multiple-barrelled criteria or descriptors in one row
• Use of subjective language • Performance descriptors do not encompass all possible
scenarios • Overlapping performance descriptors • Use of technical language that may be interpreted
differently across students and/or raters • Others?
The Writing Process
1. Split into three groups of two to three 2. Draft the three assessment tools 3. Send to whole group for feedback 4. Meeting with a Consultant 5. Groups switch assessments for further editing 6. Whole group review and commenting 7. Final edits
Launching the Rubrics
1. Roll out to Faculty 1. Accompanied by a guidance document 2. Presented as a “suite” of assessments
2. Requesting Feedback 3. Inter-rater reliability exercise 4. Validity
Inter-rater Reliability Process
Select Student Work
Score Independently
Share ratings
Discuss and Approach Consensus (Turbow,
2015)
Change assessment
tool, as needed
Inter-rater Reliability Exercises • Two sample lesson plans (Early Childhood and Middle
School Math) • Rated independently • Shared ratings (see handout) • The Delphi Technique - until consensus was reached
(Hsu & Sandford, 2007) • Consensus often required changes to the rubric and/or
to the assignment guide • Correlation – Spearman’s Rho (see handout)
Next Steps
• Cohen’s Kappa Coefficient: Inter-observer agreement?
• Next steps: Repeat with two new student artifacts using the modified rubric
• Invite other stakeholders to test the tools
Validity (Allen & Knight, 2009)
Focus: 1. Insure the academic concepts and dimensions are
learning skills students need (employer data and professional standards)
2. Insure these are professionally anchored (standards and experts)
Validity (Allen & Knight, 2009)
1. Develop learning objectives 2. Identify a sample of work 3. Develop an evaluation rubric 4. Longitudinally test student learning against professional
assessments 5. Identify problems with sub-optimal learning
performance 6. Improve the construct validity for the rubric 7. Determine the ability of the rubric to differentiate
submissions 8. Improve the reliability of the rubric with data analysis
to further validate the rubric
Professional Learning • Community of Practice (Wenger,
1998, p. 5) • Wenger describes individual
development as the negotiation of meanings through practice
• Negotiation of – Expectation – Meaning – Priority – Performance
Learning
Community learning as belonging
Identity learning as becoming
Meaning learning as experience
Practice learning as
doing
Discussion and Sharing
• Describe rubric use at your institution?
– How are rubrics created? – How are they used? – What benefits have you seen?
• What are some areas of challenge or areas for growth?
Resources
• http://www.introductiontorubrics.com/samples.html • https://www.aacu.org/value-rubrics • http://manoa.hawaii.edu/assessment/howto/rubrics.htm
References
• Allen, S. & Knight, J. (2009). A Method for collaboratively developing and validating a rubric. International Journal for the Scholarship of Teaching and Learning, 3(2), 1 – 17. Retrieved from: http://www.georgiasouthern.edu/ijsotl
• Andrade, H. G. (2005). Teaching with rubrics: The good, the bad, and the ugly. College Teaching, 53(1), 27–31.
• Chepko, S. (2014, September). Quality Assessment Workshop. Workshop presented at the annual meeting of the Council for Accreditation of Educator Preparation, Washington D.C.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
• Hsu, C. & Sandford, B. (2007). The Delphi Technique: Making sense of concensus. Practical Assessment, Research & Evaluation,12 (10), 1 – 8.
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.
• Petkov, D., & Petkova, O. (2006). Development of scoring rubrics for IS projects as an assessment tool. Issues in Informing Science and Information Technology, 3, 499–510.
References
• Powell, T. A. (2001). Improving assessment and evaluation methods in film and television production courses (Unpublished doctoral dissertation). Capella University, Minneapolis.
• Song, K. H. (2006). A conceptual model of assessing teaching performance and intellectual development of teacher candidates: A pilot study in the US. Teaching in Higher Education, 11(2), 175–190.
• Tomei, L. (2014, October). Rubric Design Webinar, Part II. Presented online on behalf of LiveText.
• Turbow, D. (2015, February). Introduction to Rubric Norming. Presented online on behalf of LiveText.
• Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Retrieved from
http://books.google.com/books?hl=en&lr=&id=heBZpgYUKdAC&oi=fnd&pg=PR11&dq=wenger+1998+&ots=kelf2kew5d&sig=6o4HfPARHoP2zkgas2Gy7DdCx3s