DesigningEffecveScoringRubrics for’any’Assignment · Sets of criteria Description of scoring...
30
Designing Effec+ve Scoring Rubrics for any Assignment Heidi Evans and Kris-n Dalby Program in ESL, UWMadison WITESOL 2016 1
Transcript of DesigningEffecveScoringRubrics for’any’Assignment · Sets of criteria Description of scoring...
Designing Effec+ve Scoring Rubrics for any Assignment Heidi Evans and Kris-n Dalby Program in ESL, UW-‐Madison
WITESOL 2016
1
Overview
• Rubric basics • Your experiences • Design framework • Involving your students • Your turn
2
What is a rubric? • “A scoring tool that lays out the specific expecta-ons for an assignment. Rubrics divide an assignment into its component parts and provide a detailed descrip-on of what cons-tutes acceptable or unacceptable levels of performance for each of those parts.” (Stevens & Levi, 2005, p. 3).
• “A scoring tools that lists the criteria for a piece of work or ‘what counts.’ … It also ar-culates grada-ons of quality for each criterion, from excellent to poor.’ (Andrade, n.d.)
• A rubric is a printed set of scoring guidelines (criteria) for evalua-ng work (a performance or a product) and for giving feedback. (Rubrics.com Website)
3
Warm-‐up ques+ons
1. Do you use rubrics? What for?
2. Describe a rubric you have used/designed for a par-cular assignment or task.
3. What challenges have you experienced in designing rubrics?
4
Task descrip-on (Op-onal) • focus students and remind teachers of the assignment parameters
Assessment scale • how well or poorly a student executes a given task/assignment; 3-‐5 levels max
Dimensions • Represent the component parts of a task/assignment
Descrip-ons of the dimensions • Explana-on of the grading criteria.
Rubric Components
5
Rubric types
6
Criteria 1 2 3 Number of Sources X1 1-‐3 4-‐6 7-‐10 Accuracy X3 Several
inaccuracies Few inaccuracies No apparent
inaccuracies
Cita-ons X1 Source of informa-on is unclear
Source of informa-on is not always clear
Source of informa-on is clear
Bibliography X1 Most entries contain mistakes
Some entries are correct
Entries are correct
Analytic Sets of criteria Description of scoring dimensions
Rubric for use of sources in an academic writing assignment
7
Rubric types Scoring Guide ESL 117 Essay 1 Detailed Outline Grade Sheet /5 The thesis statement contains a focus and topic, and effectively
introduces the supporting points. /5 Each major section of the body logically connects to the thesis statement,
with clear topic sentences. The topic sentences repeat the supporting points but express them in a different way.
/5 The outline is complete. All evidence is written in complete sentences.
The information is logically organized and follows the order of the thesis statement.
/10 Supporting points are effectively supported with a variety of specific,
appropriate evidence that has been paraphrased from the sources. /5 For each piece of evidence, the student has indicated the source used
by including the author’s last name (or article title) the publication date, and the page number in citation format.
Comments:
8
Rubric types
5% This was an interes-ng and thought-‐provoking discussion that used various elements of the discussion language that we prac-ced in class. All members of the group were on topic and the discussion sounded natural.
4% This was an interes-ng discussion that made an aiempt at some discussion language. The discussion was generally good, but there were too many simple spoken errors.
3% This was a reasonable discussion that included some basic elements of discussion language. There were too many errors and the conversa-on sounded a liile unnatural.
2% This was a slow and s-lted discussion that had very basic elements of discussion language and far too many errors. The discussion appeared to have had very liile thought or prepara-on involved in its produc-on.
1% This was too short and was poorly put together. There were too many spoken errors and the discussion sounded very unnatural.
0% Not completed.
Task: Assessed Discussion Holistic
Task: Look at figures 1, 2, and 3, and iden-fy each type of rubric and the components in each rubric.
Design framework for rubrics: Example
• Step 1: Iden-fy the learning outcomes of the task/ ac-vity / assignment.
• To introduce a classmate with sufficient and appropriate informa-on.
• To say classmate’s name clearly.
• To use one of the organiza-onal paierns and some of the transi-ons recommended from the textbook.
• To use effec-ve verbal and nonverbal skills (that correspond to what is emphasized for this presenta-on in the textbook).
9
Example con+nued • Step 2: List the specific criteria you wish to evaluate.
• Pronounces classmate’s name clearly and loudly.
• Includes informa-on that is appropriate for the audience.
• Organizes informa-on logically.
• Uses effec-ve transi-ons and connec-ng devices. • Establishes and maintains eye contact.
• Looks at all audience members during speech.
• Avoids distrac-ng movements.
• Speaks loudly enough for students in the back of the room to hear.
• Speaks at an appropriate rate (not too quickly or too slowly)
10
Example con+nued • Step 3: Group (categorize) and label criteria to form dimensions. Op-onal: weight the dimensions (e.g. content=50%, delivery=50%)
• Organiza-on: informa-on logically organized; transi-ons used effec-vely.
• Eye contact: established and maintained; looks at all students.
• Pace: speaks at an appropriate rate; doesn’t speak too quickly or too slowly.
• Step 4: Describe levels of success • Ex: for eye contact
• Very liile or no eye contact; mostly read from notes.
• Some eye contact; could look at all audience members more. May have occasionally read from notes or looked at notes frequently.
• Eye contact established and maintained. Did not use notes or just glanced at notes occasionally for support.
11
Example con+nued • Step 5: Create assessment scale (3-‐4-‐5 points).
• Developing / competent / exemplary
• Needs work / so-‐so / good / excellent • 1 / 2 / 3 • Fail / low pass / pass / (high pass) • J L
• Step 6: Put it all in a table. Title
Dimensions excellent average weak 1. 2. 3. 4.
12
Design framework: Inform with student work
13
14
Points to consider: Task design & assessment
Assignment Desired Outcomes
Assessment Rubric Dimension
Students can write the first sentence of a summary.
First sentence – author, -tle, year and main idea
Students can iden-fy major points in an ar-cle.
Content – all major points iden-fied
Students can disregard unnecessary informa-on in an ar-cle
Content – no unnecessary informa-on included
Students can use transi-ons and other cohesive devices.
Organiza:on – transi-ons used effec-vely
Points to consider: Bilingual instrument?
15
Composing Composición Develops a story/topic with suppor-ng details
Desarrolla una narración con detalles. específico.
Writes about a topic with a central idea Escribe una narración simple sobre un tema
Writes related sentences about a specific topic
Escribe oraciones relacionadas sobre un tema específico.
Writes some related sentences and some unrelated sentences
Escribe algunas oraciones relacionadas y otras no relacionadas.
Writes single sentence or unrelated sentences
Escribe oraciones no relacionadas al tema
Uses leier strings, labels and/or words only
Rotula o escribe palabras solamente.
Source: http://www.cal.org/twi/rubrics/
Points to consider: Language
16
Cohesive Devices? Transition Words! Parallel expression
Points to consider:
“I” language
17
Points to consider: Prac+ce/model
18
Design framework: Rubrics are assignment dependent
• Assignment: summary • Students: freshman undergrads • Course: developmental academic wri-ng I • # students: 16 • # dravs: 1 (no revisions) • Submiied Wed, must be returned Fri Task: Look at figures 4, 5, and 6. What are the advantages and disadvantages for the different summary rubrics? Which would you use for the above assignment? When might you use the other rubrics?
Design framework: Consider the following when you choose a rubric (1)
• If the evalua-on is forma-ve or summa-ve • Summa-ve? à holis-c • “quick and dirty?” à holis-c • Forma-ve? à analy-c or scoring guide
• What students will do with the feedback (e.g. revise their work? Write a second drav?)
• Revise? à analy-c or scoring guide (to communicate strengths and weaknesses)
• Assignment length/complexity? • Short/minor à holis-c, scoring guide, very simple analy-c • Longer/more complex à scoring guide, analy-c
Design framework: Consider the following when you choose a rubric (2)
• Type of assignment (academic or crea-ve) • Crea-ve (story, poem) à scoring guide • Oral presenta-on à analy-c • Essay à scoring guide, analy-c • Porwolio / project à analy-c, scoring guide
• How much -me you want to spend grading • Holis-c rubrics à faster • Analy-c à can be faster (familiar w/it; not too complex) BUT the more performance levels and descriptors, the longer it will take to use
• Scoring guide à can be faster (format for note-‐taking)
Design framework: Consider the following when you choose a rubric (3)
• How many students you have • Fewer? à scoring guide (customize feedback) • Many? à holis-c, analy-c
• Who the students are (K-‐12, undergrads, grads) • Recycle for self, peer, and teachers? • Use for learning (and instruc-onal purposes)? • Makes “score” less “biased” & more “objec-ve/transparent”
• Need for flexibility in giving feedback • Scoring guide à customize comments • BUT! There should be a “comments” sec-on at the end of any type of rubric.
Points to consider: A rubric is never finished!
23
Student-‐teacher collabora+on
1. How might you involve students in rubric design?
2. What are the benefits / challenges with student-‐teacher collabora-on?
24
Student-‐teacher collabora+on con+nued
• Transla-on: Students translate the instrument and prac-ce using it with example work.
• Students modify criteria statements: Aver the teacher completes the rubric, show it to students who paraphrase criteria to improve understanding.
• Students finish an incomplete rubric: Leave the “best” and “worst” descriptors blank and have students fill them in.
25
Student-‐teacher collabora+on con+nued: Create a rubric together
26
• Learners identify criteria in groups
• Learners organize criteria into dimensions (and provide labels)
• Teacher uses learner suggestions and language in the instrument
• OR learners design instrument
Your turn: Discuss how to develop a rubric for an assignment you have in mind or choose one below: • Topic: A persuasive paragraph about why Americans should or should not eat less meat (or topic of your choice) OR a group discussion on the advantages and disadvantages of adop-ng a vegetarian diet (or topic of your choice).
• Direc-ons: 1. Iden-fy the learning outcomes of the task or
assignment. 2. Brainstorm a list of criteria you will use to evaluate
the students’ performance/product. 3. What type of rubric do you want to use (analy-c,
scoring guide, holis-c)? 4. Time? How might you involve students? How can you
relate the rubric to Common Core Standards? 27
Discussion
• How can you apply these techniques to your context?
• How can rubrics be used as teaching tools? As learning tools?
• What final concerns do you have in crea-ng and using rubrics?
28
Sources consulted 1. Andrade, H.G. (2007). Understanding rubrics. hip://learnweb.harvard.edu/ALPS/
thinking/docs/rubricar.htm. 2. Bailey, K.M. (1998). Learning about language assessment: Dilemmas, decisions,
and direc5ons. Cambridge, MA: Heinle & Heinle. 3. Becker, A. (2016). Student-‐generated scoring rubrics: Examining their forma-ve
value for improving ESL students’ wri-ng performance. Assessing Wri5ng, 29, 15-‐24.
4. Ferris, D.R. (2003). Response to student wri5ng. Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc.
5. Flash, P. (2005). Teaching with wri-ng: Crea-ng rubrics for wri-ng assignments. Retrieved from hip://wri-ng.umn.edu/tww/responding_grading/crea-ng_rubrics.htm
6. O’Malley, J.M. & Pierce, L.V. (1996). Authen5c assessment for English language learners: Prac5cal approaches for teachers. Longman.
7. Rubrics.com: Helping you design assessments website hip://www.rubrics.com/ 8. Stevens, D.D. & Levi, A. (2005). Introduc5on to rubrics: An assessment tool to save
grading 5me, convey effec5ve feedback, and promote student learning. Sterling, VA: Stylus Publishing, LLC.
9. Weigle, S.C. (2002). Assessing wri5ng. Cambridge: Cambridge University Press. 10. Wolf, K. & Stevens, E. (2007). The role of rubrics in advancing student learning.
The Journal of Effec5ve Teaching, 7(1), 3-‐14.
30
