Scoring Open-Ended Literacy Items .

ScoringScoringOpen-Ended Open-Ended

Literacy ItemsLiteracy Items

www.tlionline.net

Goals for the SessionGoals for the Session

Understand the importance of open-ended literacy items

Analyze prompts and rubrics for these itemsScoring tipsRelating Scoring to Classroom InstructionPractice scoring open-response reading

questionsPractice scoring writing samplesShare best practices

Importance of Open-Ended Importance of Open-Ended Literacy ItemsLiteracy Items

Reading Open-ended Questions◦1 Open Response question for each passage

◦Total of 3 for ACTAAP tests grades 3-8◦Total of 6 for Grade 11 Literacy Exam◦Account for 50% of score (24 points out of 48 for grades 3-8)

Importance of Open-Ended Importance of Open-Ended Literacy ItemsLiteracy Items

Writing Open-ended Tests◦2 questions (prompts) requiring essay-length responses

◦Account for 40 out of 48 points◦That is 83% of total writing score in grades 3-8

Importance of Open-ended Importance of Open-ended Literacy ItemsLiteracy Items

Open-ended questions account for 67% of the total literacy score

Students need to earn significant points in these items in order to achieve a proficient score

It’s critical that we score these items accurately so that the testing experience and data will be relevant in helping students improve

Few SimilaritiesFew Similarities

Reading open response questions and writing prompts have little in common

Frustrating because teachers want students to express themselves well in all written responses

Difficult for teachers and students to accommodate the differences

Reasons to ScoreReasons to Score

The two reasons to score open-ended items are often at cross-purposes1. To upload scores quickly to get data back

Scoring should be as accurate to the ACTAAP scoring as possible

With practice this can be done with a minimum of marks and time

2. To give meaningful feedback to students and parents

This scoring is more detailed, in-depth, and takes much longer

Scoring DecisionsScoring Decisions

District, or at least building, must decide whether to score interim assessments for quick, accurate scores to be uploaded for data purposes only or for meaningful in-depth scoring for feedback and reteaching

To do both will require delayed uploading, double scoring, or some creative time use

Scoring RequirementsScoring Requirements

Writing:5 domains on the state

scoring rubric are used to assess writing.

ContentStyleSentence FormationUsageMechanics Can the student

express himself in written form?

Reading: Content-specific

rubric is used to assess the reading open response question.

Scored for content only

Did the student read and understand?

Reading Open-ended Reading Open-ended ItemsItems

Open Response Reading Open Response Reading QuestionsQuestions

Each reading passage has 1 open response question

The question will generally require more than one thinking skill, even at the lower grades:◦Analysis◦Inference◦Evaluation◦Comparison

Open Response Reading Open Response Reading QuestionsQuestions

Question is tied to the passage and requires information from the passage as support

It is a more in-depth question than can be covered by a multiple-choice format question.

Short answerScored on a 0 to 4 scaleTLI continues to use a B for Blank

Examples of Open Response Examples of Open Response QuestionsQuestions

Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage.

Grade 3, 2006


Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions.

Grade 5, 2006


Compare and contrast Sylamore Horse Trail and Ozark Bicycle Trail. Use specific examples from the passage to support your response.

Grade 11, 2006

RubricsRubrics

Each open response reading question is scored by a specific rubric◦Can be subjective

Was the response thorough or only adequate?

◦A variety of tasks requested

Analysis of Open Response Analysis of Open Response Reading QuestionsReading Questions

Based on the information that you read in the passage, describe how Molly and her mother used Molly’s old crayons to make new crayons. Then, describe Molly’s new crayons. Include details from the passage.

What does “describe” really mean?How much description is enough?How many details do we include?

Analysis of Open Response Analysis of Open Response Reading QuestionsReading Questions

Use information from the passage to describe two ways in which Jesse Owens influenced people. Be sure to use details from the passage to support each of your ideas or conclusions.

Describe or tell?How many details should support each

idea?What conclusions?

Sample Rubric Sample Rubric

Score 4: The response clearly and thoroughly describes at least two ways in which Jesse Owens influenced people and supports each with relevant information from the passage. The response demonstrates comprehensive understanding of the task.

New Open Response FormatNew Open Response Format

With the 2007 test a new format was begun by the state

Old format is still in useMost questions in 2007 and 2008 did

follow the new formatContinue to use old format in practiceTLI now uses the 4 point format

New 4-point Open Response New 4-point Open Response FormatFormat

These questions are written so that there are 4 tasks to perform

The rubrics show score points equal to the number of tasks correctly performed

4—all 4 tasks are completed correctly3—any 3 of the tasks are completed

correctly2—any 2 of the tasks are completed

Analysis of New Format ExampleAnalysis of New Format Example

Explain why it is important to guard against germs in the kitchen. Identify three ways to fight against infectious diseases in relation to food and its preparation.◦Explain why◦Identify way #1◦Identify way #2◦Identify way #3

Variations in RubricsVariations in Rubrics

Most rubrics seem to accept any 3 tasks to score a 3, any 2 tasks for a 2

Occasionally a rubric will be specific about the tasks.◦For a 4, the response explains and gives 3

examples◦For a 3, the response must explain and give 2

examples (3 examples without explanation may not be acceptable)

Sample Open Response ItemsSample Open Response Items

OR Short Answers EncouragedOR Short Answers Encouraged

Fragments OKCharts or Diagrams OKLists OK“Don’t be creative.” --ADEJust answer the questionsSome questions have parts that should be

labeled. Generally at lower grades.Some questions have charts. The missing

information should be listed in the answer box.

Respond to Match the QuestionRespond to Match the Question

Some questions are difficult to answer with lists

If the question suggests discussion, sentences may be a better approach

Complete paragraph construction is never necessary

Scoring should never take anything into consideration except what is specifically called for in the rubric (either positively or negatively)

Scoring Tips for Open Response Scoring Tips for Open Response ItemsItems

Scorers should be very familiar with the passage and the question

Scorers should work the “problem” themselves, preferably more than 1 way

TLI provides possible answer informationFollow the rubric exactly, even if the

response is unexpected. Give credit if tasks are accurately performed.


Scan the response, looking for answer information

Put a check-mark in the margin for each task performed

Tasks do not have to be placed in the order requested

If question divides answer and requests labeling, no points should be removed if student does not label


Score of B—blank, no attempt to answerPull these from the stack and score

◦Analyze why Did student run out of time? Did student waste time? Was student unaware of question? Did student refuse to attempt?

Does this reveal a hidden problem?

Score 0—Student attempted an answer but earned no points

Analyze why◦Ran out of time and jotted something down◦Did not understand the question◦Was uncertain of format

For all responses that earn points, look quickly at why points were not earned◦Only 3 tasks performed? Why not the other?◦4 tasks performed, but some incorrectly?

◦What could student do to earn just one more point?

Score and share

More OR Scoring Tips More OR Scoring Tips

Do you know that open response question and rubric templates are on the TLI Web site?

The Scoring-Teaching ConnectionThe Scoring-Teaching Connection

DO teach students to analyze the Open Response prompt

DO teach the 4 types of Open Response SkillsDo expect “analysis” to be the underlying skillDO expect 4 tasks on each questionDO teach students to score their own work—

great practiceDON’T score for mechanics, sentence

formation, and the like


DO teach students to complete tasks in order

DO encourage labeling/numbering tasksDO work with students to find ways to

increase the score by 1 pointDO encourage students to look for clues

within the question DO encourage students to mark the

passage to locate answer information for the question


DO teach students to anticipate an OR question

DO teach students to write their own OR questions and rubrics

DO help students understand the difference between the reading open response and a writing assignment:

Short box= short answer= score for content only


DO teach students to consider their OR answers to be “prewriting;” do not need to develop the answer past this stage

DO teach students to use information from the passage; prior knowledge is rarely requested

DO encourage reading the questions first with students for whom that might work◦Search for answers as they read the text

Tips to ShareTips to Share

Scoring Open-Ended Literacy Items .

Documents

Transcript of Scoring Open-Ended Literacy Items .