Summative Assessment: Rubrics and Tests Effective Teaching and Learning English Study Program FKIP...
-
Upload
cora-warren -
Category
Documents
-
view
237 -
download
0
Transcript of Summative Assessment: Rubrics and Tests Effective Teaching and Learning English Study Program FKIP...
Summative Assessment:Summative Assessment:Rubrics and TestsRubrics and TestsEffective Teaching and LearningEffective Teaching and Learning
English Study ProgramEnglish Study Program
FKIP – UNSRIFKIP – UNSRI
AprilApril 201 20144
11
OutcomesOutcomes
Apply a systematic process for creating Apply a systematic process for creating a test blueprinta test blueprint
Identify attributes of effective test Identify attributes of effective test questionsquestions
Explain the advantages and Explain the advantages and disadvantages of different types of test disadvantages of different types of test questions questions
Assess the quality of tests and test itemsAssess the quality of tests and test items Create samples of effective questionsCreate samples of effective questions
22
What type of What type of assessment?assessment? Procedural knowledgeProcedural knowledge Declarative knowledgeDeclarative knowledge
33
Test Writing ProcessTest Writing Process
Planning the testPlanning the test Writing test itemsWriting test items Selecting test itemsSelecting test items Formatting the testFormatting the test Assessing the testAssessing the test Revising the testRevising the test Using the testUsing the test After the testAfter the test
44
Planning the TestPlanning the Test
Content BlueprintContent Blueprint– Learning outcomesLearning outcomes– WeightWeight
LengthLength– Types of itemsTypes of items– Number of itemsNumber of items
55
Test BlueprintTest Blueprint
Learning outcomes Weight
Apply a systematic process for creating a test blueprint
Identify attributes of effective test questions
Explain the advantages and disadvantages of different types of test questions
Assess the quality of tests and test items
Create samples of effective questions
66
Types of ItemsTypes of Items
RecognitionRecognition– True-falseTrue-false– Multiple-choiceMultiple-choice– Multiple-answerMultiple-answer– MatchingMatching– OrderingOrdering
RecallRecall– Short AnswerShort Answer– CompletionCompletion– EssayEssay
77
Average Response Average Response TimeTimeItem Type
True-false 30 seconds
Multiple-choice and Multiple-answer
60 – 90 seconds
Matching and Ordering 30 seconds per response
Short Answer 120 seconds
Completion 60 seconds
Essay 10 – 30 minutes88
Writing Test ItemsWriting Test Items
Simple and direct wordingSimple and direct wording Avoid jargonAvoid jargon Avoid trivia itemsAvoid trivia items Match items to learning outcomesMatch items to learning outcomes Each item has an agreed upon Each item has an agreed upon
correct answercorrect answer Write more questions than you Write more questions than you
will needwill need99
Multiple-choice ItemsMultiple-choice Items
StemStem– Direct questionDirect question– Incomplete statementIncomplete statement
ResponsesResponses– One correct answerOne correct answer– Multiple distractersMultiple distracters
1010
StemStem
Clearly worded Clearly worded One ideaOne idea Avoid the use of negativesAvoid the use of negatives Enough information to answer the Enough information to answer the
questionquestion Direct questions preferredDirect questions preferred Blanks at the end of the stemBlanks at the end of the stem Include words repeated in all responsesInclude words repeated in all responses
1111
ResponsesResponses
3-5 per item3-5 per item Avoid “all of the above” and “none of Avoid “all of the above” and “none of
the above”the above” Grammatically correct with stemGrammatically correct with stem Similar length and structureSimilar length and structure Avoid absolute wordsAvoid absolute words Listed in a logical orderListed in a logical order Mutually exclusive and not overlappingMutually exclusive and not overlapping
1212
DistractersDistracters
PlausiblePlausible Common misconceptionsCommon misconceptions Logical misinterpretationsLogical misinterpretations ClichésClichés Partial answersPartial answers Technical terms or jargonTechnical terms or jargon
1313
ExampleExample
What is the minimum number of What is the minimum number of responses for a multiple-choice responses for a multiple-choice item?item?
A) 2A) 2
B) 3B) 3
C) 4C) 4
D) 5D) 5
1414
Application ExampleApplication Example
What problem exists in the following What problem exists in the following multiple-choice stem:multiple-choice stem:
________ is the most common type of test item.________ is the most common type of test item.
A)A) Absolute words should be avoided in the Absolute words should be avoided in the stem.stem.
B)B) The stem contains more than one idea or The stem contains more than one idea or concept.concept.
C)C) Not enough information is presented to Not enough information is presented to answer the question.answer the question.
D)D) The fill-in the blank should come at the end.The fill-in the blank should come at the end.
1515
Analysis and Analysis and Evaluation Example Evaluation Example StemStemAn instructor was asked the following question: "Briefly list and
explain how you develop a test.” As an answer, this instructor wrote the following:
I begin by going through the chapter and writing questions based on the material in the text and my lectures. Then I decide how many questions I want and select the best questions from the list that I have developed. I format the test and add instructions. After a few days, I review the questions and make any revisions that need to be done and remove any jargon or wording lifted directly from the text. Then I use the test in class. Based on how the class does, I may make changes for the next time I teach the class.
1616
Analysis and Analysis and Evaluation Example Evaluation Example ResponsesResponsesBased on the process described in “Effective Based on the process described in “Effective Classroom Tests,” how would you judge this Classroom Tests,” how would you judge this answer?answer?
A) EXCELLENT (all steps in the right order with correct, A) EXCELLENT (all steps in the right order with correct, clear, and complete descriptions)clear, and complete descriptions)
B) GOOD (all stages correct in the right order, but the B) GOOD (all stages correct in the right order, but the descriptions are not as complete as they should be).descriptions are not as complete as they should be).
C) MEDIOCRE (one or two stages are missing, OR the C) MEDIOCRE (one or two stages are missing, OR the stages are in the wrong order, OR the explanations stages are in the wrong order, OR the explanations are not complete, OR the explanations are are not complete, OR the explanations are irrelevant)irrelevant)
D) UNACCEPTABLE (one or more stages are missing D) UNACCEPTABLE (one or more stages are missing AND the explanations are not complete AND/OR they AND the explanations are not complete AND/OR they are irrelevant) are irrelevant)
1717
Poor Question 1Poor Question 1
Good multiple choice items:Good multiple choice items:A) are easy to writeA) are easy to write
B) can only test memorized contentB) can only test memorized content
C) are better than essay itemsC) are better than essay items
D) there is no such thingD) there is no such thing
E) can test a wide range of contentE) can test a wide range of content
1818
Poor Question 2Poor Question 2
Which of the following characteristics Which of the following characteristics is not true of completion test items is not true of completion test items but is an important distinguishing but is an important distinguishing attribute of matching tests, multiple-attribute of matching tests, multiple-choice questions, and true-false choice questions, and true-false items?items?
A) They are objective test items.A) They are objective test items.B) They require knowledge recognition B) They require knowledge recognition
but not production.but not production.C) Much more difficult to construct.C) Much more difficult to construct.
1919
Poor Question 3Poor Question 3
Which of the following statements is Which of the following statements is FALSE?FALSE?A) Misfeasance is the improperly doing of an A) Misfeasance is the improperly doing of an
illegal act.illegal act.
B) Nonfeasance is improperly doing a legal act.B) Nonfeasance is improperly doing a legal act.
C) Nonfeasance is the failure to do an act that C) Nonfeasance is the failure to do an act that one must do legally.one must do legally.
D) Misfeasance is the failure to PROPERLY do D) Misfeasance is the failure to PROPERLY do an act that one has a duty to perform.an act that one has a duty to perform.
E) None of the above.E) None of the above.
2020
Poor Question 4Poor Question 4
__________ is/are the best method to __________ is/are the best method to determine if students have learned determine if students have learned something.something.
A) Comprehensive ExamA) Comprehensive Exam
B) Homework AssignmentsB) Homework Assignments
C) Pop QuizzesC) Pop Quizzes
D) Research PaperD) Research Paper
2121
Selecting Test ItemsSelecting Test Items
Outcome Weight x # Questions by Outcome Weight x # Questions by Type = #Questions of Each Type for Type = #Questions of Each Type for
OutcomeOutcome
2222
Formatting the TestFormatting the Test
Group items by typeGroup items by type Sort items by increasing difficultySort items by increasing difficulty Add instructionsAdd instructions Review layout and paginationReview layout and pagination Write answer keyWrite answer key
2323
Assessing the TestAssessing the Test
SelfSelf– 2-3 days after writing the test2-3 days after writing the test– ClarityClarity– Clues in items to other itemsClues in items to other items
Non-expertNon-expert– ClarityClarity– Contextual cluesContextual clues
PeerPeer– ContentContent– Weighting to outcomesWeighting to outcomes– Answer keyAnswer key
StudentsStudents– ClarityClarity– ContentContent
2424
Test Taking Test Taking ProceduresProcedures Use of notes or other materialsUse of notes or other materials Time limitsTime limits
2525
After the TestAfter the Test
Item AnalysisItem Analysis Areas for reviewAreas for review Test revisionsTest revisions
2626
ActivityActivity
Write two test questions on any Write two test questions on any topictopic
One question should be an One question should be an example of a good test itemexample of a good test item
One question should be an One question should be an example of a poorly written test example of a poorly written test itemitem
2727
ShareShare
Share your two questions with a Share your two questions with a partnerpartner
Can they determine which is good Can they determine which is good and which is bad?and which is bad?
Can they explain what makes one Can they explain what makes one poorly written?poorly written?
As a team, how can you fix the As a team, how can you fix the poorly written questions?poorly written questions?
2828
OutcomesOutcomes
Determine what characteristics are Determine what characteristics are important in evaluating student workimportant in evaluating student work
Evaluate rubrics, analytic scales, and Evaluate rubrics, analytic scales, and other evaluation methodsother evaluation methods
Describe the contents of a good Describe the contents of a good rubricrubric
Identify rubrics already in use at Identify rubrics already in use at Baker CollegeBaker College
Begin work on a rubric for a classBegin work on a rubric for a class3030
What is a Rubric?What is a Rubric?
A rubric is a scoring tool or guide that A rubric is a scoring tool or guide that lists the specific criteria and the lists the specific criteria and the ranges for multiple levels of ranges for multiple levels of achievement for a piece of work or achievement for a piece of work or performance. A rubric consists of a set performance. A rubric consists of a set of well-defined factors and criteria of well-defined factors and criteria describing the dimensions of an describing the dimensions of an assignment to be assessed or assignment to be assessed or evaluated.evaluated.
3131
Parts of a RubricParts of a Rubric
Scale (columns)Scale (columns) Dimensions (rows)Dimensions (rows) Criteria descriptions (cells)Criteria descriptions (cells)
3232
Benefits of RubricsBenefits of Rubrics
Communicates the instructor’s Communicates the instructor’s expectations expectations
Streamlines the process for Streamlines the process for feedback to the studentfeedback to the student
Facilitates equitable grading Facilitates equitable grading Standardizes assessment across Standardizes assessment across
different instructorsdifferent instructors
3333
Uses for RubricsUses for Rubrics
PapersPapers PresentationsPresentations ProjectsProjects EssaysEssays HomeworkHomework
Case StudiesCase Studies Participation/Participation/
Class DiscussionClass Discussion PortfoliosPortfolios
3434
Types of RubricsTypes of Rubrics
AnalyticAnalytic– Page 11 Page 11
HolisticHolistic– Page 12 and 13Page 12 and 13
Check ListCheck List– Page 14Page 14
Scoring GuideScoring Guide– Page 15Page 15
3535
Creating a RubricCreating a Rubric
Identify components/outcomes of the Identify components/outcomes of the assignmentassignment
Determine a scaleDetermine a scale Add criteriaAdd criteria Assign pointsAssign points Set component weights (optional)Set component weights (optional) Assess the rubricAssess the rubric Test and reviseTest and revise
3636
ActivityActivity
Split into groups of 3-4 Split into groups of 3-4 Determine team rolesDetermine team roles Select an assignment that needs Select an assignment that needs
a rubrica rubric– Can be a specific assignment, such Can be a specific assignment, such
as a research paper for ENG 102as a research paper for ENG 102– Can be of a more general nature Can be of a more general nature
such as a class presentationsuch as a class presentation3737
Step 1: Identify Step 1: Identify ComponentsComponents List 5 major objectives/outcomes List 5 major objectives/outcomes
of the assignmentof the assignment Write these items as the row Write these items as the row
headers of the sheet providedheaders of the sheet provided
3838
Step 2: Determine a Step 2: Determine a ScaleScale Aim for 3-5 levelsAim for 3-5 levels Can use an odd or even number Can use an odd or even number
of itemsof items Use the headings on the next Use the headings on the next
slide for ideasslide for ideas Write these as column headings Write these as column headings
on the sheet providedon the sheet provided
3939
Potential Column Potential Column HeadingsHeadings Outstanding | Accomplished | Proficient | Developing | Outstanding | Accomplished | Proficient | Developing |
BeginningBeginning Accomplished | Average | Developing | BeginningAccomplished | Average | Developing | Beginning Excellent | Good | Needs Improvement | UnsatisfactoryExcellent | Good | Needs Improvement | Unsatisfactory Exceptional | Acceptable | Marginal | UnacceptableExceptional | Acceptable | Marginal | Unacceptable Expert | Practitioner | Apprentice | NoviceExpert | Practitioner | Apprentice | Novice Professional | Adequate | Needs Work | You’re FiredProfessional | Adequate | Needs Work | You’re Fired Exceeds Expectation | On Target | BeginningExceeds Expectation | On Target | Beginning Exemplary | Competent | DevelopingExemplary | Competent | Developing High | Medium | LowHigh | Medium | Low Outstanding | Proficient | Shows PotentialOutstanding | Proficient | Shows Potential
4040
Step 3: Add CriteriaStep 3: Add Criteria
Create descriptions for each level Create descriptions for each level of performance for each criteria in of performance for each criteria in the cells of the rubricthe cells of the rubric– Bullet pointsBullet points– ParagraphsParagraphs
Write these criteria in the cells of Write these criteria in the cells of the sheet provided the sheet provided
4141
Step 4: Assign PointsStep 4: Assign Points
Assign points for each level of Assign points for each level of performanceperformance
Can use either of the following:Can use either of the following:– Discrete values (5, 4, 3, 2, 1) Discrete values (5, 4, 3, 2, 1) – Ranges (10-9) for each levelRanges (10-9) for each level
Indicate the point value on the Indicate the point value on the sheet provided sheet provided – Normally placed with the scaleNormally placed with the scale
4242
Step 5: Set Component Step 5: Set Component WeightsWeights Allows for different levels of Allows for different levels of
importanceimportance– Spelling/grammar – more or less important Spelling/grammar – more or less important
than content?than content? Determine if weights are necessary for Determine if weights are necessary for
your rubricyour rubric Assign weights accordinglyAssign weights accordingly
– See example on page 17-18 of the See example on page 17-18 of the handouthandout
4343
Step 6: Assess the Step 6: Assess the RubricRubric Assess your rubric using a metarubricAssess your rubric using a metarubric
– See examples on page 11-15 of your See examples on page 11-15 of your handouthandout
Conduct a peer reviewConduct a peer review– Ask one or two other instructors to review Ask one or two other instructors to review
your rubricyour rubric Provide time for student reviewProvide time for student review
– Allow students to ask questions and make Allow students to ask questions and make commentscomments
4444
Group ProjectGroup Project
Trade rubrics with another groupTrade rubrics with another group Assess the rubric using a Assess the rubric using a
metarubric from page 11-15metarubric from page 11-15
4545
DiscussionDiscussion
What metarubric(s) did you use? What metarubric(s) did you use? Why?Why?
What did you see on the other What did you see on the other team’s rubric that you liked?team’s rubric that you liked?
Could you understand the Could you understand the assignment easily by reviewing assignment easily by reviewing the rubric?the rubric?
4646
Step 7: Implement and Step 7: Implement and refinerefine Refine your rubric based on Refine your rubric based on
feedback from other instructors feedback from other instructors and studentsand students
Make notes each time you use Make notes each time you use the rubric for continuous the rubric for continuous improvement purposesimprovement purposes
Share with othersShare with others
4747
Rubric Reliability & Rubric Reliability & ValidityValidity ReliabilityReliability
– ““the likelihood that a given measurement the likelihood that a given measurement procedure will yield the same description procedure will yield the same description of a given phenomena if the measurement of a given phenomena if the measurement is repeated.”is repeated.”
ValidityValidity– ““the extent to which a specific the extent to which a specific
measurement provides data that relate to measurement provides data that relate to commonly accepted meanings of a commonly accepted meanings of a particular concept.”particular concept.”
Babbie, 1986Babbie, 19864848
Reliability RequiresReliability Requires
Instructor should reach same Instructor should reach same conclusion each timeconclusion each time
Different instructors should reach Different instructors should reach similar conclusion (interrater similar conclusion (interrater reliability)reliability)
4949
Interrater ReliabilityInterrater Reliability
Independently score a set of student Independently score a set of student samplessamples
Review responses for consistent and Review responses for consistent and inconsistent responsesinconsistent responses
Discuss and reconcile inconsistenciesDiscuss and reconcile inconsistencies Repeat with second group of samplesRepeat with second group of samples
Maki, 2004, p. 127Maki, 2004, p. 127
5050
Validity RequiresValidity Requires
ReliabilityReliability Comprehensiveness Comprehensiveness
– Cover all outcomes Cover all outcomes EconomyEconomy
– Space is usually limited, so be Space is usually limited, so be selective about what goes into the selective about what goes into the rubricrubric
Balanced scoring and weightingBalanced scoring and weighting5151