Testing to Evaluate Learning New Faculty Orientation 2005 Dr. Diane Wink Professor School of Nursing...
-
Upload
lydia-dinah-hardy -
Category
Documents
-
view
214 -
download
0
Transcript of Testing to Evaluate Learning New Faculty Orientation 2005 Dr. Diane Wink Professor School of Nursing...
Testing to Evaluate Testing to Evaluate LearningLearning
New Faculty Orientation New Faculty Orientation 20052005Dr. Diane Wink
ProfessorSchool of Nursing
College of Health and Public Affairs
©Diane Wink 2005
Why test??Why test??
Achievement of objectives of learning measured with
evaluationClass activities must reflect objectives and what will be
testedClass
Experience
and Activities
ObjectivesObjectives
• All objectives should lead directly to evaluation through the learning experience itself.
• Evaluation should be of the objectives.
Class
Experience
and
Activities
Questions to askQuestions to ask
• What is the best evaluation method to determine student mastery and application of this knowledge?
Questions to ask - Test Questions to ask - Test AlternativesAlternatives
• Term paper• Class activity
– Debate and bib development
• Problem set completion• Service Learning activity• Course project
– Case Study– Development of tool
• Non written evaluation– Performance of the
competency being developed
For Testing: Questions to For Testing: Questions to askask
• How will this test help me know if learning objectives are being met?
• Is the test addressing essential information?
• Does the test reflect the readings, class activities and/or web discussion?
Questions to askQuestions to ask
• How many questions will it take to determine if student has met objectives/ has the knowledge?
Do a test blueprint!
For all types of examsFor all types of exams
• Have a colleague review your items
• Consider the possibility of cheating– Alternative forms of tests– Random seating– 3X5 key points card– New exams– No cell phones, PDAs– Open book if on-line– In person for on-line
Multiple Choice ItemsMultiple Choice Items
• The stem should:– Contain a self-contained question or
problem– Present as much of the item's content
as possible– Be precise - no extraneous
information
Multiple Choice ItemsMultiple Choice Items
• The stem should:– Be independent of all other stems.
If part of a situation, say this clearlyDo not make answer dependent on
getting a prior item correct
– Be a positive rather than a negative statement or question
Should not be a reading test!
Multiple Choice ItemsMultiple Choice Items
• The answer choices should:– Contain only one best or correct answer– Be grammatically consistent with stem– Be similar in complexity, detail, length
and plausibility– Avoid "all of the above" or "none of the
above"
Multiple Choice ItemsMultiple Choice Items
• The answer choices should:– All be plausible– Be arranged logically
Alphabetical, numerical, length
– Randomly use each alternative position for correct answers
– Contain action verbs from Bloom's Taxonomy
Multiple Choice ItemsMultiple Choice Items
• The answer choices should:– Be on the same page as the stem – Not have obscure or esoteric
language
Sample Item- PoorSample Item- Poor
If all of the following heart valves are defective but only one can be repaired, which should it be?
a. Aorticb. Mitralc. Pulmonicd. Tricuspid
Sample ItemSample Item
Narrowing and tightness of which heart valve is most likely to result in left ventricular enlargement?
a. Aorticb. Mitralc. Pulmonicd. Tricuspid
True False ItemsTrue False Items
• State the item as simply and clearly as possible
• Keep statements equal in length• Write each item so that it is clearly
true or false
True False ItemsTrue False Items
• Avoid qualifications such as "often" or "usually"
• Randomize the true and false items
• Test important information and facts
Sample T/F Item Sample T/F Item (Hard to score)(Hard to score)
15. For each of the following, choose A if true and B if false.
a. The major pump chamber of the heart is the ventricle
b. Valves control blood flow into the atriac. The pulmonary artery carries fully
oxygenated blood to the lungs
Sample T/F Item Sample T/F Item (Easy to score)(Easy to score)
For each of the following, choose A if true and B if false.
15. The major pump chamber of the heart is the left ventricle
16. Valves control blood flow into the atria17. The pulmonary artery carries fully
oxygenated blood
Matching ItemsMatching Items
• Put the basis of relationship for matching in the directions
• Put the entire set of matching items on a single page
• Arrange items in a systematic order• Number premises on the left and
letter the responses on the right a-eAllows computer grading
Matching ItemsMatching Items
• Use a larger or smaller set of responses and allow responses to be used more than once or not at all
• Use homogeneous lists that concern the same content
• Number each item separately to allow computer grading
Sample Matching Item - Sample Matching Item - PoorPoor
31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once.
Hoover a. Implemented emancipation proclamation
Roosevelt b. Established the CCCLincoln c. Major war ends Clinton d. Stock market crash
f. Was impeached
Sample Matching Item - Sample Matching Item - PoorPoor
31. Match each item in Column A with an historical event listed in Column B. Events in Column B can be used more than once.
Hoover a. Implemented emancipation proclamation
Roosevelt b. Established the CCCLincoln c. Major war ends Clinton d. Stock market crash
f. Was impeached
All questions (there are actually four) have same item number!
Confusion over which Roosevelt, which stock market crash?
Sample Matching - BetterSample Matching - BetterMatch each of the presidents listed in Column A with the historical
event listed in Column B. Events in Column B can be used more than once.
Column A Column B31. Hoover a. Post civil way reconstruction in
progress in US32. F.D. Roosevelt b. Establishment of the
CCC33. Lincoln c. WWII ends34. Clinton d. Stock market crashed starting depression
f. Impeached
Exam grading support at Exam grading support at UCFUCF
University testing service support for multiple choice/true false/matching exams
WebCTDisability
Services
Essay/Short Answer ItemsEssay/Short Answer Items
What do they test?Higher cognitive levels
ApplicationAnalysisSynthesisEvaluation
Not for simple recall of facts
Essay/Short Answer ItemsEssay/Short Answer Items
• First ask– Is item clearly written? – If extended response needed, is it too
broad? – If restricted response needed, would
the content be assessed better with an objective item?
Sample ItemSample Item
• Poor: List the components of a persuasive letter to a government official.
• Better: Write a letter to a government official in support of or opposed to gun control.
Sample ItemSample Item
• Poor: What are the themes in War and Peace?
• Better: Relate a theme in War and Peace to world events in 2005
Grading essay/short essay Grading essay/short essay itemsitems
Write out the answer Does your answer
match the intent of the item?
What problems did you encounter in answering the item?
Can the student answer in the time allowed?
Grading essay/short answer Grading essay/short answer itemsitems
• Analytical method• Points for each essential element
• Holistic method• Look for essential elements• Evaluate overall quality of response• Little feedback
Analytical Grading Method Analytical Grading Method
Identify criteria for evaluating the content of the response and assign points for each:– Essential elements required– Accuracy of content– Logical organization– Clarity– Depth and complexity of analysis– Use of references and specific citations ** Usually only used in formal papers
Analytical Grading MethodAnalytical Grading Method
Decide on the extent to which writing ability and use of style format will be included in the scoring– Spelling– Grammar– Sentence structure– Writing style– APA citation and format ** Usually only used in formal papers
Holistic Grading MethodHolistic Grading Method
Sample “A” answer:
Covers all key points with citations to major resources. Clearly written with analysis of problem clear and persuasive. Minimal to no grammatical or spelling errors.
Holistic Grading MethodHolistic Grading Method
Sample “B” answer:Covers most key points with citations to most major resources. Clearly written with good analysis of problem. Few grammatical or spelling errors.
Sample “C” answer:Covers few key points with few references to major resources. Written poorly with limited analysis of problem and a non persuasive argument Multiple grammatical or spelling errors.
Holistic Grading MethodHolistic Grading Method
Sample “D” Answer:Omits most key points with few/no citations to resources. Writing difficult to follow with consistent grammatical and spelling errors
Sample “F” answer:Omits all key points, does not answer question
Grading essay and short Grading essay and short answer itemsanswer items
Before scoring Read a random sample of papers Get a sense of how students answered Revise evaluation criteria if needed Review grading criteria with any other graders -
practice!!! Do key for commonly made comments (give to
students and all graders)1. Great job. Shows excellent analysis of your main points 2. Great ideas but needed to be developed more3. Unclear what you mean 4. Needs citation
Grading essay and short Grading essay and short answer itemsanswer items
Ideal grading technique:– Score answers to one question at a
time for all students– Read each answer twice
First for omissions of essential elementsSecond reread and score
Grading essay and short Grading essay and short answer itemsanswer items
• Ideal grading technique:– Cover up prior scores to items or
record them on a separate answer sheet
– Read papers anonymously; have students use a code number
Grading essay and short Grading essay and short answer itemsanswer items
• If fatigued, stop and rest!
Grading essay/short answer Grading essay/short answer itemsitems
•GTA•Help from peers
Consider time for grading when designing exams
Bloom’s TaxonomyBloom’s Taxonomy
CognitiveKnowledge
Identification and recall of information
ComprehensionOrganization and selection of facts and
ideas
ApplicationUse of facts, rules principles
Bloom’s TaxonomyBloom’s Taxonomy
Cognitive (continued)Analysis
Separation of a whole into component parts
SynthesisCombination of ideas to form a new whole
EvaluationDevelopment of opinions, judgments, or
decisions
Bloom’s TaxonomyBloom’s Taxonomy
Affective DomainFeeling toneEmotionDegree of acceptance or rejection
Psychomotor Domain
Resources UsedResources UsedOermann, M. (1999). Developing and scoring essay tests. Nurse
Educator, 24(2), 29-32.Tomey, A. (1999). Preparing a testing blueprint from levels of
course objectives. Nurse Educator, 24(4), 11-12.Tomey, A. (1999). Selected response test items. Nurse Educator,
24(5), 9-10.Tomey, A. (1999). Completion testing items. Nurse Educator,
24(6), 6-7.