Post on 16-Feb-2016
description
Item Development
Presented to: Majmaah UniversityNeil Wilkinson
Creating better test items
The FACTS of good test items
• Face validF• AccurateA• Consistent with current practiceC• Test blueprint representativeT• Statistically appropriateS
Creating good test items
Right Expertise
Sound Processes
Efficient Tools xxx
Right Expertise
Sound Processes
Efficient Tools xxx
Subject matter expertise
• ensures test is valid
Content development expertise
• ensures test is superiorly crafted
Psychometric expertise
• ensures test is reliable and fair
Right Expertis
e
Write items
Review & approve
items
Field test items
Review of field test results
Gap analysis of item bank
Right Expertise
Sound Processes
Efficient Tools xxxSound
Processes
Sound item writing processes
Assignments
• Gap analysis
Training
• Good item writing practices
• Style guidelines
• Cultural sensitivity
• Intellectual property rights
• Confidentiality
Review procedures
• Editorial review
• Content review
• Acceptance criteria
Item performance monitoring
• Feedback mechanisms
Item performance analysis and feedback
Why analyze items?
• Statistical behavior of “bad” items is fundamentally different from that of “good” items
• Provides quality control indicating items which should be reviewed by content experts
• Feedback on how items are performing can be used in future item writing efforts providing continuous quality improvement in the item writing process
Distractor analysis High-scoring candidates should select the correct optionLook at discrimination for each distractor Low-scoring candidates should select from among the
distractors at random Look at p-values for each distractor
Option: AKEY
B C D E Total
N 241 9 3 11 70 334Mean 219.63 215.11 196.67 210.73 210.03
p-value 0.72 0.03 0.01 0.03 0.21PtBis +0.22 -0.02 -0.10 -0.06 -0.19
Diagnosing potential problems
Option: A B CKEY
D E Total
N 2 2 113 23 194 334
Mean 233.00 177.00 215.08 215.48 218.54
p-value 0.01 0.01 0.34 0.07 0.58
PtBis +0.06 -0.16 -0.07 -0.02 +0.09
Feedback into the item development process
• Sharing performance statistics to authors of individual items may be problematic due to time between authoring and use, but...
• Item performance statistics can still inform the item development process.
• Share and discuss item performance and reasons for poorly performing items.
• Show examples of good, average, and poorly performing items.
Item Bank A repository of items with their associated metadata & historical record
Right Expertise
Sound Processes
Efficient Tools xxxEfficient Tools
Item Bank – Data• The item itself• Item author• Item type• Content area reference from test blueprint• Cognitive level• Revision history• Item status• Performance statistics
Right Expertise
Sound Processes
Efficient Tools xxxEfficient Tools
Item Bank – Features
Item authoring• Remote item authoring• Support of multiple item
types
Item banking• Search capabilities• Import/export capabilities• Batch editing capabilities
Right Expertise
Sound Processes
Efficient Tools xxxEfficient Tools
Test construction• Test assembly• Export features
Ancillary features• Security and access • Workflow management• Project tracking
Guidelines for an effective item development process
An effective item development process begins well before the first test item is written with development of test blueprints.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Carefully consider the associated information that is required and stored for each item.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Recognize that item development requires more than subject matter expertise.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Item authors require training in item writing.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Item development requires more than authoring, sound review processes are also essential.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Style and grammar guidelines and review processes are necessary to ensure items are testing what they are meant to test.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Guidelines for an effective item development process
Use item performance analysis to guide further review of the items and to provide feedback to item authors.
Right Expertise
Sound Processes
Efficient Tools
Effective Item
Development
Writing Good Multiple-Choice Test Items
What Makes a Content-Appropriate Item
• Reflects specific content• Able to be classified on the test blueprint• Does not contain trivial content• Independent from other items• Not a trick question• Vocabulary suitable for candidate population• Not opinion-based
Source: Haladyna, T.M. (1996). Writing Test Items to Evaluate Higher Order Thinking. Boston, USA: Allyn & Bacon.
Anatomy of a Multiple-Choice Item
Which city has been awardedthe 2016 Summer Olympics?
A. ChicagoB. MadridC. Rio de JaneiroD. Tokyo
Stem
Answer Options
Key
Dis
tract
ors
Not opinion based
Avoid the use of opinion based material in items• Poor Stem:
Which are the best football team in the world?a. Barcelonab. Manchester United.c. Bayern Munichd. Athletico Madrid
• Better Stem:Which football team won the European championship in 1999?
Guidelines for Writing the Stem
Place most of the phrasing in the stem.• Poor Stem:
Type II diabetes isa. also called juvenile-onset.b. characterised by insulin dependency.c. primarily seen in adults over 40.d. often managed by drug therapy.
• Better Stem: What is the most frequent age of onset for Type II diabetes?
Guidelines for Writing the Stem
Avoid excessive, unnecessary wording.• Poor Stem:
Certified individuals are required to undertake continuing education. What is the minimum number of hours required in a two-year period?
a. 10b. 15c. 20d. 25
• Better Stem:How many hours of continuing education must a
certified individual take in a two-year period?
Guidelines for Writing the Stem
Clearly describe the problem or what is being asked of the candidate.• Poor Stem:
Continuing education isa. required every two years.b. important to life-long learning.c. an optional activity.d. undertaken at certified centers.
• Better Stem:How many hours of continuing education is a certified individual required to undertake in a 5-year cycle?
• Maintain similarity in the answer options.• Keep the answer options relatively equal in
length.• Keep all answer options grammatically correct
with the stem and parallel.• Avoid cueing the right answer.• Avoid words such as “always” and “never”.• Avoid repetitive wording in the answer options.• Vary location of key.• Place answer options in logical order.
Guidelines for the Answer Options
Four is most common.• Recent research indicates three may be sufficient
if the two distractors are strong and plausible.• Rarely have three functional distractors.• “Using more options does little to improve
item and test score statistics and typically results in implausible distractors.”*
Number of Answer Options
*Rodriquez, M.C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13 (p. 11).
Ten steps to an Effective Item Writing Process
Use item bank inventories to direct item writing activities.
Ten Steps to an Effective Item Writing Process
Provide a training program for item writers.
Ten Steps to an Effective Item Writing Process
Develop a style guide.
Ten Steps to an Effective Item Writing Process
Require all items to be classified according to the test blueprint.
Ten Steps to an Effective Item Writing Process
Ensure all items are validated with current references and the validations are recorded.
Ten Steps to an Effective Item Writing Process
Editorially review all items for grammar and style.
Ten Steps to an Effective Item Writing Process
Include an item review process by Subject Matter Experts (SMEs).
Ten Steps to an Effective Item Writing Process
Share acceptance criteria with item writers and item reviewers.
Ten Steps to an Effective Item Writing Process
Use statistical analysis of items to guide further review.
Ten Steps to an Effective Item Writing Process
Provide feedback to item writers.
Ten Steps to an Effective Item Writing Process
Thank You!