Cets 2013_ gander hale_write_test_questions_session

WRITING TEST QUESTIONS FOR

ANY TEST ENGINE

Sharon Gander, M.Ed., CPT

Judith Hale, Ph.D., CPT

OVERVIEW

Overview and presenter introductions

Test engine challenges

Conditions & scenarios

Questions and responses

Quality Distractors

Quality checks

Test Engine Debrief

THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG) 6

TAKE TIME TO:1 – OPEN A WORD PROCESSOR

2 – OPEN OR DOWNLOAD A TEST ENGINE

3 – DRAFT 2-3 TEST QUESTIONS ON ANY TOPIC

4 – CHECK THAT YOU HAVE RECEIVED THE HANDOUTS

3 – WORK WITH A PARTNER ON ACTIVITIES – FIND YOUR PARTNER NOW!

Prepare for this session by doing the steps above!

OBJECTIVES1. Write a one-paragraph scenario opening to a question.

2. Write a question with one right answer.

3. Write three plausible answers and one correct answer to the posed question.

4. Identify strategies for validating the preferred response to posed question.

5. Check answers for “tells” that give away the right or wrong answer.

6. Explain the value of using Plain Language and Global English in writing test questions.

7. Leave with three tools to apply in writing test items for learning or certifications.

8. Recognize test engine characteristics that impact item design, delivery and validation.

9. Prototype one or more questions in a free test-engine.


Immediately after this session, begin building scenario-based questions for courses, certifications and test-outs as opportunities arise.

TEST ENGINE CHALLENGES

What challenges do you experience in writing test items for online test-

engine delivery?

TEST ENGINE CHALLENGES

Developer familiarity with test engine while writing

The need to write clear questions and responses

It is difficult to get valid content

Need to overcome the strong tendency to writerecall (knowledge/facts)

Difficult to write questions that make sense to the test candidate

Continuous error checking dulls the senses

Difficult to use appropriate question type


CONDITIONS AND SCENARIOS

How do you create performance-based test questions that work

in any test engine?

PERFORMANCE-BASE QUESTIONS

The heart of a performance-based question is a condition.

You could use:

Given a specific situation and resources…

Or

Describe the situation in 1-2 paragraph story(a scenario)


Given the fact that test engines and survey engines handle question sets and answer evaluation very differently and the fact that you need to produce quality assessment questions as quickly as possible.

Which two question types are most standard across all engines?

a) True/false and multiple-choice.

b) Fill-in-the-blank and ordering.

c) Multi-select and ranking.

d) Essay and matching.

13THE INSTITUTE FOR PERFORMANCE IMPROVEMENT (WWW.TIFPI.ORG)

SOAR – A SCENARIO TECHNIQUE

Situation – essential facts of the setting

Obstacles – the hurdles that will prevent action or require unique actions; problems and challenges

Action – the key decisions and/or action steps are required

Result – the desired outcome


ACTIVITY #1: PREPARE QUESTION

1. Think of a situation of your own (or use the one below).

2. Describe the conditions using your SOAR template

3. Work with partner – 3 minutes

Write for the action. What action will overcome the obstacle to gain the desired result


Can’t think of a situation? Try this:

You are tasked with assessing SME’s skills for writing content. Your organization wants SMEs to apply adult learning theory to content that will be used in e-learning solutions. Describe a scenario that will assess this skill.

QUESTIONS AND RESPONSES

Why use multiple-choice questions with only one correct response?

WHY ONE CORRECT ANSWER?

Clarity Decreases time readers spend figuring out what

the question is Decreases the time spent guessing at what the

question writer intended

Defensibility Can be supported by unambiguous documented

references Provides clear evidence of knowledge when

answering knowledge-type questions Provides clear evidence of performance when

answering performance-based questions

Technology Scoring and tracking answers Works with every test engine


WHAT IS A CLEAR QUESTION? One that gives the reader clear signals about “what to answer” by:

Starting with a question word (5Ws/H) Who, what, when, where, why, or how

Ending with a question mark Avoiding the hanging question that results from

using a colon that leaves the reader trying to read your mind

Using only one verb

Using action verbs Using either the present or past tense

Avoiding the passive “is” “was”

Using either an actor, “you” or implied “you”


ACTIVITY #2: WRITE A QUESTION

1. Using your SOAR scenario and the item writing template

2. Write a question with ONE right answer

More about distractors and incorrect responses coming soon.



Write for the action.

Find a way to ask, what action will overcome an obstacle to gain the desired result?

WHAT IS A QUALITY DISTRACTOR

Should one response be obviously wrong?

Should I use the “ none of the above” or “all of the above responses?

Should I use answers like “a and c only”?

QUALITY DISTRACTORS

Are plausible – all of them are plausible Real and real-life

Force discrimination Requires test-taker to evaluate options

Separate the less experienced from the highly experienced

Requires test-taker to have experience (and education) to choose the most effective response

Technically accurate May distract with variance in technical details

Forces accuracy in technical aspects


MULTIPLE CHOICE RESPONSES

One Correct Response With documented reference of correctness

Not just an experts “feeling” that it’s right

Three Plausible Responses One mistake that most novices make

(a common error)

One mistake that soon-to-be experts tend to make

One mistake that even many experts make


ANSWER VALIDATION STRATEGIES

Need to validate your preferred response?

Document references that validate the correct response (#1)

Ask an acknowledged expert to write a white paper on the subject

Poll experts and write a white paper about the polled results

Give question to experts and novices; determine which answer the experts choose

Other? Anything that documents a preferred response other than “I believe this to be true.”


In a multiple choice question, the optimal answer set is four answers with one right answer.

What should the other three answers be?

a) Two wrong answers plus one “all correct” answer.

b) Two plausible answers plus one obviously wrong answer.

c) One plausible answer, one wrong answer, plus an “all correct” answer.

d) Three plausible answers typical of different skill levels.


ACTIVITY #3: PLAUSIBLE RESPONSES1. Write 3 plausible and 1 correct response to your question

2. Enter your question in your test engine.

3. View your question in the test engine as though you were a test-taker.



Checkpoint

Is there a difference between what you planned to write and what the test engine allowed or how the question displayed? Why?

QUALITY ASSURANCE

How do you check your work before publishing it?

BASIC QUALITY ASSURANCE

• Grammar, spelling, punctuation

• Looking for “tells” (hints that give away the correct response)• Length of responses

• Change in verb tense

• Change in actor

• Change in technical terms

• Change in punctuation

• Ensure that the question stem• Uses only one action verb

• Starts with question work (5Ws/H)

• Ends in question mark


Key words, changes in grammar style and verb tense can provide “tells” (hints to the right answer) among the plausible distractor answers.

Which of the following techniques would also prevent a “tell” in your answer set?

a) Use mainly very short one or two word answers.

b) Use very precise technical terminology or phrases.

c) Use answers that are the relatively the same length.

d) Use only present tense verbs in plausible answers.


In a multiple choice question, the optimal answer set is four answers with one right answer and:

a) 3 wrong answers

b) 2 plausible answers +1 obviously wrong answr

c) Use one plausible answer, onw very wrong answer, and one “all correct” answer.

d) Three plausible answers typical of different skill levels.


What does the question really test?

WHAT QA ISSUES?

ADVANCED QUALITY ASSURANCEPlain Language

• Active voice (avoid “is” and “was”) – go for the action

• Use strong clear noun – a person, place or thing (avoid “it” “they”)

• Use simple sentence structures – end your thought with a period and limit the number of comma and colons.

• Use “you” or the implied “you” command when the reader is required to take action

• Spell out acronyms

• Avoid jargon

• Organize your points

Global English• Avoid the use of country specific terms (e.g., USDA, Rte 66, M1, Kindergarten)

• Describe common situations rather than using local terms (e.g., tsunamis, School District, US green building code)

Reading Level (approx. 9th grade)• Watch the long names with many capitalized nouns in a row

• Put periods at end of bullet-points

• Avoid jargon

• Limit use of technical terms

• Use short paragraphs


Your current project must provide an assessment with many questions that will be used in many countries by professionals with many language preferences.

What is the strongest reason for using Plain Language and Global English in writing test questions?

a) Your question items will all be in one verb tense for reading ease.

b) Your questions will be clear to anyone who reads English.

c) Your answers are researched, justifiable and measurable.

d) Your work can be used internationally without translation.


How did the test engine affect your question?

What happened to your quality assurance when you entered your question into the test engine?

What test engine issues did you experience?

Did you put your questions in a work processor first? If so, how did that impact your results?

TEST ENGINE IMPACT

ERROR PREVENTIONUse a QA Checklist like the one provided Check the writing, spelling and grammar

Review questions (offline) with an expert audience who has not seen them• Do they validate the question, the correct answer and the validation

resource (text, website, document)

Load your items into the online format Does the item display correctly? Does the answer correctly evaluate as right or wrong? Does the answer/score transfer to the LMS correctly?

Run an online pilot with both expert and non-experts candidates Feedback on questions and test process Cut scores

Modify the test as needed


YOU HAVE HAD THE CHANCE TO:

1. Write a one-paragraph scenario opening to a question.

2. Write a question with one right answer.

3. Write three plausible answers and one correct answer to the posed question.

4. Identify strategies for validating the preferred response to posed question.

5. Check answers for “tells” that give away the right or wrong answer.

6. Explain the value of using Plain Language and Global English in writing test questions.

7. Leave with three tools to apply in writing test items for learning or certifications.

8. Recognize test engine characteristics that impact item design, delivery and validation.

9. Prototype one or more questions in a free test-engine.


In the next 7 days, write at least 5 scenario-based, multiple-choice questions with 1 correct and 3 plausible responses.

QA them with the checklist and enter them into a test engine.

THE CHALLENGE!

WRITING TEST QUESTIONS FOR

ANY TEST ENGINE

Sharon Gander, M.Ed., CPT

Judith Hale, Ph.D., CPT

Contact us:

Sharon Gander at [email protected]

Judy Hale at [email protected]

mailto:[email protected]

mailto:[email protected]


SESSION EVALUATIONPlease take the time to provide us with your session feedback using any of the following methods.

QR Code

1. Download a QR reader to your Smartphoneor tablet

2. Scan the QR code from this slide, yoursession evaluation sign, or your program

3. Launch and complete the evaluation

URL Address

1. Locate the URL for this session from this slide, you session evaluation sign, or your program.

2. Enter the URL Code for this session into the browser of your laptop, Smartphone or tablet

3. Launch and complete the evaluation

URL: https://www.questionmark.com/cets07

https://www.questionmark.com/cets07

Cets 2013_ gander hale_write_test_questions_session

Education

Transcript of Cets 2013_ gander hale_write_test_questions_session