Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Item Writing Techniques for Surveys, Quizzes and Tests
Ivan Forward
Assessment Solutions Consultant, Questionmark
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Founded in 1988
• Offices in US, UK, Canada; Distribution partners worldwide
• Assessment technologies and portal solutions that enable organisations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.
Background
• 1,000 current enterprise customers worldwide
• Business, Government, Education, Awarding Bodies, Certification and Test Publishers
Customers
About Questionmark
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Assessment Management System
• Questionmark OnDemand (SaaS/cloud)
• Questionmark Perception (installable software)
• Multilingual
• Localized for dozens of different languages and cultures
• Global 24/7 support available
Products and Services
About Questionmark
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2016 Users Conference
Slide 4
Get vital info and training on the latest assessment technologies and best practices
Network with fellow assessment and learning professionals
Learn about the Questionmark product roadmap
w w w. q u e s t i o n m a r k . c o m /g o /c o n fe r e n c e
SAVE
THE
DATE
MiamiApril 12-15
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Webinar in “lecture mode” …
Slide 5
To ask questions, use the “Q&A” feature
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Define “fair”, “valid”, and “reliable” and describe why they’re important when writing items
Describe the different elements that make up an item
Write a well-written stimulus
Write a well-written choice
Write an effective survey question
Objectives
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions need to be:• Fair
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions need to be:• Fair• Valid
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions need to be:• Fair• Valid• Reliable
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questions need to be:
Slide 10
Well written questions essential to trustable assessment results
Reliable
•How consistently does the test measure a construct?
Valid
•Does the test is measuring what it is supposed to measure?
Fair
•Was there an unfair advantage for any given demographic group?
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Reliable: dependable, repeatable, consistent
Valid: measures the specific knowledge and skills that were intended to be measured
Reliability and validity
Slide 11
Figure 1:Reliable but not Valid
Figure 2:Not reliable, not Valid
Figure 3:Reliable and Valid
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Job Task Analysis
Slide 12
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JTA Authoring
Slide 13
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JTA Surveys to Validate Content
Slide 14
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JTA in Analytics
Slide 15
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
JTA in Analytics
Slide 16
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Items vs. Questions
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which statement is an identifying characteristic of a mixture?
Well-written stimulus:• Carefully worded – no trickery• Clear – participant knows what’s being asked• Concise – no window dressing• Avoid jargon• Positive phrasing• Use questions whenever possible
What, Who, When Where, Why and How
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Well-written choices:• Same length• As short as possible• Avoid keywords• Avoid grammatical cues• Reasonable
• Logical or numerical order• Avoid negatives, especially
double negatives• Independent – no overlap• Avoid AOTA & NOTA• Correct choices evenly
distributed
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which factors impact a presidential election?a. Previous president’s performance,
weather, local economyb. Weather, local economy, national
unemployment ratec. Congress’ performance rating,
weather, national unemployment rate
d. Local economy, weather, Congress’ performance rating
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which factors impact a presidential election? Select all that apply.
Previous president’s performance
Weather Congress’ performance rating Local economy National unemployment rate
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which of the following describes a banana?a. It’s a vegetable.b. It’s yellow.c. It’s blue.d. It’s round.e. All of the above.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which of the following describes a banana?a. It’s a vegetable.b. It’s yellow.c. It’s blue.d. It’s round.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Yes or No: When loading your snarkleblaster, should you beedlethe diggle or zix the frondle?oYesoNo
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Yes or No: When loading your snarkleblaster, should you beedle the diggle?
o Yeso No
Yes or No: When loading your snarkleblaster, should you zix the frondle?
o Yeso No
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
When putting a child to bed for the night, you should:
a. Blast him/her with a fire hose.b. Create a calm and relaxing atmosphere.c. Parade 3 elephants through the room.d. Paint yourself purple with pink polka dots
and dance a jig.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
When putting a child to bed for the night, you should:a. Give the child a cup of his/her favorite soda.b. Create a calm and relaxing atmosphere.c. Run around the house to exhaust the child.d. Turn on some music loud enough to drown out
any noise from any other part of the house.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A chimpanzee’s favorite treat is a:a. Appleb. Bananac. Oranged. Eggplant
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A chimpanzee’s favorite treat is:a. An appleb. A bananac. An oranged. An eggplant
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
When replacing strings on a bass guitar:a.Change the pre-amp’s battery.b.Seat the thick end of the string in the
bridge, lightly tighten the string with the tuning machine in the headstock, then stretch the string before fully tightening and tuning it.
c. Set the equalization.d.Adjust the strap.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
When replacing strings on a bass guitar:a.Change the pre-amp’s battery.b.Set the intonation.c. Set the equalization.d.Adjust the strap.
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which of the following is not a characteristic of a friggle in a state of grex?a.Hoomishnessb.Blue ricklesc. Excessive
sninishnessd.Lack of nongs
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which of the following is NOT a characteristic of a friggle in a state of grex?a.Hoomishnessb.Blue ricklesc. Excessive
sninishnessd.Lack of nongs
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
You are a Questionmark help desk technician. A customer calls to ask for advice about putting together an assessment. You demonstrate the assessment creation process in Questionmark Live. You discuss how to schedule an assessment. When the customer asks about using assessments in solving business problems, which video do you suggest?
a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Which Questionmark video explains using assessments in solving business problems?
a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
When putting a child to bed for the night, you should:a. Give the child a cup of his/her favorite
soda.b. Create a calm and relaxing atmosphere.c. Run around the house to exhaust the
child.d. Turn on some music loud enough to
drown out any noise from any other part of the house.
A chimpanzee’s favorite treat is:a. An appleb. A bananac. An
oranged. An
eggplant
When replacing strings on a bass guitar:
a. Change the pre-amp’s battery.
b. Set the intonation.c. Set the equalization.d. Adjust the strap.
Which of the following is NOTacharacteristic of a friggle in a state of grex?a. Hoomishnessb. Blue ricklesc. Excessive
sninishnessd. Lack of nongs
Yes or No: When loading your snarkleblaster, should you beedlethe diggle?
o Yeso No
Which Questionmark video explains using assessments in solving business problems? a. “Assessments Through the Learning Process”b. “Assessments Through the A-Model”c. “Writing Good Assessment Items”d. “Engaging with Questionmark”
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Survey Question Tips
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
List what you want to know/learn from the survey results
Write down who will use the information and for what purpose
Ensure that the output of the survey will answer the questions and meet the stakeholder needs
Then – write your questions
Deciding on what you want to know/learn from the survey
Slide 39
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Objective: Determine the amount of time per week spent reading email.
How many emails do you receive per week, on average?
On average, how many hours do you spend responding to emails every week?
How long does it take to read the average email?
On average, how many hours do you spend reading emails every week?
Ask for the Data You Want!
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Were you satisfied with our service? Yes
No
What if I was not really satisfied, but it wasn’t bad enough to be dissatisfied?
What if I was satisfied with the restaurant service, but very disappointed with room service?
Make Sure the Question Can Be Answered!
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
There was ample interaction in class and the handouts were valuable.1. Strongly disagree2. Disagree3. Agree4. Strongly agree
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
I routinely refuse to cooperate with my coworkers.
versus
Members of my workgroup routinely cooperate with each other.
Write Questions You Would be Willing to Answer
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Lots of scales to choose from but some rules of thumb: Use the same scale for all
questions (e.g., all Likert are 1-5 Strongly Agree to Strongly Disagree)
Don’t make the scale too long as participants usually can’t differentiate
Ensure that the scale is weighted appropriately (e.g., ‘strongly agree’ is ‘5’, strongly disagree is ‘1’)
Consider not using a “cop out” N/A or unsure
Great book dedicated to this subject of scale development: DeVellis, R. F. (2003). Scale
Development: Theory and Applications Second Edition (Applied Social Research Methods).
Scale development
Slide 44
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Direction, e.g.: Strongly Agree Strongly Disagree, or
Strongly Disagree Strongly Agree
Some evidence that respondents have a natural desire to want to agree with a statement If “strongly agree” is first, there may be a slight tendency to
select it without much thought
If “strongly disagree” is first, there may be more of a tendency to think before responding, and differentiate more honestly between “agree” and “strongly agree”
Rating Scales: Direction
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Keep the survey short and sweet (only ask the minimum number of questions required) This will help get the best
response rate
Put the most important questions first
Ensure that the survey flows (order questions)
General survey authoring tips
Slide 46
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Sometimes it takes just one word to bias a question. Avoid using inflammatory words in surveys, such as:
allege, allude, arbitrary, blame, claim, demand, error, failure, fault, ignore, ill-advised, ill-informed, incompetence, ineptness, insist, just, maintain, misinformed, must, neglected, one-sided, only, overreact, peremptory, purport, questionable, rejection, rigid, so-called, unfortunately, unilateral, unreasonable
Avoid “Red-Flag” words
Slide 47
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Avoid the problem altogether.
I ride my bicycle to work:
could be:
On average, I ride my bicycle to work:
Rating Scales: Labeling
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Space choices evenly.
Provide “not applicable” option if there’s any possibility the respondent may not have an answer. Make “not applicable” different in appearance.
Numbers: low to high in order.
Use the same term at each end of the scale if at all possible:“not at all shy” “very shy” instead of
“brave” “shy”
Rating Scales: In Summary
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
www.questionmark.com/go/whitepapers
Recommended White Papers
Slide 50
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Keep in Touch
Slide 51
blog.questionmark.com
twitter.com/questionmark
youtube.com/questionmarkvideo
https://www.linkedin.com/grp/home?gid=1931441
facebook.com/questionmarkgroup
plus.google.com/+QuestionmarkCorp/
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
2016 Users Conference
Slide 52
Get vital info and training on the latest assessment technologies and best practices
Network with fellow assessment and learning professionals
Learn about the Questionmark product roadmap
w w w. q u e s t i o n m a r k . c o m /g o /c o n fe r e n c e
SAVE
THE
DATE
MiamiApril 12-15
Copyright © 1995-2014 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Thank you for attending!
Ivan Forward
Assessment Solutions Consultant, Questionmark
Top Related