PRACTICAL TIPS FOR WRITING & DESIGNING BETTER SURVEY QUESTIONS

2

MOBILE FIRST• Learning to write questions more concisely• Minimising the length of your surveys• Changing the way some types of question are asked • Design factors:

◦ Optimising the use of space◦ Using fully mobile enabled question formats◦ Exploiting the touch screen features more effectively ◦ Using bigger buttons & font sizing

• A more iterative approach: more piloting

3

TARGET 15 MINUTES OR LESS

MINIMIZING THE LENGTH OF YOUR SURVEYS

4

BE CONCISE• Once you have written each question go back

and challenge yourself to write it more concisely:◦ Remove all superfluous words: That included

niceties

• Separate out the question and the instructions on how to answer

5

SEEING HIGHER DROPOUT ON TABLETS AS WELL AS MOBILES

6

PresentI am doing this now.

Which brands do you buy?More concise and clearI buy this brand = clear statement of fact

TENSE EXAMPLEPerfectI did in the very recent past.

Which brands have you bought?Lacks clarity, some may not be sure what brands they should selecte.g. “Um?...I bought this brand once but…”

PDO

ODON’T

LINGUISTIC STYLE: AVOID THE PERFECT TENSE

7

DEFINE EXPECTATIONS

• …and don’t be greedy!• Always think about the work load:

Break out into decision units:◦ Single choice question = 1 decision◦ 1 likert question = 1 decision◦ 1 tick selection = 1 binary decision

• Each decision takes 2 seconds

8

DON’T BE GREEDY!What brands do you recall

Average of 2.3 brands

10 options but Over 50% name just 1 brand

9

DON’T BE GREEDY!

What brands do you recall (name up to 5)

Average of 3.1 brands

Half the options but Over 50% list 2+ brand

10

APPLY RULES TO CONTAIN THE TASK

Which brands do you recall What are the first 3 brands you can think of?

1.7 brands 2.3 brands

The rule

11

REQUIRES MUCH MORE FOCUS ON THE CONSUMER EXPERIENCE

12

RESPONDENTS ARE MORE EASILY DISTRACTEDQuestion answer time

Distractionpoints

DESKTOP PC ERA

LAPTOP & TABLET ERA

13

MORE LIKELY TO STOP MID SURVEYQuestion answer timeDESKTOP PC ERA

LAPTOP & TABLET ERA

14

44%* completing survey on tablets in living room watching TV at the same time

LAPTOP & TABLET ERA

* Lightspeed GMI research June 2015

15

RESPONDENTS ARE SEEKING ENTERTAINMENT FROM THEIR TABLETS AND MOBILE DEVICESPC ERA

Tablet ERA

Frustrating questionsQuestion answer time

16

FUNDAMENTALLY NEED TO THINK ABOUT CHANGING OUR APPROACH BY UNDERSTANDING WHAT TRIGGERS DROPOUT• Any form of repetition• Long, overly complex

questions• Purposeless/irrational

questions

17

?

AVOID TOO MUCH REPETITION THROUGH CAREFULLY COMBINING QUESTIONS

18

COMBINING THINKING WITH REMEMBERING

19

Q1. Which of these brands have you used?

Q2. How much do you rate these brands? +

Asking the 2 questions in one:

Yes No

1 2 3 4 5 6 7 8 9 10

1 2 3 4 5 6 7 8 9 10Not used this brand

How much do you rate these brands?

20

PRESENT CLEAR CHOICES• Many an issue in survey design is caused by asking

complicated questions requiring “double thinking” • Questions should present simple clear choices• If you cannot present it as a set of single choices consider

breaking into more than one question• Complex nice looking questions often deliver very poor data

21

BREAK UP INTO DISTINCT PIECES THINKING

Water Tea Coffee

Day 1 Day 2

Juice

Soft drink Beer Wine Other

What have you drunk in the last 3 days?

Day 3

Water Tea Coffee Juice

Soft drink Beer Wine Other

Q1 What have you drunk on day 1?

Water Tea Coffee Juice

Soft drink Beer Wine Other

Q2 What have you drunk on day 2?

Water Tea Coffee Juice

Soft drink Beer Wine Other

Q3 What have you drunk on day 3?

30% fewer drinks recorded per day

ODON’T PDO

22

OPTIMIZE LONG MULTI-CHOICE LISTS…1. Respondents cannot read through a list with more than 10

options efficiently2. Remove anything selected by less than 2% of sample3. Be conscious of tick conflicts and overlaps4. Long lists best break down into logical nested sub sets where

possible

23

BETTER ORGANISE CHOICES

4 tick average 8 tick averageODON’T PDO

24

BREAK UP REPETITIVE LIKERT SCALES INTO UNIQUE QUESTION CLUSTERS

25

AGREEMENT SCALES

Premium Ethical Environmentally friendlyHealthy Convenient Energy boostingNatural Nutritious Exciting Unique Flavorful IndulgentPremium 0.517 0.471 0.410 0.315 0.414 0.525 0.488 0.624 0.545 0.479 0.473Ethical 0.517 0.642 0.384 0.370 0.389 0.496 0.451 0.496 0.391 0.417 0.326Environmentally friendly 0.471 0.642 0.462 0.256 0.376 0.601 0.549 0.455 0.452 0.310 0.345Healthy 0.410 0.384 0.462 0.279 0.482 0.594 0.704 0.411 0.418 0.255 0.261Convenient 0.315 0.370 0.256 0.279 0.228 0.206 0.263 0.372 0.346 0.494 0.369Energy boosting 0.404 0.377 0.365 0.475 0.208 0.504 0.520 0.401 0.411 0.301 0.294Natural 0.509 0.487 0.596 0.583 0.186 0.496 0.700 0.444 0.504 0.278 0.321Nutritious 0.488 0.451 0.549 0.704 0.263 0.527 0.710 0.416 0.509 0.314 0.327Exciting 0.624 0.496 0.455 0.411 0.372 0.410 0.461 0.416 0.613 0.531 0.551Unique 0.545 0.391 0.452 0.418 0.346 0.421 0.520 0.509 0.613 0.397 0.490Flavorful 0.479 0.417 0.310 0.255 0.494 0.317 0.299 0.314 0.531 0.397 0.546Indulgent 0.473 0.326 0.345 0.261 0.369 0.307 0.343 0.327 0.551 0.490 0.546

C=0.44

26

BREAKING OUT INTO DISTINCT, UNIQUE PIECES OF THINKING

price unique taste pack quality healthHow would you rate the price?_Price 0.14 -0.19 -0.25 -0.23 -0.18Compared to other product how unique/similar is it?_Unique 0.14 0.05 -0.11 -0.10 -0.11What do you think of the taste/flavor/recipe? -0.19 0.05 0.47 0.37 0.22In the race to gain sales, how well do you think it will compete? -0.25 -0.11 0.47 0.54 0.54Rate the quality & nutritional value_quality -0.23 -0.10 0.37 0.54 0.74Rate the quality & nutritional value_Health & nutrition -0.18 -0.11 0.22 0.54 0.74How many shares do you want to buy or sell in this product? -0.28 -0.05 0.48 0.60 0.39 0.33

C=0.24

27

THINK ABOUT WHEN TO USE A LIKERT/RANGE SCALE• Range scales

◦ Forced selection choice so contains a lot more noise from those not concentrating which makes the data harder to differentiate

◦ Thinking time: A range scale question takes 2 times longer to answer than a tick choice or binary choice

◦ System 2 thinking process: compared to a binary choice which is more system 1

• A tick choice is an active selection• No point asking a likert scale question if all you are going to

do is present the top 2 box answers

28

SHOT SELECTION: LIKERT OR TICK?• Think about how the data is

going to be analysed 

• Think if you are after intuitive system 1 responses or more considered system 2 responses

29

ANSWER VARIANCE: RATING SCALES V TICK

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 360.00.51.01.52.02.53.03.54.04.55.0

Healthy: Rating scale

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 400%

10%20%30%40%50%60%70%80%90%

Healthy: tick

30

THE IMPORTANCE OF LEGITIMACY• Boring, repetitive & irrational questions that respondents don’t

know how to answer really do cause problems in surveys

COMMON ISSUES WE SEE• Irrational answers choices• Conflicting choices• Multi-choice thinking

FOCUS ON THE CONSUMER EXPERIENCE

31

Celebrity endorsement• It provides you with an unbiased and trusted recommendation about

brands or products• It allows you to compare prices / get the best deal• It's good for comparing prices• It helps you to share information with other people• It makes you feel connected to a brand or product• It provides you with reassurance that you have made the right choice

of brand or product• It provides you with content you can talk about or share with other

people• It provides you with more/detailed information about brands or

products self evident

IRRATIONAL ANSWER CHOICES…FOCUS ON THE CONSUMER EXPERIENCE

Case study:

Removal of three minutes of irrational questions reduced in survey dropout by 20%

Icons are processed twice as fast as text

Details matter2 colour design works best

Don’t use for one-off & usage questions

Aid repetitive tasks Reduces straight-lining

Help people think more!

33

THE PHILOSOPHY OF GOOD SURVEY DESIGN

1. Recognising that it is a consumer experience 2. Think of the first few questions as the ad for the survey3. Build your survey narrative around one central question4. Apply narrative structure5. Encourage respondents to think6. Think mobile first when designing surveys7. Containing tasks by setting rules8. Combining thinking and remembering9. Ask legitimate questions10. Simplify: break questions into component parts / thinking

chunks 11. Break up agreement scale 12. Understand when to use a range scale and when not?13. Exploit icons and visuals