Introduction to Questionnaire

DesignSpring 2005 Seminar

Series

Survey Research LaboratoryUniversity of Illinois at Chicago

www.srl.uic.edu

Questionnaire design encompasses:

1. How to write questions

2. How to draft and organize a questionnaire

The Art of Asking Questions

You must ask the right questionRespondents must understand your questionRespondents must know the answerRespondents must be willing and able to tell you the answer

Three Simple Rules for Writing a Good Questionnaire

1) Think through your research questions and objectives before you write questions

2) Prepare an analysis plan before you write questions

3) Ask yourself, in relation to points #1 and #2 above, if each question on your list is necessary? Even if the data would be ‘interesting’ it has to ultimately be used in analysis to make the cut!

Types of Survey Questions

1) Those that ask about behaviors or facts Non-threatening behavior questions Threatening behavior questions Demographics

2) Those that ask about psychological states or attitudes

3) Those that ask about knowledge

What Is A Good Question?

One that yields a truthful, accurate answer

One that asks for one answer on one dimension

One that accommodates all possible contingencies of response

One that uses specific, simple language

One that has mutually exclusive response options

One that produces variability in response

One that minimizes social desirability One that is pretested

Ask questions one at a time

Bad question:In the past 6 months, what major appliances has your household purchased new from the store?

Better question:Now I’m going to read a list of household appliances. As I read each one, please tell me whether or not your household has purchased this type of appliance new from the store during the past 6 months. How about…a refrigerator?a kitchen range or oven?a microwave?

Ask questions one at a time

Bad Question:Compared to one year ago, are you paying more, less, or about the same for your auto and life insurance?

Better Question:Compared to one year ago, are you now paying more, less, or about the same for …

a. auto insurance?b. life insurance?

SpecifySpecify who, what, when, where and how.

For example, whose income? What’s included? Over what period of time? Example:

“In 2002, what was your total household income, before taxes? Please count income from all members of your household, including wages from employment, disability, social security, and public aid.”

Specify through cuesExample

People drink beer in many places – for example, at home, at restaurants, at bars, sporting events, at friends’ homes, etc. During the past 30 days, did you drink any beer?

Use words with singular meanings

Ambiguous:How would you compare how close you are to family in your current neighborhood to how close you were in your old neighborhood? Would you say your family is closer here, further here, or the same?

More Clear:Compared to your last neighborhood, do you now live closer to your family, are you further from your family, or are you about the same distance?

Social Desirability

Respondents will try to represent themselves to the interviewer in a way that reflects positively on them

As questions become more threatening, respondents are more likely to overstate or understate behavior, even when the best question wording is used

Minimizing Social Desirability

• For socially desirable behavior, it is better to ask whether respondents have ever engaged in the behavior before asking whether they currently engage in the behavior• For socially undesirable behavior, it is better to ask about current behavior first, rather than ask about their usual or typical behavior• Train interviewers to maintain a professional attitude• Self-administered computer-assisted procedures can reduce question threat and improve reporting on sensitive questions• Longer questions reduce sensitivity when obtaining information on frequencies of socially undesirable behavior

Open vs. Closed Questions

General rule: closed questions are usually better– Easier for the respondent– Less coding later– Better to have respondent do

categorizing– Categories help define the question

Disadvantages of Closed Questions

• Categories may be leading to respondents

• May make it too easy to answer without thinking

• May limit spontaneity • Not best when

– asking for frequency of sensitive behaviors

– there are numerous possible responses

General Principles for Response Options

• Response categories should be consistent with the question

• Categories must be exhaustive, including every possible answer

• Categories must be mutually exclusive (no overlap)

• If appropriate, include a “don’t know” category

Constructing Response Scales

• Respondents can generally only remember a maximum of 5 responses unless visual cues are used

• Using graphic images such as thermometers and ladders and using card sorting for compex ratings is effective

• Number of points in scale should be determined by how you intend to use the data

• With scales with few points, every scale can be labeled; in longer scales, only the endpoints are labeled

Ordering Response Categories

• Usually better to list responses from the lower level to the higher level• Associate greater response levels w/ greater numbers• Start with end of a scale that is least socially desirable

Should you use a middle category?

– Common practice is to omit it to push respondents (Rs) toward one end or the other, on the theory that few individuals are truly in the middle on a particular issue

– Evidence from empirical studies shows that use of an explicit middle alternative will often be taken by Rs in a forced choice situation if offered; at the same time, it does not affect the ratio of ‘pro’ to ‘con’ responses or the size of the don’t know category

– Our usual recommendation is to include it unless there are persuasive reasons to exclude

Part 2:

How to draft and organize your questionnaire

Ordering the Questions

• Segment by topic• Ask about related topics together• Salient questions take precedence over less

salient ones• Ask recall backwards in time• Use transitions when changing topics – give a

sense of progress through the questionnaire• Leave objectionable questions (e.g., income) for

the end• Put demographic questions at the end

How to Start the Questionnaire

• Start with easy questions that all respondents can answer with little effort

• First questions should also be non-threatening

• Don’t start with knowledge or awareness questions

• First questions should be directly related to the topic as described in the introduction or advance/cover letter

Survey Intro/Cover Letter

Introduction should indicate:– who is conducting the survey– the topics to be covered in the survey– an assurance of confidentiality– any IRB stipulations– whether you offer how long it will

take depends on mode, topic, population

Physical Format of the Self-Administered

Questionnaire• Careful formatting is necessary

to decrease errors and increase motivation

• Respondent’s needs must always take priority, followed by interviewer and data processors

Physical Format Checklist

• Number all questions sequentially • Use large, clear type; don’t crowd• ‘White space:’ Place more blank space between

questions than between subcomponents of questions

• List answer categories vertically instead of horizontally

• Avoid double/triple ‘banking’ of response choices• Be consistent with direction of response

categories• Be consistent with placement of response

categories

Physical Format, Continued

• Don’t split questions across pages. If necessary (e.g., question requires 1.5 pages), restate question and response categories on next page

• Put special instructions on questionnaire as needed, next to question

• Distinguish directions from questions• Precode the questionnaire (vs. check

boxes)

Mail questionnaires

• Include a cover letter and contact information if the respondent needs help

• Use a booklet format– Easier to turn pages– Prevents lost pages– Permits double-page formats– Looks more professional– Include a title, graphic, name/address

of sponsor on cover

Testing the Questionnaire

• Preferable to test the questionnaires with people like those in your main study study population

• Test in same mode to be used for main study

• Consider cognitive pretesting

Questionnaire Design Steps

1. Decide what information is needed2. Search for existing questions3. Focus groups4. Draft new questions/revise

existing ones5. Sequence the questions6. Get peer evaluation7. Revise and test on self/co-

workers

Questionnaire Design Steps

8. Think-aloud interviews9. Revise/eliminate questions10. Prepare interviewer instructions

for pilot test11. Pilot test (10-20 cases)12. Revise eliminate questions based

on respondent & interviewer comments

Questionnaire Design Steps

13. Pilot test again, if necessary14. Prepare final interviewer instructions15. Be prepared to modify questionnaires

if interviewer training raises problems16. After interviewing is complete,

debrief interviewers for potential problems

17. Use experience from one study for future planning

Key References To Get Started

Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, 2000.

Bradburn, N, Sudman, S. and Wansink, B.. Asking Questions: The Definitive Guide to Questionnaire design – for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004.

See also past issues of Public Opinion Quarterly