S T E P H A N I E L . W E H N A UN I C O L E L . S T U R G E S

C E N T E R F O R S U RV E Y R E SE A R C HP E N N STAT E H A R R I SB U R G

How to Design a Survey Instrument

Define research objectives

Choose mode of collection

Choose sampling frame

Construct and pretest questionnaire

Design and select sample

Design and implement data collection

Code and edit data

Make post-survey adjustments

Perform analysis

The Process of Survey Research

1. Topic Mapping

Topic Mapping

Start with the research objectives

List the topics to be investigated

Translate topics into questions (measures)

Topic Mapping Example

Questionnaire that evaluates alcohol use at Penn State Harrisburg (PSH) for Student Health Services:

To what extent are students using alcohol at PSH? What beliefs exist about alcohol use at PSH? What do students know about the negative affects of

alcohol use? What are the demographics of survey respondents?

2. Writing Questions

Guidelines forSurvey Design

Mode (paper, phone, web, face-to-face)Keep your instrument short (respondent

burden)Non-threatening questions to more

sensitive topics (ask demographics last) - build rapport!

Ask the same question different ways (validity)

Adopt or adapt Q’s (Census)PRE-TEST! (More on this later…)

Types of Questions: Open-ended Questions

Respondents have the opportunity to provide an answer of their own.

May be more difficult to analyze given the extensive amount of information that can be collected.

Examples: What do you think is the most important problem

facing Pennsylvania today? What is your age?

Types of Questions: Closed-ended Questions

The respondent is asked to select an answer from a list provided by the researchers.

Widely used because they provide a greater uniformity of responses and are more easily analyzed.

Select one response or “Select all that apply”

Types of Questions: Closed-ended Questions

“Select one” Example: In general, how satisfied are you with the way things

are going in Pennsylvania today? Very satisfied More or less satisfied Not satisfied at all Don’t know Declined to answer

Types of Questions: Closed-ended Questions

“Select all that apply” Example: What type of insurance do you have? Please select all

that apply. Employer-based health insurance Medicare Government-provided insurance Military health care Purchased health insurance Other insurance Don’t know Declined to answer

Types of Questions: Partially Closed-ended Questions

The respondent can select from one of the response options or can supply his/her own response in an “other” category.

Example: Which of the following is your favorite college men’s

sport? Football Basketball Baseball Other: ___________

GENERAL TIPS

Writing Questions

Avoid ambiguous words (write concrete questions)

Be specific (with definitions, time frames, locations, etc.).

EXAMPLES - Poor:

When did you move to Pennsylvania? Right after I finished college. When I was 20. In 1998.

Better: In what year did you move to Pennsylvania?

Poor: How often do you exercise in a typical week?

Better: How often did you exercise during the past week (start with today’s

date and count back 7 days)?

Define terms and Avoid abbreviations

Don’t leave it up to the respondent to interpret a term – provide a definition so that everyone is answering the same question. Example:

A drink of alcohol is considered 1 can or bottle of beer, 1 glass of wine, 1 can or bottle of wine cooler, 1 cocktail, or 1 shot of liquor. During your current pregnancy, how many days per week have you had at least one drink of any alcoholic beverage?

Do not assume that your respondents will be familiar with abbreviations.

Ask one question at a time (avoid double-barreled questions)

Poor: Do you like apples and oranges? Do you want to be rich and famous?

Better: Do you like apples? Do you like oranges? Do you want to be rich? Do you want to be famous?

Beware of questions that elicit socially desirable responses

Respondents may give false answers because they are embarrassed or feel bad about their answer.

Examples: Do you view sexual material on the Internet?

Have you ever smoked marijuana?

Avoid pre-disposing respondents

Remain NEUTRAL! Don’t guide your respondents to an answer.

Feedback during an interview (in-person or via phone) can also bias a respondent.

Examples: More people have seen the movie, Gone with the Wind,

than any other motion picture. Have you seen this movie?

Newspapers and television started talking about patient safety and similar problems in our healthcare system about five years ago. Do you think that, in the past five years, patient safety has gotten better, stayed the same, or gotten worse?

Avoid negative questions

Negative questions can be confusing!Examples:

The United Nations should NOT have more authority to intervene in military affairs.

Our Diocese today has not developed innovative ministries for small and rural communities.

3. Writing Responses

Creating Response Sets

Mutually exclusive (can’t fall into 2 categories) and exhaustive (all possible) categories

Appropriate number of options (too many = confusing; too few= respondent can’t answer)

Appropriate range of options (look at distribution of responses; income example)

Consider rotating response options because respondents tend to answer with the first or last response option they read/hear (if applicable)

FOUR STAGES OF TRADITIONAL TESTING

4. Testing Your Instrument

Testing – Stage 1

Developmental work Typical aims of this stage are:

Explore new topic areas (consult professional experts and cultural insiders)

Test feasibility of methods

Focus on problem areas

Testing – Stage 2

Question testingTypical aims of this stage are:

Test and refine the wording of individual questions Can respondents understand questions? Can interviewers ask the questions? Do the questions answer what you think they should

answer? Check the flow of the questionnaire as a whole

Do the questions and sections flow well? Look for possible context effects Check skip patterns (if not computer-assisted) Look for respondent burden (interest in topic, length,

etc.)

Testing – Stage 2a

Informal testing (before you test questions on members of your population) Read your questionnaire aloud to yourself

Highlights the differences between written and spoken language

A question that looks great on paper can still be difficult to read aloud

Interview yourself Can YOU answer the question? Is the question easy to answer?

Do mock interviews with colleagues

*These suggestions may seem simple, but they are very effective for quickly finding errors early in the questionnaire design process.

Testing – Stage 2b

Test your questions with members of your population

Examples: Respondent debriefing (2 part process: go through

survey questions, then debriefing; ask respondents how they came up with their responses)

Cognitive interviewing (think aloud method)

Converse and Presser (1986) suggest testing questions with 25-75 individuals.

Fowler (1995) suggests 10-25, 30 individuals

How many tests at Stage 2b?

If time and money allow, it is a good idea to do two.

In the first pre-test, you identify which questions don’t work.

You then improve them (or at least think you do!) – the only way to know for sure is to re-test them.

Ideally, you should not include a question in the final instrument that has not been tested.

Testing – Stage 3

Usability testing

If the survey is a computer-assisted questionnaire (phone or web survey), then you will need to test the usability of the questionnaire.

In other words, is the programming correct?

Testing – Stage 4

Dress Rehearsal PilotsTypical aims of the stage are:

1. Test the survey procedures as a whole in survey conditions2. Estimate rates of response3. Check timings4. Develop pre-codes for open-ended questions5. Smooth coordination and establish routines6. Test other aspects of the survey methodology

How many respondents? Converse and Presser (1986): n=25-75+ U.S. Census long form test n=16,000

*This stage is not always necessary. The first four purposes could be carried out as part of Stage 2b.

Considerations for Testing

The scale of your testing will depend on:

Budget

Nature of the questionnaire

Size of the population (in some cases)

QUESTIONS?

Stephanie L. Wehnau

Assistant Director,

Center for Survey Research

Phone: 717-948-6429

Email: slh227@psu.edu

Nicole L. Sturges

Project Coordinator,

Center for Survey Research

Phone: 717-948-6117

Email: nls17@psu.edu