The (mis)measure of people

Agenda

• Basics of survey structure & questionnaire design

• Types and formatting of items

• What kind of surveys do you typically receive during the year?

• What is your usual response to them?

• What makes them more intriguing so that you take them?

• Think of an example for your survey– customer satisfaction

Questions

Targeting your survey

• who is your intended market?

• how diverse are they?

• what do you most need to know?

• what level of information is needed?

• how are they best accessed?

• what are their characteristics that may affect response?

• what are their expected objections?

Levels of measurement

Designing the Questionnaire

Step 1: Specify what information will be sought

Step 2: Determine type of questionnaire and administration

method

Step 3: Determine content of individual questions

Step 4: Determine form of response to each question

Step 5: Determine wording of each question

Step 6: Determine sequence of questions

Step 7: Determine physical characteristics of questionnaire

Step 8: Reexamine steps 1 - 7 and revise if necessary

Step 9: Pretest questionnaire and revise if necessary

Wording of Questions

(a) Use simple words & short sentences

(b) Avoid ambiguous words and questions

(occasionally, sometimes, often, etc. Try to anchor each point with a specific description)

(c) Avoid leading questions

(d) Avoid implicit alternatives

(e) Avoid implicit assumptions

(f) Avoid generalizations and estimates

(g) Avoid double-barreled questions

(h) Avoid response order (i.e., position) bias

The attribute "customer service," for example, can be phrased as any one of the following:• Direct question

How satisfied are you with customer service? [Very satisfied, satisfied, Neither

satisfied nor disatisfied, Disatisfied, Very Dissatisfied

• Direct request

Please rate customer service. [Excellent, Very Good, Good, Fair, Poor]

• Statement with which to agree or disagree (Likert scale)

ABC Company provides good customer service. [Strongly agree, Agree,

Neither agree nor disagree, Disagree, Strongly Disagree]

• Question of frequency

How often does customer service exceed expectations? [Very Frequently,

Frequently, Neither frequently nor infrequently, Infrequently, Very infrequently]

• Question of extent

To what extent does customer service exceed expectations? [To very great

extent, To great extent, To some extent, To little extent, To very little extent]

Flame-broiled or Fried?

Original Question: “Do you prefer your hamburgers flame-broiled or fried?”Result: flame-broiling (Burger King) beat frying (McDonald’s) by a 3-1 margin

Revised Question #1: “Do you prefer a hamburger that is grilled on a hotstainless-steel grill or cooked by passing the raw meat through an open gas flame?”

Result: 53% preferred the grill (McDonald’s)

Revised Question #2: “The chain that grills on a hot stainless-steel griddle serves its cooked hamburgers at the proper temperature without having to use a microwave oven. And the chain that uses the gas flame puts the hamburgersafter they are cooked into a microwave oven before serving them. Just knowingthis, from which of these two chains would you prefer to buy a hamburger?”

Result: the grill (McDonald’s) beat the flame (Burger King) by a 5.5 to 1 margin

Handling Sensitive Questions

• Don’t ask unless absolutely necessary!• Give broad response categories (e.g., income, age)

• Place near end of questionnaire• “Hide” them in less sensitive questions• Use counterbiasing statement• Use randomized response technique• Transition sections & inform of rights

Use of Counterbiasing Statement

“Recent studies have shown that a high percentage of males use their wives’ cosmetics to hide blemishes. Have you used your wife’s cosmetics in the past week?”

Providing an acceptable context for answering what might have been sensitized before

Rank-Order Scales

Rank the following soft-drinks from 1 (best) to 5 (worst) according to your taste preference:

Coca-Cola _____

7-Up _____

Dr. Pepper _____

Pepsi-Cola _____

Tab _____

Paired Comparison Scale

For each of the following pairs, which soft-drink do you think is better (please check one soft-drink for each pair).

____Coca-Cola or ______7-Up

____7-Up or ______Pepsi-cola

____Tab or ______Coca-Cola

____Dr. Pepper or ______Pepsi-cola

____ Pepsi-cola or ______Tab

____ Coca-Cola or ______ DR. Pepper

____ Pepsi-cola or ______Coca-Cola

____Tab or ______ Dr. Pepper

____7-Up or ______Tab

____ Dr. Pepper or ______ 7-Up

Constant Sum Scales

Allocate a total of 100 points among the following soft-drinks depending on how favorable you feel toward each; the more highly you think of each soft-drink, the more points you should allocate to it. (Please check that the allocated points add to 100.)

Coca-Cola _____ points

7-Up _____ points

Dr. Pepper _____ points

Tab _____ points

Pepsi-Cola _____ points

100 points

Customized Rating Scales

Odd vs. Even Scale Points

Odd

Strongly Agree _____

Agree _____

Neutral _____

Disagree _____

Strongly disagree _____

Even

Strongly Agree_____

Agree _____

Disagree _____

Strongly disagree___

Balanced vs. Unbalanced Scales

Balanced

Very good ______

Good ______

Fair ______

Poor ______

Very Poor ______

Unbalanced

Excellent ______

Very Good ______

Good ______

Fair ______

Poor ______

Forced vs. Unforced Scales

Forced

Extremely Reliable ___

Very Reliable ___

Somewhat Reliable ___

Somewhat Unreliable ___

Very Unreliable ___

Extremely Unreliable ___

Unforced

Extremely Reliable ___

Very Reliable ___

Somewhat Reliable ___

Somewhat Unreliable ___

Very Unreliable ___

Extremely Unreliable ___

Don’t know ___

Labeled vs. End Anchored Scales

Labeled

Excellent _____

Very Good _____

Fair _____

Poor _____

Very Poor _____

End Anchored

Excellent __________

_____

_____

Poor _____

Number of Scale Points

5 Point

Excellent _____

_____

_____

_____

Poor _____

10 Point

Excellent __________________

_____________

_____________

_____________

_____________

_____________

_____________

_____________

_____________

Poor

Recommended Number of Scale Points

Amount of

Information

Number of Scale Points1 2 3 4 5 10 .................100

More than 5 options will provide little additional

discrimination

Begins to flatten out in return of information

Agreement:

• Strongly Agree

• Agree

• Undecided

• Disagree

• Strongly Disagree

Frequency:

• Very Frequently

• Frequently

• Occasionally

• Rarely

• Very Rarely

• Never

Importance:

• Very Important

• Important

• Moderately Important

• Of Little Importance

• Unimportant

Quality:

• Extremely Poor

• Below Average

• Average

• Above Average

• Excellent

Likelihood:

• Almost Always True

• Usually True

• Often True

• Occasionally True

• Sometimes But Infrequently True

• Usually Not True

• Almost Never True

Formats for Likert Items

1…2…3…4…5

Semantic Differential Scale

X= ABC Bank0= NBC Bank

old-fashioned ____ ____ ____ ____ ____ ____ ____ modern

not friendly ____ ____ ____ ____ ____ ____ ____ friendly

expensive ____ ____ ____ ____ ____ ____ ____ inexpensive

not trustworthy ____ ____ ____ ____ ____ ____ ____ trustworthy

X O

XO

XO

XO

Student’s fruit preferences by eating occasion(importance of time in consumption decisions)

grape

kiwi

apple

plum

orange

stra

wberry

banan

a

peach

grape

kiwi

apple

plum

orange

stra

wberry

banan

a

peach

grape

kiwi

apple

plum

orange

stra

wberry

banan

a

peach

Breakfast

Snack during day

Supper dessert

Preferences

Physical Characteristics of Questionnaire: Recommendations

• questionnaire should appear simple to complete (white space is your friend!)

• minimize number of pages (smaller fonts are OK, provided form appears simple) & items

• mix-up response formats occasionally (avoids response set bias and breaks monotony)

• use directions as necessary for each group of items, but keep them short and simple

• number items within each section• have respondents check boxes rather than lines• Use consistent response formats (don’t change too often)• use shading, boxes, lines, etc., to keep it interesting

Question considerations

• start with easy, non-threatening questions (warm up)

• place more personal, threatening questions at end

• never start mail survey with open-ended questions

• for historical demographics, follow chronological order

• ask about one topic at a time

• when switching topics use a transition

• reduce response set

• for filter or contingency questions, make a flow chart

• thank the respondent at beginning and end

• keep the survey short

• keep language simple

• include “don’t know” and “NA” for appropriate options

• try not to make questions dependent on each other

Foot in the door…

First make a small request, then when granted, make a larger (more desirable) request

• works best with prosocial/altruistic requests

• works best if no extra incentives are offered

• 10% improvement over simple request, 20% if no incentive

• works by cognitive dissonance & self esteem

Door in the face…

Make a large request, then when it is refused, make a smaller (more desired) request

• more effective if prosocial/altruistic (do it for everybody) than selfish

• should be no delay between requests

• slight delay produces 10% improvement, no delay 20%

• works by reciprocal concessions

Supplementary Slides(not assigned for study)

Scale Development

• define target population

• define specific domains to be measured

• generate items reflect each domain

• construct item format

• construct survey

• test items for feedback

• revise

• administer to representative group

• frequency distribution (representative?)

• Cronbach’s alpha (above .70?)

• validation & reliability checks

• complete final version

Items Items

Items

Cronbach’s alpha as a measure of scale internal reliability

• Frequency distribution• Cross tab & chi-square• Correlation & regression• t-test & ANOVA• Cluster analysis• CHAID• MDS

Analyzing Data

Frequency distribution

Crosstabs & chi-square

If significant, then rows & columns are dependent

Correlation & regression

r = .50r2 = .25

T-test:Comparison of differences

between means

ANOVA &Post Hoc

Comparisons

Each cluster represents a similarity grouping

Cluster Analysis: hierarchical grouping on the basis of similarity

Unity

Multidimensional Scaling

People with older computers more likely

to upgrade

Sole user more likely to personalize

by upgrading

Influence of system

characteristics

Online realize need for faster modem

CHAID

Focus on less recent PC buyers, target for upgrade packages with faster modems

Comparison of Three Primary Methods of Administration

mail telephone personal(A) Sampling Controlability to secure sampling frame + +ability to secure correct respondent - + +response rate - + +

(B) Information Controlability to probe for detailed answers - +ability to handle complex information - +amount of information obtained - +flexibility of question sequencing - + +protection from interviewer bias + - -ability to obtain personal information + - -ability to show visual displays - +ability to establish rapport with respondent - +ability to offer anonymity + - -

(C) Administrative Controltime requirements - +cost requirements + -quality control / supervisory requirements + -

Return Rate

• sample size calculator

• notify beforehand

• free postage (SASE)

• personalize (name & address)

• relevant to needs & interests

• incentives

• closing deadline in 2-2.5 weeks

• reminder

• security, confidentiality, anonymity

• if <50% check for bias (what have in common)

• mail-ins have 3-5% response rate (10% is good)

Considerations for Web Based Surveys

• Draft the questions

• Provide a password (if needed)

• E-mail reminders (know return cutoffs)

• Personalize them (mailing list)

• Design for people with minimal skills

• Use a larger font

• Response icons

• Textured background

• Open questions use a text box

• E-mail to researcher for comments

• Thank you page

Personal and Telephone Interviews:

“foot-in-the-door” strategy (Reingen & Kernan 1977)

Mail Surveys

(1) pre-notification, especially by telephone (Schelegelmilch & Diamantopoulos 1991)

(2) cover letter / type of appeal (Houston and Nevin 1977)

(3) use SASE, not business reply envelope

(4) personalized address

(5) no bulk mail

(6) sponsorship (if corporate, refer to “research department” and never include sales pitch

(7) monetary incentives (Brennan, Hoek & Astridge 1991)

(8) offer survey results

(9) follow-up postcard, duplicate questionnaire, or telephone call