Measuring Sensitivity

Robin KaplanErica Yu

Presentation for theAmerican Association of Public Opinion Research

(AAPOR)*All views expressed in this paper are those of the authors and do

not reflect the views or policies of the BLS.

Forgiving Wording andSensitivity

Wines and champagnes have become increasingly popular inthis country over the last few years. Did you ever drink, evenonce, wine or champagne? (Source: Gallup, 1977)

In any election, some people are not able to vote because theyare sick or busy or have some other reason, and others do notwant to vote. Did you vote in the election held on Tuesday,November 7th? (Source: U.S. Census Bureau, 2001)

Some people work part time because they cannot find full timework or because business is poor. Others work part timebecause of family obligations or other personal reasons. What isyour main reason for working part time? (Source: Current PopulationSurvey, Bureau of Labor Statistics)

2

Effectiveness of ForgivingWording

Decreased overreportingof socially desirable behavior (e.g., Belli et al., 1999, 2006; Holtgraves et al., 1997)

No effect on the over- or underreporting of sociallydesirable behavior (e.g., Abelson et al., 1992; Sudman & Bradburn; Presser, 1990)

Only effective for respondents high in socialdesirability or who have strong attitudes about thesurvey topic (e.g., Peter & Valekenburg, 2011; Naher & Krumpal, 2012)

3

Research Questions

How does forgiving wording affect theperceived sensitivity of surveyquestions?

How does forgiving wording affectsensitivity for both the respondent andinterviewer?

4

The Present Study 2x2 between groups factors

1 repeated measure: 4 vignettes Varied the how sensitive the context would likely feel

Dependent variables (5-point Likert-type scales) Sensitivity ratings (respondent and interviewer) Empathy for the vignette character Attitudes toward the vignette character and survey topic 5

Ask forgivingwording questions

Answer directquestions

Perspective

Que

stio

nW

ordi

ng

Interviewer Respondent

Direct

Forgiving

Vignette Examples

Neutral Context Sensitive Context

6

Wording Intervention:Unemployed and Looking for Work

Direct Question

Have you been doing anythingto find work during the last 4weeks?

Forgiving WordingDespite wanting fulfilling jobs thatcontribute to society, many peopletake a long time to find workbecause business is poor, or dueto a lack of job openings in theirindustry or geographic location.

Have you been doing anythingto find work during the last 4weeks?

7

Participants

N = 432 from Amazon’s Mechanical Turk Demographics:50.3% Male; 49.4% FemaleMean age = 35.61 (SD = 11.58)Median education = Associate’s/Bachelor’s degree

Compensation:$1 with a mean task duration ~12 minutes

8

Results:Sensitivity Ratings by Perspective

The neutral vignette was rated as less sensitive than the 3 sensitive vignettes across bothperspective types, F(1,422) = 73.86, p < .001.

Participants taking on the respondent perspective rated the vignettes as more sensitivethan those taking on the interviewer perspective, F(1, 270) = 55.64, p < .001.

Participants taking on the interviewer perspective rated the sensitive vignettes as moresensitive than the neutral vignette (all ps < .001). 9

1

2

3

4

5

Respondent Perspective Interviewer Perspective

Mea

n Se

nsiti

vity

Rat

ing

(low

to

high

)

Neutral Context

Sensitive Context*

*

Results:Sensitivity Ratings by Wording

Forgiving wording increased sensitivity across all participants, F(1,270) = 4.26, p =.048.

10

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Direct Question Forgiving Wording

Mea

n Se

nsit

ivit

y R

atin

gs(l

ow t

o hi

gh)

*

Results:Respondent Perspective

Sensitivity ratings for direct questions and forgiving wording did not differin the neutral or sensitive survey contexts for participants taking on therespondent perspective (ps > .30). 11

1

1.5

2

2.5

3

3.5

4

Neutral Context Sensitive Context

Mea

n Se

nsiti

vity

Rat

ing

(low

to

high

)

Direct Question

Forgiving Wording

n.s.

n.s.

Results:Interviewer Perspective

12

Participants taking on the role of the interviewer rated questions usingforgiving wording as more sensitive than direct questions in both theneutral context, t(142) = -2.63, p = .009, and the sensitive context,t(139) = -1.99, p = .049.

1

1.5

2

2.5

3

3.5

4

Neutral Context Sensitive Context

Mea

n Se

nsiti

vity

Rat

ing

(low

to

high

)

Direct Question

Forgiving Wording*

*

Regression Analysis onSensitivity Ratings

13Note: Empathy and Attitudes were centered at Step 3; Adjusted R2=.29 at Step 4.*** p < .001; ** p < .01, * p < .05; N = 432

Predictor β p-value

Perspective Type(1 = interviewer, 0 = character)

-0.48 <0.001

Question Wording Intervention Type(1 = forgiving wording, 0 = direct wording)

0.10 0.05

Empathy rating 0.03 0.54

Attitude ratings 0.004 0.94

Empathy x Attitudes Interaction 0.16 0.003

Interaction:Empathy X Attitude

14

1

1.5

2

2.5

3

3.5

4

4.5

5

Negative Attitude Positive Attitude

Ave

rage

Sen

siti

vity

Rat

ing

(low

to h

igh)

Low empathyHigh empathy

Negative(Bad rapport)

Positive(Good rapport)

Summary

“Respondents” felt more sensitivity than“interviewers”

Interviewer sensitivity was also present Forgiving wording increased sensitivityPositive sensitivity: A lot of empathy and

positive attitudesNegative sensitivity: Little empathy and

negative attitudes

15

Conclusionsand Future Directions

Relationship between forgiving wording and sensitivitymay be more nuanced than previously thoughtPositive and negative forms of sensitivity

The concept of interviewer sensitivity deserves furtherinvestigationEffects on data quality are unknown

Develop standardized measures to assess bothrespondent and interviewer sensitivity

16

Contact Information

Robin KaplanErica Yu

Research PsychologistsOffice of Survey Methods Research

www.bls.gov/osmrKaplan.Robin@bls.gov

Yu.Erica@bls.gov

Construct of ‘Sensitivity’

How personal, invasive, threatening, oruneasy a question makes respondentsfeel

3 major components (Tourangeau & Yan, 2007):

Intrusiveness of the questionThreat of disclosureSocial desirability

18

Sensitivity Measure

How sensitive do you think [vignettecharacter / the interviewer] feltwhile [answering / asking] this question?

o Not at all sensitiveo Slightly sensitiveo Moderately sensitiveo Very sensitiveo Extremely sensitive

19

Empathy and AttitudeQuestions

How easy or difficult was it to put yourself in [vignettecharacter’s] shoes when reading the scenario?o Very easyo Somewhat easyo Neither easy nor difficulto Somewhat difficulto Very difficult

How negatively or positively do you think most peoplewould feel about [vignette character]?o Very negativelyo Somewhat negativelyo Neither negatively nor positivelyo Somewhat positivelyo Very positively

20

Individual Vignette Attitude Questions(5-point Likert-type scales)

Thinking about the real world now…

How easy or difficult is the job market facing recentcollege graduates these days?

In your opinion, how important would you say thatmanufacturing is to the health of the US economy?

In your opinion, how easy or difficult is it for olderpeople to find jobs these days?

In your opinion, how important would you say it is forcompanies to ensure all of their employees can makeends meet?

21

Wendy Interview Excerpt

22

Interviewer: [Forgiving wording] Have you beendoing anything to find work during the last 4weeks?

Wendy: Well, I worked on my resume and went on 4job interviews. I heard back from 2 of them, but Ihaven’t gotten an offer yet.

Interviewer: Did you do anything else?

Wendy: No.

Beth Interview Excerpt

23

Interviewer: [Forgiving wording] Have you beendoing anything to find work during the last 4weeks?

Beth: Like I mentioned earlier, the assemblyplant was shut down. I looked for work for over 6weeks – it was 46 days, I counted - and havecome up with nothing so far.

Other Vignettes

24

Participant EmploymentStatus

Employed full-time 57.9% Employed part-time 14.4% Unemployed 18.8% Student 6.0% Retired 3.0%

25

Sensitivity Ratings by VignetteContext and Wording

26

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Neutral Context Sensitive Context

Mea

n Se

nsiti

vity

Rat

ing

(low

to

high

)

Direct Question

Forgiving Wording

*

Instructions –Respondent Perspective

27