Using Visual Design Theory to Improve Skip …Using Visual Design Theory to Improve Skip...

Survey Research and MethodologyUNL-GALLUP Research Center


Using Visual Design Theory toImprove Skip Instructions:

An Experimental Test

Using Visual Design Theory toImprove Skip Instructions:

An Experimental Test

Nikki Gohring, University of Nebraska – Lincoln

Dr. Jolene Smyth, University of Nebraska - Lincoln

11



Using Address Based Sampling

• Due to availability of the Delivery Sequence File,Address Based Sampling (ABS) becoming more popular

• ABS often uses mail surveys

• Potential issues with mail surveys:– Respondents are left to navigate the survey themselves– More complicated surveys cause more problems

• Skip patterns can increase complexity

22



Purpose

• If respondents are not following skip instructionscorrectly, this can lead to nonresponse andmeasurement error

• Visual design can be used to reduce respondent burden,leading to fewer errors in the survey

• Experiment using visual design principles to test whetherrespondents are able to navigate skip patterns correctly

33



Previous work done with visual design andskip instructions

Redline, Dillman, Dajani, & Scaggs (2003)

• Nationwide experiment using the 2000 Census

• Five Treatments

– Census 2000 skip instruction

44




– The “Go To” instruction

– The (Go To) Reverse Print Instruction

55




– The “Go To” Prevention Instruction

– The (Go To) Detection Instruction

66



Types of Errors

• Errors of omission– Failure to answer a follow-up question when instructed to do so

– Leads to nonresponse

• Errors of commission– Answering a follow-up question when instructed to skip it

– Can lead to measurement error• Ex. Are you currently employed? No

What is your current occupation? Retail

Which answer is correct?

(Redline and Dillman, 2002)

77



Implications of the Census Experiment

• Errors of Omission– Ranged 4.0% to 7.6%

• Errors of Commission– Ranged 13.5% to 20.8%

• We still have a lot to learn about how to effectivelydesign skip instructions

• We have learned a lot about visual design since thisexperiment was done

(Redline, Dillman, Dajani, Scaggs, 2003)

88



Visual Design Principles

• Information can be visually communicated through fourelements: Words, Numbers, Symbols and Graphics

• We can use the visual properties of these elements toguide respondents to process our questionnaire how wewant them to– Size, Color, Font, Brightness, Contrast, Shape, Orientation, etc.

• In surveys, use of graphic tools, such as alignment, cancreate a path through the survey

(Dillman, Gertseva, & Mahon-Haft, 2005)

99



Gestalt Psychology Principles

• Gestalt principles – How we use these visual properties tocreate visual groupings

– Principle of Continuity• Items that visually run together will be seen as a continuous single

element

– Principle of Proximity• Items that appear close together will be perceived as belonging to a

group

– Principle of Common Region• Elements enclosed in a single region will be perceived as belonging to a

group

– Principle of Element Connectedness• Elements tend to be grouped together if they are connected by other

elements

(Dillman, Smyth, & Christian, 2009)

1010



Experiment - Data

• Nebraska Annual Social Indicators Survey (NASIS)– ABS mail survey of the state of Nebraska (3600 households)– Measures quality of life and topical information

• Conducted Summer 2012 (June-September 2012)

• 959 completed surveys (AAPOR RR1 28.7%)– Version 1 - Conventional Design

• 466 completes

– Version 2 - Experimental Design• 493 completes

• Six questions within the survey triggering a skip

1111



Experimental Design

• Two versions of the survey– Same questions, order, orientation– Same skip pattern formatting within each version– Randomly assigned to households

1212

Version 1: Conventional Design Version 2: Experimental Design



Experimental Design

Q4: Have you had interaction with an employee of theDepartment of Roads within the past year?

Q9: In the past 12 months, have you stopped at one ofNebraska’s I-80 rest areas?

Q50: Do you think females commit sex crimes?

1313



1414

Version 1: Conventional Design Version 2: Experimental Design

Experimental Design



Experimental Design

Q62: What is your current marital or relationship status?

Q91: Do you typically work full-time, part-time, go to school,keep house, or something else? (Check all that apply)

Q97: Do you consider yourself to be Protestant, Catholic,Jewish, Muslim or something else?

1515



Hypothesis

The changes to the visual design will aid the respondent incorrectly navigating skip patterns, leading to fewer errors

– Total number of errors– Errors of Omission– Errors of Commission– Respondents that skip the “feeder” question, but still follow the

skip pattern

1616



Findings

1717



0%

2%

4%

6%

8%

10%

12%

14%

16%

Percentage of Respondents making any type oferror in skip patterns

1818



0%2%4%6%8%10%12%14%16%18%Percentage of respondents making any type of

error – by design

Conventional Design Experimental Design

1919

† p<0.1 * p<0.05 ** p<0.01



2020

0%

2%

4%

6%

8%

10%

12%

14%

Percentage of respondents makingErrors of Omission


† p<0.1 * p<0.05 ** p<0.01



2121

0%1%2%3%4%5%6%7%8%9%

10%

Percentage of respondents makingErrors of Commission


† p<0.1 * p<0.05 ** p<0.01



Demographics

• No significant difference in errors made between agegroups

• No significant difference in errors made between menand women

• Respondents with less than high school education weremore likely to make errors – Errors of Commission

• In using the Self-Assessed Literacy Index we did seesignificant difference errors of commission between highand low literacy on two variables (Olson et al., 2011)

2222



Breakdown of the types of errors made in theExperimental Version

• Errors of Omission– Ranged <1% to 12.5%– Much like the Census Experiment, the change in design didn’t do

much to reduce the number of respondents making this type oferror

• Errors of Commission– Ranged 1% to 4.6%– Saw a significant reduction in 3 of the 6 variables

• Skipping the “Feeder” question– Ranged 0.2% to 3%

2323



Limitations

• Yes/No options reversed on skip questions only

• “Other specify” options

• “Check all that apply” option

• Some questions required skippingmultiple questions

• Location at the bottom of a page– Lost the benefit of the indention– Turning the page interrupts cognitive

process, resets(Dillman, Redline, & Carley-Baxter, 1999)

2424



In Conclusion…

• We came up with more questions than we did answers

• Saw promising improvements on Errors of Commission– Potentially reduce measurement error

• Not willing to give up yet!– Further research needed to address questions– As long as we continue to use ABS and mail questionnaires, it is

important that we continue to explore the best way to design ourquestionnaires.

2525



Thank you!Contact information:

Nikki [email protected]

2626



References• Dillman, D. A., Gertseva, A., & Mahon-Haft, T. (2005). Achieving Usability in Establishing

Surveys Through the Application of Visual Design Principles. Journal of Official Statistics,183-214.

• Dillman, D. A., Redline, C. D., & Carley-Baxter, L. R. (1999). Influence of Type of Questionon Skip Pattern Compliance in Self-Administered Questionnaires. Proceedings of theSurvey Research Methods Section, American Statistical Association.

• Dillman, D. A., Smyth, J. D., & Christian, L. (2009). Internet, Mail, and Mixed-ModeSurveys: The Tailored Design Method. Hoboken, New Jersey: John Wiley & Sons, Inc.

• Redline, C. D., & Dillman, D. A. (2002). The Influence of Alternative Visual Designs ofRespondents' Performance with Branching Instructions in Self-AdministeredQuestionnaires. In R. Groves, D. Dillman, E. Eltinge, & R. Little, In Survey Response. NewYork: John Wiley and Sons, Inc.

• Redline, C., Dillman, D. A., Dajani, A. N., & Scaggs, M. (2003). Improving NavigationalPerformance in U.S. Census 2000 by Altering the Visually Administered Languages ofBranching Instructions. Journal of Official Statistics, 408-419

2727



2828

Variables



2929



3030



3131



3232



3333

Using Visual Design Theory to Improve Skip …Using Visual Design Theory to Improve Skip...

Documents

Transcript of Using Visual Design Theory to Improve Skip …Using Visual Design Theory to Improve Skip...