Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @...
Transcript of Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @...
![Page 1: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/1.jpg)
Designing a Web Survey for
the 2019 SCF
Lisa Lee, NORC @ University of Chicago*
Shannon Nelson, NORC @ University of Chicago
Catherine Haggerty, NORC @ University of Chicago
2019 FedCASIC
April 16, 2019
Washington, D.C
![Page 2: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/2.jpg)
GOALS
Conduct 2019 SCF Web Test
Are respondents willing to provide financial information via
web?
Confidentiality
Data security
Are respondents able to provide high quality financial
data?
Dollar amounts instead of ranges
No missing data
Use of records
![Page 3: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/3.jpg)
POTENTIAL THREATS TO DATA QUALITY
Potential threats to data quality in web version
• Increased burden
• Satisficing
• Missing data
• Poor quality data
![Page 4: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/4.jpg)
METHOD
Sample is subset of the AP sample
• 2500 cases in three areas
• Chicago, Ithaca, Albuquerque
Web survey includes several sections of the CAPI SCF
Data collection activities
• Invitation(s) to participate
• Complete web component
• Complete CAPI component
• Incentives
![Page 5: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/5.jpg)
PROCESS: DESIGNING, TESTING, AND FIELDING
Designing the SCF Web Test Survey
STEP 1
Usability testing
STEP 2
Cognitive-
usability
testing
STEP 3
Field pretest
STEP 4
2019 SCF Web Test
data collection
STEP 5
Data processing
and delivery
STEP 6
![Page 6: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/6.jpg)
STEP 1: DESIGN DECISIONS
Includes five sections of the full survey
• Household Listing
• Section A: Economic Expectations, Credit Attitudes,
Financial Institutions
• Section G: Vehicles
• Section N: Financial Assets
• Section T: Income
Programmed in MrInterview
Designed to facilitate the question answering process in
the absence of an interviewer
Designing the SCF Web Test Survey
STEP 1
![Page 7: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/7.jpg)
STEP 2: USABILITY TESTING
Conducted five usability testing sessions
• NORC colleagues provided feedback
Usability testing protocol
• Enter data based on mock respondent profile
• Visual layout
• Navigation
• Clarity of instructions, questions, and response options
• Unfamiliar terms
• Suggestions for improvements
Usability testing
STEP 2
![Page 8: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/8.jpg)
STEP 3: COGNITIVE-USABILITY TESTING
Test usability of the instrument while exploring
issues related to response
In-person interviews at NORC’s downtown Chicago
office
• Recruited 18 respondents
• Incentive for participation
• Encouraged to bring and use records
Cognitive-
usability
testing
STEP 3
![Page 9: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/9.jpg)
STEP 3: COGNITIVE-USABILITY TESTING
Cognitive interview protocol
• Explored comprehension of questions and
terms, accuracy of responses
• Observed respondent ability to navigate the
instrument and enter responses
• Collected feedback on how to improve the
survey
Cognitive-
usability
testing
STEP 3
![Page 10: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/10.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Household Listing—Before
![Page 11: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/11.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Household Listing—After
![Page 12: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/12.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Household Listing—After
![Page 13: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/13.jpg)
Knowledgeable about Personal Finance—Before
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
![Page 14: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/14.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Knowledgeable about Personal Finance—After
![Page 15: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/15.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Listing Financial Institutions—Before
![Page 16: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/16.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Listing Financial Institutions—Before
![Page 17: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/17.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Listing Financial Institutions—After
![Page 18: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/18.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Listing Financial Institutions—After
![Page 19: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/19.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Collecting Dollar Amounts and Ranges
![Page 20: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/20.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Collecting Dollar Amounts and Ranges
![Page 21: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/21.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Collecting Dollar Amounts and Ranges
![Page 22: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/22.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Help Text
![Page 23: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/23.jpg)
STEPS 2 & 3: COGNITIVE-USABILITY TESTING RESULTS
Help Text
![Page 24: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/24.jpg)
STEP 4: PRETEST METHODOLOGY
27 respondents participated in web pretest
• Completed web survey
• In office vs. at home
• Debriefing
• Completed CAPI survey
• Debriefing
Field
Pretest
STEP 4
![Page 25: Designing a Web Survey for the 2019 SCFDesigning a Web Survey for the 2019 SCF Lisa Lee, NORC @ University of Chicago* Shannon Nelson, NORC @ University of Chicago Catherine Haggerty,](https://reader033.fdocuments.us/reader033/viewer/2022050311/5f72ee55828ce47e9c505915/html5/thumbnails/25.jpg)
STEP 4: SELECTED PRETEST FINDINGS
Field
Pretest
STEP 4
Percent of respondents reporting exact dollar values
Type of Asset 2016 SCF CAPI 2019 SCF
Web Test
Checking 85 96
Savings 86 93