Post on 07-Jun-2020
The Promises and Perils of Pushing Respondents to the
Web in Mixed-Mode Surveys For
9th Colloque Francophone sur les sondagesGatineau, Quebec, Canada
11 au 14 Octobre 2016
ByDon A. Dillman, Regents Professor
Department of Sociology and the Social and Economic Sciences Research Center
Washington State University, Pullman, WA 99164-4014 dillman@wsu.edu
A brief look to the past
• In 1989 an iconic American movie, Field of Dreams, was created around the idea that if a baseball field were to be built in a rural Iowa corn field, “They (the players and people) will come.”
• In the late 1990’s a similar idea developed about conducting surveys by email with responses sent over the Internet.
• Hopefully, thepeople would not only come, but the cost of surveying would decline dramatically, making it a surveyor’s field of dreams.
• This reality has yet to be fully realized.
c Don A. Dillman, September 1, 2016 2
Three parts to this presentation
• Part I. The decline of telephone and difficulty with finding alternatives.
• Part II. A decade of research aimed at developing an effective web-push methodology.
• Part III. Specific challenges associated with the promises and perils of web-push surveying.
c Don A. Dillman, September 1, 2016 3
My specific focus
• My focus in this talk is on probability sampling of the general public.
• In this context I am also focused on simultaneously reducing four types of error: coverage, sampling, nonresponse and measurement.
• By web-push I am referring to initially asking people to respond only over the web, while reserving alternative response modes until later in the data collection process.
• Mixed-mode refers to using more than one survey mode as contact and/or response mode.
c Don A. Dillman, September 1, 2016 4
Part I.
The Decline of Telephoneand difficulty with finding
alternatives
c Don A. Dillman, September 1, 2016 5
Around 2005 it became clear that the voice telephone era of RDD household surveys
was coming to an end
• The presenting problems for RDD surveys were:– Low response rates.– Need to combine individual (cell) and household (land) lines.– Need for brevity.– Need to devote questions to correcting for area code transportability,
ownership of cell/landline access, not driving a vehicle, age of respondent.
– People answered phone only once, so that it is becoming a “one chance to persuade” (in 5 seconds or less) methodology
• However, cultural change is the biggest problem; the telephone is no longer used for ordinary conversations.
• Most telephone communication is asynchronous; texts and emails.
c Don A. Dillman, September 1, 2016 6
The hoped for Email contact/web response replacement, is not yet working
well. • Coverage for household populations remains
limited (<80%), some on slow connections.• Individual internet skills were limited among some
adults, especially the less educated, older and lower income individuals.
• No email sample frame for household selection comparable to telephone RDD.
• Email only contact produces response rates similar to phone.
c Don A. Dillman, September 1, 2016 7
The hoped for Email contact web response replacement, is not yet working well (2).
• Lack of TRUST on multiple levels; sponsorship, misrepresentation of purpose, threat of malware, and consequences of making a mistake.
• We are moving from age of desktops, to laptops, (and now) to devices, which are not optimally designed for answering surveys—screen size affects when and how they get used.
• The “purse and pocket” problem combined with the smartphone response situation of people being “on the move” from one task to another makes getting lengthy and thoughtful survey responses very difficult.
c Don A. Dillman, September 1, 2016 8
I am uncomfortable with some of the assessments and proposed solutions
• “Response rates no longer matter” (Higher the response, the better our chance of going after non-responders)
• “All response modes are facing declining response rates” (less true for mail than telephone or the Internet)
• “Repair the telephone” (Too many problems: coverage, brevity, extra questions, minimal opportunity to persuade, CULTURE moving away from voice exchanges)
• “Give households the Internet and rely on web-only panels” (Too many people don’t trust or are uncomfortable with computers and internet; results are biased)
• “Give people a choice of modes” (Most choose mail)
c Don A. Dillman, September 1, 2016 9
Postal addresses have emerged as sample frame with most complete household
coverage• Early focus (2007) was on drawing area-probability samples for personal
interviewing.• I am struck by the high coverage rates for households—98 %—in the U.S. Postal
Service “Delivery Sequence File” available as addresses without names.• If this could work for personal interview samples, why not mail surveys.• Reason for my interest in mail surveys was:
– I could deliver incentives with the survey request (much more effective that post payments)– I could deliver follow-up contacts in different forms to offer additional arguments for
responding.– Research done in the 90’s at the Census Bureau convinced me that we didn’t need names
for mail methods to be effective.– I had not seen evidence of a precipitous decline in mail response rates as happened for
telephone.– I could try to “push” people to the web instead of asking for a paper response.
c Don A. Dillman, September 1, 2016 10
In order to replace telephone with a web-push methodology…
• We needed a better sample frame than “random email addresses”. Postal mail appeared to be the only realistic choice.
• Using postal addresses would require mixing survey modes:– a different contact mode than response mode. – And might involve obtaining responses from more than one mode (e.g.
mail or telephone)– Mixing modes could introduce measurement issues
• Potential cost savings provided a powerful motivation for finding ways to make this work.
• Acceptance by the survey methodology community could be challenge because of losing “interviewer assistance”, our standard way of eliciting answers to survey questions.
c Don A. Dillman, September 1, 2016 11
Part II.
A decade of research aimed at developing an effective web-
push methodology
c Don A. Dillman, September 1, 2016 12
The Prototype (Lewiston-Clarkston) Experiment
• Address-based sample.• Push-to-web with mail contact (send mail response option in
third contact). • Undo old mail strategies, e.g. name personalization, envelopes,
use of postcard.• Token cash incentives with response request• Unified mode construction to reduce measurement differences.• Tailor to population with graphic design and color.• Reasonable length: At least 20 minutes of questions• My background model was, “What I thought could be applied
to the American Community Survey”
c Don A. Dillman, September 1, 2016 13
Conceptually, I was trying to undo earlier mail procedures (1978-2000) and use the visual design and mixed-mode concepts
introduced in (2000 and 2009) versions of book. Each revision was substantial (see% below). New concepts are coming at us faster and
faster
1978 2009~75%
2014~65%
2000~95%
c Don A. Dillman, September 1, 2016 14
c Don A. Dillman, September 1, 2016
2007 Lewiston-Clarkston Study
• 51 numbered questions, 90 responses, 10 pages (paper)—a 20 minute (if it were telephone) conversation
• Four contacts.1. Pre-notice letter.2. Questionnaire (or web request).3. Thank-you post card.4. Replacement questionnaire (adjusted by treatment).
• $5 token cash incentive included with mail questionnaire or web contact
(Later studies would change number and nature of contacts)
15
We compared four treatments
1. Mail preference with web mention: Send mail questionnaire and mention web with initial request
2. Push-to-mail: Send mail questionnaire but withhold mention of web for about two weeks
3. Push-to-web: Web invitation with no mail questionnaire, but explain that mail questionnaire will be sent in about two weeks
4. Equal preference: It is your choice!
c Don A. Dillman, September 1, 2016 16
Larger plain envelopes (even for web-only request) to get them opened
• Normal business stationery more likely to be ignored.• Used a return label showing the photo from survey cover and the
survey title to increase familiarity.
© Dillman17c Don A. Dillman, September 1, 2016
Correspondence tailored to city address
• All letters used WSU stationary (legitimation).
• $5 affixed to letter (to get it read)
• Photo of questionnaire cover used to tie different elements together (interest).
18c Don A. Dillman, September 1, 2016
c Don A. Dillman, September 1, 2016
A picture of the Lewiston and Clarkston region identified the survey area (confluence of the
Snake and Clearwater Rivers)
19
c Don A. Dillman, September 1, 2016
Additional pictures identified the survey area
• Photos taken of local landmarks, artwork, and symbols to make survey recognizable and visually attractive.
20
c Don A. Dillman, September 1, 2016
Mail Questionnaire tailored with broad survey topic, who should respond, and back-page pictures
Use of tailored images to help connect respondents to survey and to place an emphasis on study area instead of on survey source. 90 responses requested
21
c Don A. Dillman, September 1, 2016
Web questionnaire was similarly tailored
• Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images.
22
c Don A. Dillman, September 1, 2016
Web survey followed unified mode construction principles to link modes
• Question 2– Similar design format to paper survey, and use of familiar image in upper left-hand corner of
the screen. Emphasized respondents over sponsors (using other communications to do that).
23
c Don A. Dillman, September 1, 2016
Second example of unifying web presentation with mail presentation
Question 13 The small image would change every ten questions. Trying to show all of the
same images as the paper survey, as well as keep interest and familiarity.
24
Mail (on left) and web (on right) unified to control measurement
c Don A. Dillman, September 1, 2016 25
Initial withholding of mail drove 41% to the web; paper follow-up added 14%. Offering
choice drove ~80% to mail .
Treatments Web (%) Paper(%) Total (%)
Mail preference with web mention
4 58 62
Push-to-Mail(web in third contact)
1 70 71
Push-to-webMail questionnaire sent in 3rd of 4 contacts
41 14 55
Equal preference (choice) 13 50 63
c Don A. Dillman, September 1, 2016 26
High response rates are desirable but we need also to focus on nonresponse error
• It does not help much to improve response rates if our respondents are different from non-respondents on variables important to the study objectives
• Thus, we need to compare respondent characteristics on web vs. mail within the different treatment groups.
• If respondents are different from non-respondents in ways that affect results of the data analysis, then we have a significant problem
c Don A. Dillman, September 1, 2016 27
In the push-to-web treatment, web and mail respondents
were quite different
c Don A. Dillman, September 1, 2016
0
10
20
30
40
50
60
70
80
90
100
* p ≤ .05
** * *
*
*
*
28
But the complete push-to-web group was quite similar to the complete
push-to-mail treatment
c Don A. Dillman, September 1, 2016
*
* p ≤ .0529
0
10
20
30
40
50
60
70
80
90
100
Push-to-web
Push-to-mail
Initial success encouraged us to do four more large-scale tests to evaluate the web-push methodology
0 Lewiston, ID-Clarkston, WA Mail Test 20051. Lewiston, ID-Clarkston, WA Survey 20072. Washington Community Survey 20083. Washington Economic Survey 20094. WA, PA, AL Tri-state Electricity Survey20115. WA and NE Water Management Survey2012• Needed multiple topics and populations• Research goal was to refine through subtraction and addition experiments,
e.g. token cash incentives, other state populations, questionnaire design, respondent selection, location of sponsor, etc.)
• ~ 28 additional experiment treatments, with successful panels being carried forward.
For summarized detail, see Chapter 11 of Dillman, D., Smyth, J.D., Christian, L.M. 2014. “Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition. (John Wiley, Hoboken, NJ)
c Don A. Dillman, September 1, 2016 30
Tests 2-5; Changes in burden, topic, and procedures (e.g. no pre-notice, no postcard reminder)
• Questionnaire— 20-30 minutes to complete. (equivalent to ten 8 ½ x 11” pages of paper. Up to 140 individual questions
• General Implementation Protocol : specifics varied somewhat across studies.Week 1: Postal request that includes $4- $5 token cash incentive with
request to respond by web.Week 2: A thank you/reminder letter (not postcard).Week 5: Another letter request with a mail questionnaire and, in some cases, an additional smaller token cash incentiveWeek 6: A final thankyou/reminder.
c Don A. Dillman, September 1, 2016 31
2008 Washington Community Survey (WCS): first statewide test
• Example of the mail version:
c Don A. Dillman, September 1, 2016 32
A $5 cash incentive (WCS) more effective for web-push than paper-only
$5 Mail Mail w/o $50
10
20
30
40
50
60
39,2
$5 Internet Internet w/o $50
10
20
30
40
50
60
31,3
13,4
(+17.9*)
(+13.3*)
* p ≤ .05 c Don A. Dillman, September 1, 2016 33
Push-to-web had lower total response, but 2/3 of responses came over the web. For push-to-mail, a
late web request had very small effect.
$5 Push-to-web w/card $5 Push-to-mail $5 Mail-only0
10
20
30
40
50
60
31,3
51,9
$5 Mail-only; 0
15,0
3,1
56,7
c Don A. Dillman, September 1, 2016
46%
55%57%
34
Introducing a paper questionnaire in third contact brought in different kinds of respondents than did the initial push-to-web effort. (Messer
and Dillman, 2011)
Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)0
10
20
30
40
50
60
70
80
90
Web Mail follow-up
**
** *
*
* p ≤ .05
c Don A. Dillman, September 1, 2016 35
The combined push-to-web group (web plus mail respondents) was demographically
similar to the mail-only treatment
c Don A. Dillman, September 1, 2016 36
Education (HS or less) Age (65+) # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)0
10
20
30
40
50
60
70
Push-to-web Mail-only
Complete push-to-web group was more representative than the web-only
respondents(comparison to U.S. Census Bureau’s American Community Survey)
c Don A. Dillman, September 1, 2016 37
Education (HS or less)Children in HH # in HH (2 or less) Married (% Yes) Employed (% Yes)Income ($25/year or less)5
15
25
35
45
55
65
75
Web-only Push-to-web ACS
2009 statewide economic survey, tailored to state with map and pictures
• 11” X 8.5” booklet with ‘personalized’ images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey. (Messer and Dillman, 2012)
38c Don A. Dillman, September 1, 2016
2011 Three state Electricity Survey, with tailored covers
• Examples of the mail covers:
c Don A. Dillman, September 1, 2016 39
2012 water management survey: tailored to state by pictures and sponsorship• Examples of the mail covers:
c Don A. Dillman, September 1, 2016 40
Response rates for 10 statewide web-push (mean, 43%, with 62% over the web) vs. mail out/mail back (mean, 53%) surveys,
2007-2012 (Dillman, Smyth and Christian, 2014, Chapter 11).
c Don A. Dillman, September 1, 2016 41
Does mail item non-response negate gains in response rates?
• If mail higher item non-response rates then perhaps that cancels out the benefits of gaining the additional responses in web-push designs.
• Thus, it was important to evaluate this. (See Messer, Edwards and Dillman, Survey Practice, 2012).
c Don A. Dillman, September 1, 2016 42
Web vs. mail item non-response in the push-to-web groups for LCS, WCS, and WES
• Item nonresponse rates lower for web
2007 LCS 2008 WCS 2009 WES0
2
4
6
8
10
12
14
2,7 2,7
6,16,26,9
11,6
c Don A. Dillman, September 1, 2016 43
BUT, push-to-web and mail-only groups have similar overall item nonresponse rates; Late mail
respondents in push-to-web groups are probably less capable respondents(older with less education).
2007 LCS 2008 WCS 2009 WES0
2
4
6
8
10
12
14
3,64,2
8,0
5,04,2
8,1
Push-to-web
c Don A. Dillman, September 1, 2016 44
Summary of Major findings 1. 10 Postal-only response = 53% (38% to 71%). 2. 10 Push-to-web response = 43%(31% to 55%).3. Offering initial choice pushes most people (~80%)
to mail.4. Web-push produces about 60% of responses
over the web.5. Token cash incentives with request significantly
improves web and overall response rates.
c Don A. Dillman, September 1, 2016 45
Summary of Major findings (2)
1. Significant differences exist between web and mail responders (education, age, income, marital status).
2. Web+mail treatment respondents are quite similar to all mail-only treatment respondents.
3. Item non-response to mail is a less of a problem than we had anticipated when visual design principles are applied.
4. Response decreases when surveying in distant states.5. Web-push costs slightly more per respondent than mail-only
because of lower response rates.
c Don A. Dillman, September 1, 2016 46
Web-push methods being used for many national surveys
• 2015 Japanese Census.• American Community Survey (3.5million
households per year) • Screen and collect data for U.S. child education
survey.• Screen and collect household data for U.S. child
health survey.• 2016 Australian Census • 2020 U.S. Census
c Don A. Dillman, September 1, 2016 47
Part III.
Additional challenges associated with the promises
and perils of web-push surveys
c Don A. Dillman, September 1, 2016 48
Promises and Perils of Web-Push
• In the long run, we need to push respondents to the web, but we need to do it with eyes wide-open.
• There are significant promises to be realized.• There are also some perils in going this
direction that we need to contemplate
c Don A. Dillman, September 1, 2016 49
12 ways Web-push exhibits promise
1 Address-based sampling provides superior coverage.
2 Response rates to web-push are dramatically higher than RDD telephone.
3 Cost reductions seem likely.4 Quicker responses from “most” people seems
likely.5 A better fit with society’s “do-it-myself” culture.
c Don A. Dillman, September 1, 2016 50
Web-push exhibits promise (2)
6 Reduction in socially desirable answers (vs. interviews)7 Reduction in extremeness of scalar responses.8 More effective branching (vs. mail)9 Initial mail contact increases feelings of trust.10 Pre-incentives improve likelihood of response.11 Two-phase data collection may be collapsed into one
phase (screen, plus longer questionnaire).12 Better fit with organizational preferences and
pressures.
c Don A. Dillman, September 1, 2016 51
Web-push perils calling for research and innovation
1 How to overcome “irritation” of not providing immediate choice.2 Pre-incentives needed for getting people to transfer across modes
(i.e. mail contact to web response).3 Decisions on communication access have switched from
organizational control to individual preference and behavior.4 Pocket/Purse devices are dominant (and perhaps only) interface
some people have with computers.5 Web response is not sufficient for most surveys—bias towards
higher education, income and younger households with kids.6 Mail follow-up to get balance faces constraints—branching plus
need for unified mode construction.
c Don A. Dillman, September 1, 2016 52
Additional web-push perils calling for attention (2)
7 Mixing of response modes are likely to create measurement differences, if allowed to ask questions in different ways.
8 Switching to mail in 3rd or 4th contact is not being handled well, e.g. still emphasizing web, minimal incentive and/or follow-up.
9 Not developing coordinated or mutually supportive communications focused on the process of responding
c Don A. Dillman, September 1, 2016 53
Additional web-push perils calling for attention (3)
10 Not treating mail and web as help-mates. Respondents can use mail to think through answers before entering them on web, but different construction makes that difficult.
11 Are we on a slippery slope because of do it yourself software? Let’s use only people contactable by emailLet’s only obtain web responsesIncrease sample size because of low responseTreating large sample sizes as if they have sampling error.Use of weighting to create data for underrepresented
categories.A catch, release, and catch-again mentality develops.
c Don A. Dillman, September 1, 2016 54
Some additional steps
• Introducing more effective mail follow-up to the initial web-push strategy, e.g. multiple contacts with additional incentives to get those communications read.
• Dealing with unified design between mail follow-up and smartphones that are requiring redesign of laptop/desktop surveys.
• Dealing with branching issues in visual design of paper surveys.
• Is there a role for telephone follow-up despite social desirability and aural vs. visual design effects?
• There is a lot more that can and needs to be done.
c Don A. Dillman, September 1, 2016 55
Selected references
1. Smyth, J.D., Dillman, D.A., Christian, L.M., & O’Neill, A. (2010). “Using the Internet to survey small towns and communities: Limitations and possibilities in the early 21st century.” American Behavioral Scientist 53: 1423-1448.
2. Dillman, D.A., Smyth, J.D., Christian, L.M. 2014. Internet, Phone, Mail and Mixed-Mode Surveys; The Tailored Design Method 4th edition. (John Wiley Co Hoboken, NJ)
3. Messer, Benjamin L. and Don A. Dillman. 2011. “Surveying the General Public Over the Internet Using Address-Based Sampling and Mail Contact Procedures.” Public Opinion Quarterly 75(3):429-57.
4. Millar, Morgan M. and Don A. Dillman. 2011. Improving Response to Web and Mixed-Mode Surveys. Public Opinion Quarterly 75 (2): 249-269
5. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): 734-750.
c Don A. Dillman, September 1, 2016 56
Selected references, page 2
6. Edwards, Michelle L., Don A. Dillman and Jolene D. Smyth. 2014. An Experimental Test of the Effects of Survey Sponsorship on Internet and Mail Survey Response. Public Opinion Quarterly. 78 (3): 734-750.
7. Messer, Benjamin L., Michelle L. Edwards, & Don A. Dillman. (2012). “Determinants of Web & Mail Item Nonresponse in Address-Based Samples of the General Public.” Survey Practice, April:. http://wwww.surveypractice.org
8. Messer, Benjamin L. 2012. “Pushing households to the web: Results from Web+Mail experiments using address based samples of the general public and mail contact procedures.” Ph.D. Dissertation. Washington State University, Pullman.
8. Edwards, Michelle L. 2013. “Measuring Public Perceptions of Water Governance in Nebraska and Washington.” Ph.D. Dissertation. Washington State University, Pullman.
c Don A. Dillman, September 1, 2016 57
Selected references page 3• Dillman, Don A. and Michelle L. Edwards. In Press. Chapter 17. Designing a Mixed-Mode
Survey. Sage Handbook of Survey Methodology. Sage Publications Wolf, Joye, Smith and Fu. Thousand Oaks. CA
• Dillman, Don A., Feng Hao, Morgan M. Millar. In Press. Chapter 15. Improving the
Effectiveness of Online Data Collection by Mixing Survey Modes. In Fielding, Nigel, Raymond M. Lee and Grant Blank. The Sage handbook of Online Research Methods, 2nd edition. Sage Publications, London.
• Harter, Rachel, Battaglia, Michael P., Buskirk, Trent D., Dillman, Don A., English, Ned, Mansour Fahimi, Frankel, Martin R., Kennel, Timothy, McMichael, Joseph, McPhee, Cameron Brook, Montaquila, Jill, Yancey, Tracie, Zukerberg, Andrew L. 2016. Address-base Sampling. American Association for Public Opinion Research Task Force Report 133 pp. http://www.aapor.org/AAPORKentico/AAPOR_Main/media/MainSiteFiles/AAPOR_Report_1_7_16_CLEAN-COPY-FINAL.pdf
c Don A. Dillman, September 1, 2016 58
Thank you!For additional information contact:
Dillman@wsu.edu133 Wilson-Short Hall
Washington State UniversityPullman, WA 99164-4014
dillman@wsu.edu509-335-4150