Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference...
Transcript of Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference...
![Page 1: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/1.jpg)
Effects of Incentive Amount
and Type on Web Survey
Response Rates
Presentation at the 69th
Annual AAPOR Conference
Anaheim, CA
Jared Coopersmith • Lisa Klein Vogel • Tim Bruursema
Kathleen Feeney
May 2014
![Page 2: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/2.jpg)
2
Presentation Overview
• Introduction
– Challenge
– Using incentives to increase response rates
– Hypotheses
– Related work
• Methods
– Survey and sample description
– Experimental conditions
• Results
• Summary
• Discussion
![Page 3: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/3.jpg)
3
Introduction
![Page 4: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/4.jpg)
4
Challenge
• The National College Ready Survey (NCRS) is a web-administered survey
of school principals
– Sponsored by the Bill & Melinda Gates Foundation
– Four waves of data collection
• Principals face many competing demands on
– Personal time and availability
– School participation in research
• These factors can affect principals’ willingness to respond to survey requests
![Page 5: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/5.jpg)
5
Using Incentives to Increase Response Rates
• The NCRS offers significant ($50) post-paid incentives for completion
– Increase response
– Minimize costs
• Wave 2 included an incentive experiment to inform future data collection
• Incentives can help increase response rates and sample representativeness,
but depends on a number of factors
• Web administration poses unique administrative challenges
– Cash is more effective than gifts
– Pre-paid incentives are most effective, but difficult to administer via web
![Page 6: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/6.jpg)
6
Hypotheses
• H1: Offering a differential incentive for completion within the first three weeks
of the field period will yield higher response rates both early on and overall
• H2: Providing a pre-paid incentive before data collection yield higher response
rates due to the “norm of reciprocity”
• H3: Providing a pre-paid incentive to nonresponders midway through the field
period will yield higher response rates
![Page 7: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/7.jpg)
7
Related Work
• Pre-paid incentives
– Often more effective than post-paid or “promised” incentives
(Singer and Ye 2013; Göritz 2010)
• Early response or “early bird” incentives
– Can be more effective than even pre-paid incentives (LeClere et al. 2012)
• Refusal conversion incentives
– Some studies point to only using refusal conversion payments (Singer and Ye
2013)
– But is this ethically problematic or unfair? (Presser 2008)
![Page 8: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/8.jpg)
8
Incentive Experiment
• Based on these hypotheses, we implemented an experiment to compare the relative
effectiveness of
– $50 post-completion incentive + additional $50 for completion in first three weeks
– $50 post-completion incentive + $25 pre-paid incentive
• Control condition
– $50 post-completion incentive only
• Subgroup eligible for $25 refusal conversion incentive
Group 1 Group 2 Group 3 Control
Standard
incentive
$50 $50 $50 $50
Additional
incentive
$50 (early
response)
$25 (pre-pay) $25 (refusal
conversion)
None
Total possible
incentive
$100 $75 $75 $50
![Page 9: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/9.jpg)
9
Methods
![Page 10: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/10.jpg)
10
Data Collection Activities
• Mode: Web-only
– 39 questions, 15 minutes
• Data collection consisted of
– Invitation email and postal mailing
– Weekly email reminders
– Monthly reminder calls to non-completers
• Incentive administration: Amazon.com gift card code
– Pre-payment sent via email and postal letter
• Included Amazon.com $25 gift card code and web address for Amazon.com
– Post-payment sent via email
• Included gift card code, payment amount, link to Amazon.com, and redemption instructions
– Post-payments sent approximately every 8–10 days after survey completion
![Page 11: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/11.jpg)
11
Sample Description (1)
• Size: n = 2,034
• Composition: Elementary, middle, and high school principals
• Selection method: Stratified probability proportional to size (PPS)
• Field period: 10/30/13–3/31/14 (21 weeks)
• Sample was fielded in four different groups during the data collection period
– Group 1: n = 1,062; in the field for 21 weeks
– Group 2: n = 259; in the field for 17 weeks
– Group 3: n = 402 ; in the field for 12 weeks
– Group 4: n = 311; in the field for 5.5 weeks
![Page 12: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/12.jpg)
12
Sample Description (2)
• $50 post-pay incentive, plus
$50 (early
response)
$25
(pre-pay)
$25 (refusal
conversion)
None
(control)
Total 560 560 280 280
Sample
released523 524 262 258
![Page 13: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/13.jpg)
13
Results
![Page 14: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/14.jpg)
14
Results: $100 Early Response Incentive
**p < 0.05
29.7**
55.4
20.1
56.6
0
10
20
30
40
50
60
70
80
90
100
Early Response Cutoff Final Response
EarlyResponseIncentive
Post-PayOnly
![Page 15: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/15.jpg)
15
Results: Early Response (cont.)
0
10
20
30
40
50
60
70
80
90
100
EarlyResponseIncentive
Post-PayOnly
![Page 16: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/16.jpg)
16
Results: $25 Pre-Paid Incentive
**p < 0.05
54.6 56.6
0
10
20
30
40
50
60
70
80
90
100
Final Response
Pre-Pay Incentive
Post-Pay Only
![Page 17: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/17.jpg)
17
Results: $25 Refusal Conversion (Overall)
**p < 0.05
56.5 56.7
0
10
20
30
40
50
60
70
80
90
100
Final Response
RefusalConversionIncentive
Post-Pay Only
![Page 18: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/18.jpg)
18
Results: $25 Refusal Conversion (Targeted)
*p < 0.10
24.1*
12.9
0
10
20
30
40
50
60
70
80
90
100
Final Response
Refusal ConversionIncentive
Post-Pay Only
![Page 19: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/19.jpg)
19
Results: Time in Field
***p < 0.01, **p < 0.05
Average
days-to-complete
Incentive
Treatment
group
Control
group
Early response
incentive41.8** 48.8
Pre-paid incentive 46.1 48.8
Nonresponse
conversion incentive55.3*** 42.4
![Page 20: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/20.jpg)
20
Summary
![Page 21: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/21.jpg)
21
Overall Impact of Incentives
• Early response incentive
– Higher response rate within the incentive period
• Significant effects overall and among most subgroups
– No effect on final response rates
• Effect diminished immediately following the incentive period
• No difference in response rates by the second half of data collection
– Reduced average time in field by 1 week
• Pre-paid incentive
– No significant effect on response rates or length of time to complete
• Post-paid incentive significantly more effective in two subgroups
• Refusal conversion incentive
– No significant effect on response rates overall
– Among those eligible for incentive, did show some evidence of effect
– Increased average time to complete
![Page 22: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/22.jpg)
22
Discussion
![Page 23: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/23.jpg)
23
Areas for Further Investigation
• Cash vs. electronic incentive administration: Effects of incentive delivery
• Incentive amounts: Is bigger always better?
• Timing of early response period
• Effects on data quality
![Page 24: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/24.jpg)
24
Citations
• Göritz, Anja S. “Using Lotteries, Loyalty Points, and Other Incentives to Increase
Participant Response and Completion.” In Advanced Methods for Conducting Online Behavioral Research, edited by Samuel D. Gosling and John A. Johnson.
Washington, DC: American Psychological Association, 2010.
• Gouldner, Alvin W. “The Norm of Reciprocity: A Preliminary Statement.”
American Sociological Review, vol. 25, no. 2, 1960, pp. 161–78.
• LeClere, Felicia, Sheldonn Plummer, Jennifer Vanicek, Ashley Amaya and Kari
Carris. “Household Early Bird Incentives: Leveraging Family Influence to
Improve Household Response Rates.” Presented at the Joint Statistical
Meetings, San Diego, CA, 2012.
• Singer, Eleanor, and Cong Ye. “The Use and Effects of Incentives in Surveys.”
The Annals of the American Academy of Political and Social Science, vol. 645,
2013, p. 112–141.
![Page 25: Effects of Incentive Amount and Type on Web Survey ...dc-aapor.org/2014 conference slides/Coopersmith.pdf4 Challenge •The National College Ready Survey (NCRS) is a web-administered](https://reader033.fdocuments.us/reader033/viewer/2022060922/60ae2b9da6715062146f1e24/html5/thumbnails/25.jpg)
25
For More Information
• Jared Coopersmith
• Lisa Klein Vogel
• Tim Bruursema
• Kathleen Feeney