Innovative Retention Methods in Panel Research · 2015. 1. 28. · much more useful and more...
30
icfi.com | May 17, 2013 Innovative Retention Methods in Panel Research: Can Smartphones Improve Long-term Panel Participation? James Dayton Andrew Dyer
Transcript of Innovative Retention Methods in Panel Research · 2015. 1. 28. · much more useful and more...
icfi.com |
May 17, 2013
Innovative RetentionMethods in Panel Research:Can Smartphones Improve Long-term Panel Participation?
James DaytonAndrew Dyer
2icfi.com |
Panels and Panel Management for LongitudinalResearch Panel Recruitment Panel Attrition
− Threats to data quality
Panel Retention Tactics− Traditional
− Alternative
Leverage Technology to Support Panelist Retention− Can tech innovation increase retention or is it necessary to
maintain former levels?
Agenda
3icfi.com |
ICF Experiment− Our panel
− Treatment groups
− Data collection
− Analysis and data usage
− Next steps
Agenda - continued
4icfi.com |
Panel studies - where the same individuals or householdsare interviewed at multiple points in time
− Provides additional depth for low incidence studies, trackbehavior and other attributes over time
Selection− Recruit from a probability-based statistically valid sampling method
• Address Based Sampling (ABS)
• Dual Frame RDD
Consent and purpose (engagement/buy-in) Contact information for follow-up and tracking
− Mail,
− Phone(s), Email,
− Relatives/friends/neighbor contact information
Panel Creation Best Practices
5icfi.com |
Establish expectations− Respondent burden (type, length of interview)
− Duration of panel commitment (if known)
− Importance of continued participation and engagement to validityand reliability of study results
− Remuneration
− Other non-financial benefits to respondents – passion aboutsubject, altruism
− Frequency of contacts
− How sponsor will maintain regular contact with participants– request notification of changes in contact information
– Improve/increase engagement - provide non-biasing updates and otherinformation about study
Panel Creation Best Practices - continued
6icfi.com |
Why it’s bad and why retention is important− can bias sample
− can reduce statistical power
− can be costly to replace panelists
Industry rates of panel attrition− Literature search revealed published annual continued
participation rates as high as 99% to as low as 50%1
Panel Attrition
1Una Lee, Center for Social Science Research, Panel Attrition in Survey Data, September 2003
7icfi.com |
Factors in attrition that researchers can control(somewhat)
− Losing contact with panelists
− Refusal to continue
Will traditional retention methods and tactics be effectivein the future?
Panel Attrition - continued
8icfi.com |
Traditional methods− Managing contact information
− Good rapport
− Participant-oriented actions2
• Calls, visits, mailings
New/alternative methods− Electronic information searches
• Locate “lost” panelists
− Smartphone app to help manage contact information andrelationship
Attrition Mitigation Tactics
2http://www.wpic.pitt.edu/research/famhist/PDF_Articles/Springer/RI%206.pdf
9icfi.com |
Purpose:
− Update contact info
− Alerts on next survey opportunity
− Receive/track gift codes
− Respond to survey questions
− Other features?
Typical app development costs: $1,500 to over $50,000k.− Additional costs for each additional platform
− Minimal app for panel use $10,000
Requires large and ongoing panel to be cost effective
Smartphone Panel App
10icfi.com |
Extension to traditional tactics (mail, telephone, email)
Panel of recreational boat owners− Q: Would a smartphone app support panel retention?
Which panelists own a smartphone?
Smartphone App
11icfi.com |
Recreational boat owners recruited from a dual-frame, dual-modestudy
− National RDD, CATI
− State lists of registered boat owners, Mail
Invited to join panel and report recreational boating activitiesperiodically throughout year
Provide contact details
Offered $10 Amazon.com gift code for each interview
Approximately 24,000 people—all states, D.C. and Puerto Rico
Range of boat types
Who are these Panelists?
12icfi.com |
13icfi.com |
Sample SelectionPhase 1—quantitative
SMS
Selected SampleN = 2,412
Selected SampleN = 2,412
14icfi.com |
SMS, 804
EMAIL, 804
MAIL, 804
Sample by Experimental Mode of ContactPhase 1—quantitative
N = 2,412
15icfi.com |
We asked panelists…− Do you own a smartphone?
− Would you download a smartphone panel app?
− How do you prefer to be contacted in the future?
Phase 1—quantitative
16icfi.com |
Mini-survey identified effect of contact mode on…− Response rate
− Willingness to download smartphone panel app
− Preferred mode of future contact
Phase 1—quantitative
17icfi.com |
Respondents by StatePhase 1—quantitative
N = 586N = 561
18icfi.com |
Response RatePhase 1—quantitative analysis
0%
20%
40%
60%
80%
100%
OVERALL SMS EMAIL MAIL
23.3% 12.7% 36.6%
χ2(2) = 133.50, p < .001
ResponseNon-Response
n = 2,412 n = 804 n = 804 n = 804
20.5%
Experimental Mode of Contact
19icfi.com |
Smartphone OwnershipPhase 1—quantitative analysis
0%
20%
40%
60%
80%
100%
OVERALL SMS EMAIL MAIL
69.1% 91.2% 66.0%
χ2(2) = 29.46, p < .001
Own SPDon’t Own SP
n = 561 n = 102 n = 294 n = 165
61.2%
Experimental Mode of Contact
20icfi.com |
Mode of Web Survey Access(following SMS or EMAIL contact)
Phase 1—quantitative analysis
0%
20%
40%
60%
80%
100%
SMS EMAIL
30.5% 76.0%
PCTabletSmartphone
χ2(2) = 99.76, p < .001
67.4% 14.6%
n = 95 n = 288
21icfi.com |
Willing to Download Panel App(among SP Owners)
Phase 1—quantitative analysis
0%
20%
40%
60%
80%
100%
OVERALL SMS EMAIL MAIL
56.0% 62.6% 47.4%
χ2(2) = 11.80, p = .003
YesNo
n = 386 n = 91 n = 194 n = 101
66.3%
Experimental Mode of Contact
22icfi.com |
0%
20%
40%
60%
80%
100%
OVERALL SMS EMAIL MAIL
MailPhoneEmailSMS
Preferred Mode of Future Panel ContactPhase 1—quantitative analysis
Experimental Mode of Contact
χ2(6) = 47.83, p < .001n = 546 n = 94 n = 288 n = 164
23icfi.com |
Provided Current Contact Information(Address, Phone, or Email)
Phase 1—quantitative analysis
0%
20%
40%
60%
80%
100%
OVERALL PC TABLET SP
84.3% 83.9% 96.6%
χ2(2) = 3.73, p = .155
YesNo
n = 383 n = 248 n = 29 n = 106
82.1%
Mode of Access to Web Survey
24icfi.com |
Phase 2—qualitative
Methods
45 minute interviews with smartphone-owning panelists− IDIs
− Mini focus groups
Objectives
Explore smartphone app usage
Brainstorm features panelists seek in recreational boating app
Discuss whether app would keep panelists engaged
25icfi.com |
Ideas for functionality
Facilitate survey-taking experience- Send alerts/notifications via app rather than via email
- Log boating trips in real-time
- GPS to track trips and aid trip recall
- Answer survey questions via the app
- “Dashboard” to track surveys and incentives
Local boating information− Water conditions, gas pump locations, lunar/fish calendar
Phase 2—qualitative
26icfi.com |
There is interest!
Other examples out there?
Hook or draw to download and use app
App must provide value to panelists in some way- Alerts/notifications
- Distribute incentives
- Facilitate the survey-taking experience
Ease of use is critical
Phase 2—qualitative
27icfi.com |
Panelist feedback
“I shouldn’t have to remember that I made 5 trips. If the app couldcompile that information using GPS, that would be interesting.”
“If you could do it standing at the lake even, that just makes it thatmuch more useful and more usable.”
“I would be more inclined to do it right then and there if it wasprovided to me, versus waiting to go home and do it online, because Iwould forget.”
“It would be really great to take the survey through the app, and havenotification bubbles when it’s time to take the survey. I would be likelyto do it because the little red dot is irritating!”
Phase 2—qualitative
28icfi.com |
Phase 3 Question− Does intention to download an app predict actual behavior? (correlational)
Phase 3
Next StepsPhase 1
SmartphoneOwners
“App availablefor DL!”
DL StartsDL Completes
Notification DL Behavior
…Phase 4
Panelistswith App
29icfi.com |
Phase 4 Questions− Does mode of retention communications influence attrition? (experimental)
− Does mode of retention communications influence survey response rates? (experimental)
Sam
ple
(App
+ Em
ail +
Mai
l)Smartphone
AppAttrition
Response %
Email AttritionResponse %
Mail AttritionResponse %
Phase 4
Mode of RetentionCommunications
Retention
ICF Experiment - continued
