Survey Completion Rates and Resource Use at Each Step of a Dillman-Style Multi-Modal Survey

1 Abt Associates Inc. AAPOR 2003

Survey Completion Rates and Resource Use at Each Step

of a Dillman-Style Multi-Modal Survey

Andreal Hassol, Holly Harrison, Ricki Jarmon, Brenda Rodriguez, Austin Frakt

Abt Associates Inc.


Introduction• Designing DC efforts becoming an increasingly complex task.

• Declining participation in surveys• Projects must be competitive in costs• Rely on multi-mode DC efforts to achieve high response rates

• Designing DC efforts we must weigh resources against expected

returns.

• Seek mode(s) of DC yielding high quality data and high response rates.

• Minimizing nonresponse bias (increase generalizability of data)• Delivered on schedule, on budget

• “Total Design Method” widely known / cited in the industry (Dillman, 1978).


Background: Elements of the Total Design Method (TDM)

• Respondent friendly questionnaire

• Topic of survey salient to survey recipient

• Up to 5 contacts with survey recipients (via mail, poss. with phone follow-up)

• Inclusion of a stamped return envelope• Personalized Correspondence

• Token Financial incentives


Utilizing the TDM in this Project

• Lit. review - sought out resources that demonstrated results of TSD implementation and related costs and returns.

• Minimal work tangibly demonstrating effects of reminder post cards or additional mailings over time.

• This research seeks to fill that gap.


Project Overview

Client: Centers for Medicare and Medicaid Services (CMS)

Response Rate Goal: 70%

Mode of DC: Mail Survey with Phone Follow-up.

Field Period: 21 weeks. (Mid-July to December 2002).

Sample: 7,730 Community-dwelling Medicare Beneficiaries

Topic: Evaluation of beneficiaries’ experiences with a new Medicare Insurance Option.

Design: 6 contacts by mail with phone follow-up.

No $ incentive.


Methodology

• 7,730 cases from Medicare Enrollment database, across 25 states.

• Included three subgroups:• Those enrolled in new Private Fee for Service (PFFS) plan

• Those remaining in traditional Medicare managed care plans.

• Those with Medigap supplemental policy

• Proxies allowed (spouse / partner / adult child)

• Toll-free number staffed by interviewers where recipients could dial to do survey or ask questions

• Conducted English and Spanish

• 13 page SAQ / CAPI


Timeline of Project

Advance Letter

Survey Packet I – First Class Mail

Reminder Postcard I

Survey Packet II – First Class Mail

Postcard II

Survey III – Priority Mail

Phone follow-up (47% of non responders matched)

Continued receipt of mail quex

Week 1

Week 1

Week 5

Week 6

Week 9

Week 13

Weeks 14-21

Weeks 14-21


Sample Management

• Integral to success of multi-modal project.

• Database designed to fulfill receipting and reporting needs for mail and CATI completes.• Downloads of CATI efforts each night

• Receipt of mail completes

• Enable us to track progress carefully by mode, sample type, site

• Minimized burden on sampled beneficiaries• Avoiding re-mails to already complete or identified ineligible

• Avoid phone follow-up to those submitted by mail

• Minimized number of duplicate completes (mail and phone)


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Week

Completion Rate through the Field Period

Quex mailing Weeks 1, 6, 13

Postcards Weeks 5, 9

CATI 14-21


Completes by Week

• 87% of completes by mail

• 13% of completes by phone

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

5500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Week of DC

Co

mp

lete

d C

as

es


Costs Incurred Through the Field Period

DC Costs

(87% of budget):

23% on phone

64% on Mail

Percent of Total Budget Spent

5.7%

19.6%

43.4%55.9%

71.0%

86.9%

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

1 2 3 4 5 6

Month of DC

Remaining 13% budget for DP


Results on Salience of Survey Topic

2675

1712

898

5285

3406

2499

1331

7236

0

1000

2000

3000

4000

5000

6000

7000

8000

PFFS Medicare (with orwithout Medigap

Supplement)

Medicare managed CarePlan

TOTAL

Num

ber

of In

divi

dual

s

Total CompleteTotal Eligible

79% 69% 67%

73%


Conclusions • TDM worked, minor adjustments to tailor to our population and study.

• Each remail effort saw spike in returns, peaking within 2 weeks of mail out

• Less noticeable returns following postcards

• Telephone efforts were 13% of completed surveys but 23% of costs.

• Salience of instrument proves true – as highest RR in PFFS group (79%)

• Respondent friendly design in quex –skip pattern confusion yielded extra data

• Database essential component to success• Minimized burden

• Maximize labor resources in phone follow-up effort to nonresponders only.

• Over 7 week dual effort – 21 cases completed phone and mail.

• Attained a 73% response rate (goal 70%), using AAPOR calc. standards.

Abt Associates Inc.

Survey Completion Rates and Resource Use at Each Step of a Dillman-Style Multi-Modal Survey

Documents

Transcript of Survey Completion Rates and Resource Use at Each Step of a Dillman-Style Multi-Modal Survey