RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

Computer Audio Recording: A Practical Technology for Managing Survey Quality

M. Rita Thissen, Hyunjoo Park, Mai Nguyen

International Total Survey Error WorkshopQuébec City, QC, June 2011

RTI International

Overview Survey process steps Technology and methodology of CARI* Incorporating CARI into the survey process Addressing structural and functional errors CARI adoption by the US Census Bureau

– Overall implementation plan– American Community Survey Content Test use

of CARI for behavior coding User feedback and future possibilities

*CARI: Computer Audio-Recorded Interviewing

RTI International

Survey Process Steps

1. Define research objectives2. *Choose methods of data collection3. *Detailed planning, including quality assurance plan4. *Construct and pretest questionnaire5. *Collect data6. *Analyze and report

* All except the first step may be affected by use of audio recording methods

RTI International

CARI Technology

Collect audio recordings during actual interview Capture screen image and response data for

comparison with recorded audio Monitor recordings centrally after the interviewing

session ends Quantify content through coding Review coding results and choose what action to

take, if any

RTI International

CARI System Concepts

Support for total error management

Multimode Multi-purpose More efficient

and less burdensome than real-time monitoring

RTI International

Survey Step 2: Choose Data Collection Methods

ISSUE Concern about mode effects

between telephone (CATI) and in-person (CAPI) interviews

Determine what data points might be sensitive to mode effects

Difficulty of data capture in semi-structured interviews or free-response items

Standardized coding, such as occupation and industry, or drug and disease codes

APPROACH Audio recordings for

comparison across “electronic” modes

Plan to record those questions and responses, plus “buffering” by recording adjacent items

Capture open-ended responses through audio recording for later coding

Allow standardized coding by centrally-located experts instead of by interviewers

RTI International

Survey Step 3: Quality Assurance (QA) Plan

ISSUE QA plan is for detection, control and remediation of error

Data entry challenges: “Other, specify” questions, long lists of response options, “Code all that apply”, or free-form responses

Effort and expense of live-monitoring, field observation and verification

APPROACH Build CARI capabilities into QA processes for data quality and

performance management

Can use audio recordings for primary data entry or as a form of double data entry to confirm/correct data from CATI or CAPI

May be able to reduce costs through use of recordings for performance review

RTI International

Survey Step 4: Construct and Pretest the Questionnaire

ISSUE Question-level problems: Wording, ambiguity, lack of clarity

Translation effects: Dialects, regionalisms or country-specific vocabulary may not be recognized until production

Key data points need extra attention

APPROACH Use recordings and behavior coding during pretest and production

to identify issues

Review of production responses by bilingual staff can confirm correctness or identify problems

Select questions to record, including buffer zones as needed.

RTI International

Survey Step 5: Collect Data

ISSUE Managing interviewer performance & adherence to protocol

Motivating interviewers when the going gets rough, especially late in a cycle

Responsive design or adaptive total design; when subpopulations don’t respond

APPROACH Frequent review of a sample of cases helps “keep an eye” on

interviewing

Positive feedback can reinforce good behavior and provide encouragement

Review of recordings from targeted subpopulations may give insight into reluctance

RTI International

Survey Step 6: Analyze and Report

ISSUE Quantifying operational data for methodology reports

Explaining difficulties or unusual situations encountered

Estimating survey error

APPROACH CARI coding contributes to paradata metrics

Audio recordings may provide insight into or detailed examples of the challenges

Coded monitoring results offer a way to standardize measurement of ‘subjective’ error, such as consistency of questionnaire delivery

RTI International

Addressing Structural Errors

Translation errors Regional effects, such as vocabulary differences

(Do you drink soda, pop or soft drinks? Would you drink from a water fountain?)

Logic errors: complex questionnaire logic may hide errors, in question sequences rarely followed

Values computed based on earlier responses and then used as fills; errors may be hard to detect during testing

RTI International

Addressing Functional Errors

Human error: Data entry mistakes, presentation or mode effects, respondent reactions

Advance warning of mode effects, indications prior to analysis

Additional information about item-level non-response, including both “don’t know” answers and refusals

RTI International

CARI Adoption by the US Census Bureau

CARI Interactive Data Access System Multiple-year project, 2009 - 2012

Stage 1: Develop behavior coding and quality assurance (QA) interfaces (completed)Stage 2: Field test for behavior coding, recommendations for revision (completed)

– Stage 3: Enhancements and revisions for BC and QA and addition of a Coaching component (in progress)

– Stage 4: Field test for QA and Coaching (future)

RTI International

Behavior Coding Field Test

American Community Survey (ACS) Content Test Data collection, 2010

– Mail: August – September– CATI: October– CAPI: November

Behavior coding immediately after data collection– Dec 2010 – March 2011

Data analysis in progress at the Census Bureau (Contact: Joanne Pascale)

RTI International

ACS Content Test

CATI CAPISample size 23,673 15,202Completed + partial cases 4,523 6,384Recording consent rate 88.5% 64.8%Interviews recorded 4,005 4,137Interviews selected for coding 807 708Training cases 32 0Cases coded out as inaudible 48 7Interviews coded 727 701Recordings coded 27,163 22,676

RTI International

Coding Page• Compact, feature-rich

interface• Playback for audio• Option to view image• Navigation bar at left

(blue) for non-sequential access to recordings

• Next / Prev buttons for sequential access

• Dynamic code lists are structured into categories, groups and codes

• Grid display summarizes codes assigned

• Room for notes

RTI International

General FeedbackFeatures that the CARI System users found most

valuable included the following:

Unobtrusive nature of CARI Ability to select cases (or questions) for review

from among those recorded Flexibility in defining what behaviors to code Image display offering the exact display of wording

(including fills) as well as the actual data entry value

Real-time monitoring of coding Data available for extraction at any time

RTI International

Future Possibilities for Error Management

Greater availability of recordings as a routine part of data collection

Expanded use of recordings in performance management and training or retraining of interviewers

Expanded use of CARI paradata in survey management

Development of low-burden performance metrics for interviewers and coders

Creation of an “error profile” for a survey And… whatever else you can imagine!

RTI International

Acknowledgements

This project would not have been possible without the support of many people at the Census Bureau

and RTI.

We especially thank Sherry Thorpe, CARI Project Manager, Joanne Pascale, and the ACS Office of

the Census Bureau.

For more information, contact:Rita Thissen – rthissen@rti.orgHyunjoo Park – mpark@rti.org Mai Nguyen – mnguyen@rti.org