Diane K. Willimack U.S. Census Bureau Any views expressed on
statistical, methodological, or technical issues are those of the
authors and not necessarily those of the U.S. Census Bureau. The
Response Process and Burden in Business Surveys: Implications for
Official Statistics Business Survey
Slide 2
Outline Business vs. household surveys Business context and
Survey response process Response burden Consequences for error
sources Focus on measurement error Implications for Official
Statistics 2
Slide 3
Businesses vs. Households What are businesses? Economic units
Made up of one or more establishments Locations where business is
conducted or industrial operations performed (FCSM, 1988) Goals To
produce goods and services To remain open and viable over tim To
remain open and viable over time 3
Slide 4
Businesses vs. Households How are business surveys different?
Population Data Survey design Response process 4
Slide 5
Household Surveys vs. Business Surveys Household Surveys
Business Surveys Qs are aboutSelf Business or Organization Types of
data Autobiographical, Attitudes, Behaviors Information abt the
organization and/or its employees or clients; Financial; Quantities
Data sourcesMemory Records & Rs knowledge of them Respondent
Identity Self or ProxyInformant 5
Slide 6
Business Survey Topics Business characteristics Production
measured by volume or value Number of employees, their
qualifications, field of work, hours worked, payroll and benefits
Expenditures and investments Volume sold and financial results
(e.g., figures from balance sheets and income statements)
Consumption of energy and other resources Transactions between
businesses Business trend evaluations Evaluations of working
conditions 6
Slide 7
Household Surveys Business Surveys R selection; R rule Single R
or Selected proxy Multiple informants Mode Interviewer administered
Self-administered Authority Voluntary or Mandatory Response Process
Cognitive Cognitive and Organizational Household Surveys vs.
Business Surveys continued 7
Slide 8
8 Complex Response Process in Business Surveys 1)Record
formation & encoding 2)Organizing response tasks a)Respondent
selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data Sudman et al. (2000) / Willimack & Nichols (2010);
Bavdaz (2007, 2010) Cognitive Response Process (Tourangeau, 1984)
Organizational Steps
Slide 9
What is Response Burden? Traditional Approach: Burden is the
time taken to complete the survey Actual burden = Costs Perceived
Response Burden: Respondents perceptions of their survey experience
Respondents rarely equate burden with the time it takes to respond
to a survey; instead they perceive burden in relation to mode, who
is conducting the survey, and whether the produced statistics are
useful to businesses/society. (Hedlin et al. 2005) 9 R = Number of
people involved in completing the questionnaire T = Time taken by
people involved C = Salary costs of these people
Slide 10
Causes of Perceived Response Burden (PRB) 10
Slide 11
Respondent(s) Authority Capacity Motivation 11 (Dale et al.,
2007) Origins and Flows of Response Burden in Business Surveys
Slide 12
Respondent(s) Authority Capacity Motivation Origins and Flows
of Response Burden in Business Surveys 12 1 2 3 Management
considerations 4 Burden starts here with stakeholders and data
users needs.
Slide 13
13 Official Statistics and Stakeholders Needs 1. Theoretical
world: Conceptualization Research Needs: Questions, Issues,
Problems, Decisions Empirical world: Physical Observations
Attributes Data Concepts Measurements Question(s) Questionnaire
Response
Slide 14
14 1. Theoretical world: Conceptualization Research Needs:
Questions, Issues, Problems, Decisions Empirical world: Physical
Observations Attributes Data Concepts Measurements Question(s)
Questionnaire Response Specification errors Operationalization
errors Measurement & Non-response errors Processing errors
Official Statistics in the Context of Total Survey Error Through
survey development and design, burden transfers from stakeholders
& data users to the survey organization.
Slide 15
1 2 15 Origins and Flows of Response Burden in Business Surveys
Respondent(s) Authority Capacity Motivation Through survey
development and design, burden transfers from stakeholders &
data users to the survey organization. So far, nothing particularly
unique to business surveys, right? . but were just getting to the
good part .. ..the business survey response process .. ..the
business survey response process .. 3 Management considerations
4
Slide 16
16 Response Process in Business Surveys 1)Record formation
& encoding 2)Organizing response tasks a)Respondent selection
b)Scheduling c)Prioritizing & motivation 3)Comprehension
4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman
et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007,
2010)
18 Business Reasons for Records Sudman et al. (2000) /
Willimack & Nichols (2010) External requirements: Legal
purposes Regulatory purposes Internal business Manage resources To
remain open and viable To produce goods and services Recorded data
requested data Data are owned by the business
Performing Response Tasks Respondent selection, scheduling and
prioritizing the Work to provide a survey response 20 Benefits
Costs Surveys represent a cost without associated benefit
non-productive cost
Slide 21
21 The most knowledgeable respondent The person who Knows the
records and their location The meaning and purpose of the data in
the records Determined in the context of record formation Response
Process: Respondent Selection Who is the Best Respondent? Edwards
& Cantor (1991)
Slide 22
22 Division of labor Compartmentalize the work for efficiency
Organizational structures reflect the distribution of the work
Distributed knowledge Data are located where they are needed and
used Response Process: Respondent Selection Who is the Most
Knowledgeable Respondent? Benefits Costs
Slide 23
23
Slide 24
24 Technical Core Those whose job is the production of goods
& services And the associated data Boundary Spanners Interface
with the outside world and across internal units Response Process:
Respondent Selection Who is the Best Respondent? Tomaskovic-Devey,
et al. (1994)
Slide 25
25 Boundary Spanners protect the Technical Core
Slide 26
26 Response Process Who is the Best Respondent? Willimack &
Nichols (2010); Bavdaz (2007, 2010) Benefits Costs Routine Boundary
spanners Respondent selection is under the control of the business
Response Process Who is the Best Respondent? Willimack &
Nichols (2010); Bavdaz (2007, 2010)
Slide 27
27 Response Process: Respondent Selection Who is the Best
Respondent? Willimack & Nichols (2010); Tuttle (2009); Bavdaz
(2007, 2010) Distributed data requires multiple respondents and
data providers Respondent manages the collection of data from
others Gathers and compiles data Respondent = Survey
Coordinator
Slide 28
28 SURVEY COORDINATOR obtains data from others Emails/faxes
actual survey question(s) including instructions Phone/email; does
not include actual survey questions & instructions SURVEY
COORDINATOR completes the questionnaire Respondent Data Provider
Understands contents of records Expectations of the Survey
Coordinator Lorenc (2006, 2007), Willimack (2007), Bavdaz (2007),
Tuttle & Willimack (2012) Knows who to contact!
Slide 29
29 Response Process: Personal Cognitive Steps 1)Record
formation & encoding 2)Organizing response tasks a)Respondent
selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data
Slide 30
Retrieval Data sources Personal knowledge from memory Records
Knowledge of records Retrieval incorporates Cognitive step
retrieval of knowledge of data sources Access to records Physical
act of retrieving data Extracting information from files Consulting
multiple sources Compiling information 30
Slide 31
Judgment Judging the adequacy of the response Does the
Respondents answer retrieved from records or memory match the data
being requested? The Respondent doesnt have the requested data in
records. How does the Respondent estimate that figure? The
Respondent thinks, My answer is good enough. 31 Business
Survey
Slide 32
32 Retrieval + Judgment: Requested Data Recorded Data Bavdaz
(2010)
Slide 33
33 Mismatch Problems Adapted from Bavdaz (2010)
Slide 34
34 Mismatch Problems Adapted from Bavdaz (2010)
Slide 35
Business Survey Response Process Summary Management
Considerations Survey response tasks = COST with no associated
production Respondent selection Scheduling, prioritizing &
motivation Data release process Respondent Requirements Taking the
role as survey coordinator Knowledge of available information
Handling differences between recorded data and requested data
35
Slide 36
36 1 Respondent(s) Authority Capacity Motivation Evidence of
Response Burden in Business Surveys Management considerations 4 2
3
Slide 37
37 Perceived Response Burden 9,6 29,3 39,9 15,5 5,8 Very easy
Quite easy Neither easy nor difficult Quite difficult Very
difficult Sources of PRB E-technical E-Usability Other Layout No of
questions Wait for information Terms Calculations Need of help
Mismatch Response categories (2) Survey Design Perceived Response
Burden (PRB) Source: Norwegian Structural Statistics, 2010 n =
16,572
Slide 38
(3) Time (Actual) Cost of Response Burden 38 Source: Statistics
Canada, Survey of Regulatory Compliance Costs, 2005 & 2008
Average reporting costs, per business, increase was the size of the
business increases (based on employment). Average reporting costs,
per employee, decrease as the size of the business increases (based
on employment).
Slide 39
39 (3) Conditions Associated with the Business Perceived
Response Burden (PRB) PRB- index Size of0 9 employees0,11
establishment10 19 employees0,13 20 49 employees0,06 50 or more
employees0,10 DocumentationNone0,15 problems1-0,14 2-0,23 3-0,45 No
of10,15 Respondents2 or more-0,01 Value of PRB-Index PRB
Slide 40
40 (4) Respondent Characteristics Perceived Response Burden
PRB- index Available timeVery easy to find0,68 Quite easy to
find0,41 Neither easy nor difficult0,06 Quite difficult-0,09 Very
difficult-0,37 Perception ofVery useful0,22 usefulnessQuite
useful0,30 Neither useful nor useless0,18 Quite useless0,09 Very
useless-0,11 ExperienceNo0,04 Yes0,11 Value of PRB-Index PRB
Slide 41
41 How Does Perceived Response Burden Affect Response
Quality?
Slide 42
Mismatches & Measurement Error 42
Slide 43
Respondent(s) Authority Capacity Motivation Example: Official
Statistics on Research & Development 43 R&D EXPENSES
R&D PERFORMED Management considerations
Slide 44
R&D: Requested vs Recorded Data 44 Some Attributes of
R&D Stakeholder : PERFORMED R&D PERFORMED by This Company
Accountant: EXPENSES R&D EXPENSES TypeBasic, Applied,
DevelopmentHuh? Tech Support & Maintenance No Yes, if done by
same employees doing R&D GeographyYesHuh? Outsourced No (some
other company did the R&D for this company) Yes (my company
paid for this R&D for the benefit of my company) Reimbursed Yes
(this company did the R&D, even though someone else paid for it
No (someone else paid my company to do the R&D, so netted out
of our costs) TimingSpecific reference period Differs depending on
accounting purpose, e.g, for taxes, shareholders, other
Slide 45
Respondent(s) Authority Capacity Motivation Example: Official
Statistics on Research & Development 45 R&D EXPENSES
Management considerations Survey organisation R&D
PERFORMED
Slide 46
Respondent(s) Authority Capacity Motivation 46 Management
considerations Official Statistics Implications for Official
Statistics Survey organisation Business Records Role of the
National Statistical Institute is to: Bridge the gap between
Stakeholder requirements & data in business records. Translate
Stakeholder requirements into speci- fications for survey design.
Use survey design tools & features to aid business Rs, despite
variations across company Records/sources. Aid business R
(informant) in applying survey specs to data in records, to reduce
measurement error. Bear burden for both Stakeholders & Rs.
Survey Design Implications for Official Statisticians
Slide 47
47 1 2 Origins and Flows of Response Burden in Business Surveys
3 Management considerations Respondent(s) Authority Capacity
Motivation 4 Mismatches Burden flows back to survey organization,
and the data are reviewed and edited.
Slide 48
Respondent(s) Authority Capacity Motivation 48 1 2 4 Origins
and Flows of Response Burden in Business Surveys 3 Management
considerations Through data collection and processing, burden flows
from the Survey Organization back to stakeholders and data users
for any (remaining) differences between final data set and the
underlying concepts associated with the research question.
Slide 49
49 1. Theoretical world: Conceptualization Research Needs:
Questions, Issues, Problems, Decisions Empirical world: Physical
Observations Attributes Data Concepts Measurements Question(s)
Questionnaire Response Specification errors Operationalization
errors Measurement & Non-response errors Processing errors
Official Statistics in the Context of Total Survey Error Data
Mismatch Implications for Official Statistics: Final data retain
survey errors. Consequences of survey error for the underlying
concepts associated with the research question?
Slide 50
Respondent(s) Authority Capacity Motivation Enter BIG DATA
50
Slide 51
Respondent(s) Authority Capacity Motivation 51 Origins and
Flows of Response Burden in Business Surveys The correct and least
burdensome data for a business are data found in records.
Slide 52
Respondent(s) Authority Capacity Motivation 52 Origins and
Flows of Response Burden in Business Surveys The correct and least
burdensome data for a business are data found in records. BURDEN
Data Processing
Slide 53
Respondent(s) Authority Capacity Motivation 53 The correct and
least burdensome data for a business are data found in records.
BURDEN Data Processing Implications for Official Statisticians Role
of the National Statistical Institute is to: Bridge the gap between
Stakeholder requirements & data in business records. Use IT and
statistical methodologies to translate data from recorded data to
requested data, despite variations across company Records/sources.
Bears burden for Stakeholders. Survey Design Edits
Slide 54
Concluding Thoughts The role of Official Statisticians is to
bridge the conceptual and definitional gaps between Stakeholder
needs and the Respondents response process via survey design, data
collection and processing. 54
Slide 55
Concluding Remarks What is unique about Business Surveys with
respect to Burden? Survey response is a non-productive cost to the
business. Complex response process Organizationally determined R
selection controlled by the business, not the survey organization
Mismatch between data in business records and data needed to
support Official Statistics 55
Slide 56
Concluding Challenge Implications of the Business Survey
Response Process and Burden for Official Statistics: Data obtained
directly from or derived based on Business Records retain Burden,
Error, Costs. Burden flows from the Stakeholder to the Survey
Organization to the Respondent, and, ultimately, back to the
Stakeholder. Who has responsibility for associated Errors and
Costs??? 56