Transcript of Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or...
- Slide 1
- Diane K. Willimack U.S. Census Bureau Any views expressed on
statistical, methodological, or technical issues are those of the
authors and not necessarily those of the U.S. Census Bureau. The
Response Process and Burden in Business Surveys: Implications for
Official Statistics Business Survey
- Slide 2
- Outline Business vs. household surveys Business context and
Survey response process Response burden Consequences for error
sources Focus on measurement error Implications for Official
Statistics 2
- Slide 3
- Businesses vs. Households What are businesses? Economic units
Made up of one or more establishments Locations where business is
conducted or industrial operations performed (FCSM, 1988) Goals To
produce goods and services To remain open and viable over tim To
remain open and viable over time 3
- Slide 4
- Businesses vs. Households How are business surveys different?
Population Data Survey design Response process 4
- Slide 5
- Household Surveys vs. Business Surveys Household Surveys
Business Surveys Qs are aboutSelf Business or Organization Types of
data Autobiographical, Attitudes, Behaviors Information abt the
organization and/or its employees or clients; Financial; Quantities
Data sourcesMemory Records & Rs knowledge of them Respondent
Identity Self or ProxyInformant 5
- Slide 6
- Business Survey Topics Business characteristics Production
measured by volume or value Number of employees, their
qualifications, field of work, hours worked, payroll and benefits
Expenditures and investments Volume sold and financial results
(e.g., figures from balance sheets and income statements)
Consumption of energy and other resources Transactions between
businesses Business trend evaluations Evaluations of working
conditions 6
- Slide 7
- Household Surveys Business Surveys R selection; R rule Single R
or Selected proxy Multiple informants Mode Interviewer administered
Self-administered Authority Voluntary or Mandatory Response Process
Cognitive Cognitive and Organizational Household Surveys vs.
Business Surveys continued 7
- Slide 8
- 8 Complex Response Process in Business Surveys 1)Record
formation & encoding 2)Organizing response tasks a)Respondent
selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data Sudman et al. (2000) / Willimack & Nichols (2010);
Bavdaz (2007, 2010) Cognitive Response Process (Tourangeau, 1984)
Organizational Steps
- Slide 9
- What is Response Burden? Traditional Approach: Burden is the
time taken to complete the survey Actual burden = Costs Perceived
Response Burden: Respondents perceptions of their survey experience
Respondents rarely equate burden with the time it takes to respond
to a survey; instead they perceive burden in relation to mode, who
is conducting the survey, and whether the produced statistics are
useful to businesses/society. (Hedlin et al. 2005) 9 R = Number of
people involved in completing the questionnaire T = Time taken by
people involved C = Salary costs of these people
- Slide 10
- Causes of Perceived Response Burden (PRB) 10
- Slide 11
- Respondent(s) Authority Capacity Motivation 11 (Dale et al.,
2007) Origins and Flows of Response Burden in Business Surveys
- Slide 12
- Respondent(s) Authority Capacity Motivation Origins and Flows
of Response Burden in Business Surveys 12 1 2 3 Management
considerations 4 Burden starts here with stakeholders and data
users needs.
- Slide 13
- 13 Official Statistics and Stakeholders Needs 1. Theoretical
world: Conceptualization Research Needs: Questions, Issues,
Problems, Decisions Empirical world: Physical Observations
Attributes Data Concepts Measurements Question(s) Questionnaire
Response
- Slide 14
- 14 1. Theoretical world: Conceptualization Research Needs:
Questions, Issues, Problems, Decisions Empirical world: Physical
Observations Attributes Data Concepts Measurements Question(s)
Questionnaire Response Specification errors Operationalization
errors Measurement & Non-response errors Processing errors
Official Statistics in the Context of Total Survey Error Through
survey development and design, burden transfers from stakeholders
& data users to the survey organization.
- Slide 15
- 1 2 15 Origins and Flows of Response Burden in Business Surveys
Respondent(s) Authority Capacity Motivation Through survey
development and design, burden transfers from stakeholders &
data users to the survey organization. So far, nothing particularly
unique to business surveys, right? . but were just getting to the
good part .. ..the business survey response process .. ..the
business survey response process .. 3 Management considerations
4
- Slide 16
- 16 Response Process in Business Surveys 1)Record formation
& encoding 2)Organizing response tasks a)Respondent selection
b)Scheduling c)Prioritizing & motivation 3)Comprehension
4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman
et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007,
2010)
- Slide 17
- 17 Response Process: Organizationally Determined 1)Record
formation & encoding 2)Organizing response tasks a)Respondent
selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data
- Slide 18
- 18 Business Reasons for Records Sudman et al. (2000) /
Willimack & Nichols (2010) External requirements: Legal
purposes Regulatory purposes Internal business Manage resources To
remain open and viable To produce goods and services Recorded data
requested data Data are owned by the business
- Slide 19
- 19 Response Process: Organizationally Determined Bavdaz (2007,
2010) 1)Record formation & encoding 2)Organizing response tasks
a)Respondent selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data
- Slide 20
- Performing Response Tasks Respondent selection, scheduling and
prioritizing the Work to provide a survey response 20 Benefits
Costs Surveys represent a cost without associated benefit
non-productive cost
- Slide 21
- 21 The most knowledgeable respondent The person who Knows the
records and their location The meaning and purpose of the data in
the records Determined in the context of record formation Response
Process: Respondent Selection Who is the Best Respondent? Edwards
& Cantor (1991)
- Slide 22
- 22 Division of labor Compartmentalize the work for efficiency
Organizational structures reflect the distribution of the work
Distributed knowledge Data are located where they are needed and
used Response Process: Respondent Selection Who is the Most
Knowledgeable Respondent? Benefits Costs
- Slide 23
- 23
- Slide 24
- 24 Technical Core Those whose job is the production of goods
& services And the associated data Boundary Spanners Interface
with the outside world and across internal units Response Process:
Respondent Selection Who is the Best Respondent? Tomaskovic-Devey,
et al. (1994)
- Slide 25
- 25 Boundary Spanners protect the Technical Core
- Slide 26
- 26 Response Process Who is the Best Respondent? Willimack &
Nichols (2010); Bavdaz (2007, 2010) Benefits Costs Routine Boundary
spanners Respondent selection is under the control of the business
Response Process Who is the Best Respondent? Willimack &
Nichols (2010); Bavdaz (2007, 2010)
- Slide 27
- 27 Response Process: Respondent Selection Who is the Best
Respondent? Willimack & Nichols (2010); Tuttle (2009); Bavdaz
(2007, 2010) Distributed data requires multiple respondents and
data providers Respondent manages the collection of data from
others Gathers and compiles data Respondent = Survey
Coordinator
- Slide 28
- 28 SURVEY COORDINATOR obtains data from others Emails/faxes
actual survey question(s) including instructions Phone/email; does
not include actual survey questions & instructions SURVEY
COORDINATOR completes the questionnaire Respondent Data Provider
Understands contents of records Expectations of the Survey
Coordinator Lorenc (2006, 2007), Willimack (2007), Bavdaz (2007),
Tuttle & Willimack (2012) Knows who to contact!
- Slide 29
- 29 Response Process: Personal Cognitive Steps 1)Record
formation & encoding 2)Organizing response tasks a)Respondent
selection b)Scheduling c)Prioritizing & motivation
3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing
the data
- Slide 30
- Retrieval Data sources Personal knowledge from memory Records
Knowledge of records Retrieval incorporates Cognitive step
retrieval of knowledge of data sources Access to records Physical
act of retrieving data Extracting information from files Consulting
multiple sources Compiling information 30
- Slide 31
- Judgment Judging the adequacy of the response Does the
Respondents answer retrieved from records or memory match the data
being requested? The Respondent doesnt have the requested data in
records. How does the Respondent estimate that figure? The
Respondent thinks, My answer is good enough. 31 Business
Survey
- Slide 32
- 32 Retrieval + Judgment: Requested Data Recorded Data Bavdaz
(2010)
- Slide 33
- 33 Mismatch Problems Adapted from Bavdaz (2010)
- Slide 34
- 34 Mismatch Problems Adapted from Bavdaz (2010)
- Slide 35
- Business Survey Response Process Summary Management
Considerations Survey response tasks = COST with no associated
production Respondent selection Scheduling, prioritizing &
motivation Data release process Respondent Requirements Taking the
role as survey coordinator Knowledge of available information
Handling differences between recorded data and requested data
35
- Slide 36
- 36 1 Respondent(s) Authority Capacity Motivation Evidence of
Response Burden in Business Surveys Management considerations 4 2
3
- Slide 37
- 37 Perceived Response Burden 9,6 29,3 39,9 15,5 5,8 Very easy
Quite easy Neither easy nor difficult Quite difficult Very
difficult Sources of PRB E-technical E-Usability Other Layout No of
questions Wait for information Terms Calculations Need of help
Mismatch Response categories (2) Survey Design Perceived Response
Burden (PRB) Source: Norwegian Structural Statistics, 2010 n =
16,572
- Slide 38
- (3) Time (Actual) Cost of Response Burden 38 Source: Statistics
Canada, Survey of Regulatory Compliance Costs, 2005 & 2008
Average reporting costs, per business, increase was the size of the
business increases (based on employment). Average reporting costs,
per employee, decrease as the size of the business increases (based
on employment).
- Slide 39
- 39 (3) Conditions Associated with the Business Perceived
Response Burden (PRB) PRB- index Size of0 9 employees0,11
establishment10 19 employees0,13 20 49 employees0,06 50 or more
employees0,10 DocumentationNone0,15 problems1-0,14 2-0,23 3-0,45 No
of10,15 Respondents2 or more-0,01 Value of PRB-Index PRB
- Slide 40
- 40 (4) Respondent Characteristics Perceived Response Burden
PRB- index Available timeVery easy to find0,68 Quite easy to
find0,41 Neither easy nor difficult0,06 Quite difficult-0,09 Very
difficult-0,37 Perception ofVery useful0,22 usefulnessQuite
useful0,30 Neither useful nor useless0,18 Quite useless0,09 Very
useless-0,11 ExperienceNo0,04 Yes0,11 Value of PRB-Index PRB
- Slide 41
- 41 How Does Perceived Response Burden Affect Response
Quality?
- Slide 42
- Mismatches & Measurement Error 42
- Slide 43
- Respondent(s) Authority Capacity Motivation Example: Official
Statistics on Research & Development 43 R&D EXPENSES
R&D PERFORMED Management considerations
- Slide 44
- R&D: Requested vs Recorded Data 44 Some Attributes of
R&D Stakeholder : PERFORMED R&D PERFORMED by This Company
Accountant: EXPENSES R&D EXPENSES TypeBasic, Applied,
DevelopmentHuh? Tech Support & Maintenance No Yes, if done by
same employees doing R&D GeographyYesHuh? Outsourced No (some
other company did the R&D for this company) Yes (my company
paid for this R&D for the benefit of my company) Reimbursed Yes
(this company did the R&D, even though someone else paid for it
No (someone else paid my company to do the R&D, so netted out
of our costs) TimingSpecific reference period Differs depending on
accounting purpose, e.g, for taxes, shareholders, other
- Slide 45
- Respondent(s) Authority Capacity Motivation Example: Official
Statistics on Research & Development 45 R&D EXPENSES
Management considerations Survey organisation R&D
PERFORMED
- Slide 46
- Respondent(s) Authority Capacity Motivation 46 Management
considerations Official Statistics Implications for Official
Statistics Survey organisation Business Records Role of the
National Statistical Institute is to: Bridge the gap between
Stakeholder requirements & data in business records. Translate
Stakeholder requirements into speci- fications for survey design.
Use survey design tools & features to aid business Rs, despite
variations across company Records/sources. Aid business R
(informant) in applying survey specs to data in records, to reduce
measurement error. Bear burden for both Stakeholders & Rs.
Survey Design Implications for Official Statisticians
- Slide 47
- 47 1 2 Origins and Flows of Response Burden in Business Surveys
3 Management considerations Respondent(s) Authority Capacity
Motivation 4 Mismatches Burden flows back to survey organization,
and the data are reviewed and edited.
- Slide 48
- Respondent(s) Authority Capacity Motivation 48 1 2 4 Origins
and Flows of Response Burden in Business Surveys 3 Management
considerations Through data collection and processing, burden flows
from the Survey Organization back to stakeholders and data users
for any (remaining) differences between final data set and the
underlying concepts associated with the research question.
- Slide 49
- 49 1. Theoretical world: Conceptualization Research Needs:
Questions, Issues, Problems, Decisions Empirical world: Physical
Observations Attributes Data Concepts Measurements Question(s)
Questionnaire Response Specification errors Operationalization
errors Measurement & Non-response errors Processing errors
Official Statistics in the Context of Total Survey Error Data
Mismatch Implications for Official Statistics: Final data retain
survey errors. Consequences of survey error for the underlying
concepts associated with the research question?
- Slide 50
- Respondent(s) Authority Capacity Motivation Enter BIG DATA
50
- Slide 51
- Respondent(s) Authority Capacity Motivation 51 Origins and
Flows of Response Burden in Business Surveys The correct and least
burdensome data for a business are data found in records.
- Slide 52
- Respondent(s) Authority Capacity Motivation 52 Origins and
Flows of Response Burden in Business Surveys The correct and least
burdensome data for a business are data found in records. BURDEN
Data Processing
- Slide 53
- Respondent(s) Authority Capacity Motivation 53 The correct and
least burdensome data for a business are data found in records.
BURDEN Data Processing Implications for Official Statisticians Role
of the National Statistical Institute is to: Bridge the gap between
Stakeholder requirements & data in business records. Use IT and
statistical methodologies to translate data from recorded data to
requested data, despite variations across company Records/sources.
Bears burden for Stakeholders. Survey Design Edits
- Slide 54
- Concluding Thoughts The role of Official Statisticians is to
bridge the conceptual and definitional gaps between Stakeholder
needs and the Respondents response process via survey design, data
collection and processing. 54
- Slide 55
- Concluding Remarks What is unique about Business Surveys with
respect to Burden? Survey response is a non-productive cost to the
business. Complex response process Organizationally determined R
selection controlled by the business, not the survey organization
Mismatch between data in business records and data needed to
support Official Statistics 55
- Slide 56
- Concluding Challenge Implications of the Business Survey
Response Process and Burden for Official Statistics: Data obtained
directly from or derived based on Business Records retain Burden,
Error, Costs. Burden flows from the Stakeholder to the Survey
Organization to the Respondent, and, ultimately, back to the
Stakeholder. Who has responsibility for associated Errors and
Costs??? 56
- Slide 57
- Thank You! Diane K. Willimack diane.k.willimack@census.gov
57