Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or...

Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or technical issues are those of the authors and not necessarily those of the U.S. Census Bureau. The Response Process and Burden in Business Surveys: Implications for Official Statistics Business Survey

Outline Business vs. household surveys Business context and Survey response process Response burden Consequences for error sources Focus on measurement error Implications for Official Statistics 2

Businesses vs. Households What are businesses? Economic units Made up of one or more establishments Locations where business is conducted or industrial operations performed (FCSM, 1988) Goals To produce goods and services To remain open and viable over tim To remain open and viable over time 3

Businesses vs. Households How are business surveys different? Population Data Survey design Response process 4

Household Surveys vs. Business Surveys Household Surveys Business Surveys Qs are aboutSelf Business or Organization Types of data Autobiographical, Attitudes, Behaviors Information abt the organization and/or its employees or clients; Financial; Quantities Data sourcesMemory Records & Rs knowledge of them Respondent Identity Self or ProxyInformant 5

Business Survey Topics Business characteristics Production measured by volume or value Number of employees, their qualifications, field of work, hours worked, payroll and benefits Expenditures and investments Volume sold and financial results (e.g., figures from balance sheets and income statements) Consumption of energy and other resources Transactions between businesses Business trend evaluations Evaluations of working conditions 6

Household Surveys Business Surveys R selection; R rule Single R or Selected proxy Multiple informants Mode Interviewer administered Self-administered Authority Voluntary or Mandatory Response Process Cognitive Cognitive and Organizational Household Surveys vs. Business Surveys continued 7

8 Complex Response Process in Business Surveys 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007, 2010) Cognitive Response Process (Tourangeau, 1984) Organizational Steps

What is Response Burden? Traditional Approach: Burden is the time taken to complete the survey Actual burden = Costs Perceived Response Burden: Respondents perceptions of their survey experience Respondents rarely equate burden with the time it takes to respond to a survey; instead they perceive burden in relation to mode, who is conducting the survey, and whether the produced statistics are useful to businesses/society. (Hedlin et al. 2005) 9 R = Number of people involved in completing the questionnaire T = Time taken by people involved C = Salary costs of these people

Causes of Perceived Response Burden (PRB) 10

Respondent(s) Authority Capacity Motivation 11 (Dale et al., 2007) Origins and Flows of Response Burden in Business Surveys

Respondent(s) Authority Capacity Motivation Origins and Flows of Response Burden in Business Surveys 12 1 2 3 Management considerations 4 Burden starts here with stakeholders and data users needs.

13 Official Statistics and Stakeholders Needs 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response

14 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response Specification errors Operationalization errors Measurement & Non-response errors Processing errors Official Statistics in the Context of Total Survey Error Through survey development and design, burden transfers from stakeholders & data users to the survey organization.

1 2 15 Origins and Flows of Response Burden in Business Surveys Respondent(s) Authority Capacity Motivation Through survey development and design, burden transfers from stakeholders & data users to the survey organization. So far, nothing particularly unique to business surveys, right? . but were just getting to the good part .. ..the business survey response process .. ..the business survey response process .. 3 Management considerations 4

16 Response Process in Business Surveys 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data Sudman et al. (2000) / Willimack & Nichols (2010); Bavdaz (2007, 2010)

17 Response Process: Organizationally Determined 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

18 Business Reasons for Records Sudman et al. (2000) / Willimack & Nichols (2010) External requirements: Legal purposes Regulatory purposes Internal business Manage resources To remain open and viable To produce goods and services Recorded data requested data Data are owned by the business

19 Response Process: Organizationally Determined Bavdaz (2007, 2010) 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

Performing Response Tasks Respondent selection, scheduling and prioritizing the Work to provide a survey response 20 Benefits Costs Surveys represent a cost without associated benefit non-productive cost

21 The most knowledgeable respondent The person who Knows the records and their location The meaning and purpose of the data in the records Determined in the context of record formation Response Process: Respondent Selection Who is the Best Respondent? Edwards & Cantor (1991)

22 Division of labor Compartmentalize the work for efficiency Organizational structures reflect the distribution of the work Distributed knowledge Data are located where they are needed and used Response Process: Respondent Selection Who is the Most Knowledgeable Respondent? Benefits Costs

24 Technical Core Those whose job is the production of goods & services And the associated data Boundary Spanners Interface with the outside world and across internal units Response Process: Respondent Selection Who is the Best Respondent? Tomaskovic-Devey, et al. (1994)

25 Boundary Spanners protect the Technical Core

26 Response Process Who is the Best Respondent? Willimack & Nichols (2010); Bavdaz (2007, 2010) Benefits Costs Routine Boundary spanners Respondent selection is under the control of the business Response Process Who is the Best Respondent? Willimack & Nichols (2010); Bavdaz (2007, 2010)

27 Response Process: Respondent Selection Who is the Best Respondent? Willimack & Nichols (2010); Tuttle (2009); Bavdaz (2007, 2010) Distributed data requires multiple respondents and data providers Respondent manages the collection of data from others Gathers and compiles data Respondent = Survey Coordinator

28 SURVEY COORDINATOR obtains data from others Emails/faxes actual survey question(s) including instructions Phone/email; does not include actual survey questions & instructions SURVEY COORDINATOR completes the questionnaire Respondent Data Provider Understands contents of records Expectations of the Survey Coordinator Lorenc (2006, 2007), Willimack (2007), Bavdaz (2007), Tuttle & Willimack (2012) Knows who to contact!

29 Response Process: Personal Cognitive Steps 1)Record formation & encoding 2)Organizing response tasks a)Respondent selection b)Scheduling c)Prioritizing & motivation 3)Comprehension 4)Retrieval 5)Judgment 6)Communication 7)Releasing the data

Retrieval Data sources Personal knowledge from memory Records Knowledge of records Retrieval incorporates Cognitive step retrieval of knowledge of data sources Access to records Physical act of retrieving data Extracting information from files Consulting multiple sources Compiling information 30

Judgment Judging the adequacy of the response Does the Respondents answer retrieved from records or memory match the data being requested? The Respondent doesnt have the requested data in records. How does the Respondent estimate that figure? The Respondent thinks, My answer is good enough. 31 Business Survey

32 Retrieval + Judgment: Requested Data Recorded Data Bavdaz (2010)

33 Mismatch Problems Adapted from Bavdaz (2010)

34 Mismatch Problems Adapted from Bavdaz (2010)

Business Survey Response Process Summary Management Considerations Survey response tasks = COST with no associated production Respondent selection Scheduling, prioritizing & motivation Data release process Respondent Requirements Taking the role as survey coordinator Knowledge of available information Handling differences between recorded data and requested data 35

36 1 Respondent(s) Authority Capacity Motivation Evidence of Response Burden in Business Surveys Management considerations 4 2 3

37 Perceived Response Burden 9,6 29,3 39,9 15,5 5,8 Very easy Quite easy Neither easy nor difficult Quite difficult Very difficult Sources of PRB E-technical E-Usability Other Layout No of questions Wait for information Terms Calculations Need of help Mismatch Response categories (2) Survey Design Perceived Response Burden (PRB) Source: Norwegian Structural Statistics, 2010 n = 16,572

(3) Time (Actual) Cost of Response Burden 38 Source: Statistics Canada, Survey of Regulatory Compliance Costs, 2005 & 2008 Average reporting costs, per business, increase was the size of the business increases (based on employment). Average reporting costs, per employee, decrease as the size of the business increases (based on employment).

39 (3) Conditions Associated with the Business Perceived Response Burden (PRB) PRB- index Size of0 9 employees0,11 establishment10 19 employees0,13 20 49 employees0,06 50 or more employees0,10 DocumentationNone0,15 problems1-0,14 2-0,23 3-0,45 No of10,15 Respondents2 or more-0,01 Value of PRB-Index PRB

40 (4) Respondent Characteristics Perceived Response Burden PRB- index Available timeVery easy to find0,68 Quite easy to find0,41 Neither easy nor difficult0,06 Quite difficult-0,09 Very difficult-0,37 Perception ofVery useful0,22 usefulnessQuite useful0,30 Neither useful nor useless0,18 Quite useless0,09 Very useless-0,11 ExperienceNo0,04 Yes0,11 Value of PRB-Index PRB

41 How Does Perceived Response Burden Affect Response Quality?

Mismatches & Measurement Error 42

Respondent(s) Authority Capacity Motivation Example: Official Statistics on Research & Development 43 R&D EXPENSES R&D PERFORMED Management considerations

R&D: Requested vs Recorded Data 44 Some Attributes of R&D Stakeholder : PERFORMED R&D PERFORMED by This Company Accountant: EXPENSES R&D EXPENSES TypeBasic, Applied, DevelopmentHuh? Tech Support & Maintenance No Yes, if done by same employees doing R&D GeographyYesHuh? Outsourced No (some other company did the R&D for this company) Yes (my company paid for this R&D for the benefit of my company) Reimbursed Yes (this company did the R&D, even though someone else paid for it No (someone else paid my company to do the R&D, so netted out of our costs) TimingSpecific reference period Differs depending on accounting purpose, e.g, for taxes, shareholders, other

Respondent(s) Authority Capacity Motivation Example: Official Statistics on Research & Development 45 R&D EXPENSES Management considerations Survey organisation R&D PERFORMED

Respondent(s) Authority Capacity Motivation 46 Management considerations Official Statistics Implications for Official Statistics Survey organisation Business Records Role of the National Statistical Institute is to: Bridge the gap between Stakeholder requirements & data in business records. Translate Stakeholder requirements into speci- fications for survey design. Use survey design tools & features to aid business Rs, despite variations across company Records/sources. Aid business R (informant) in applying survey specs to data in records, to reduce measurement error. Bear burden for both Stakeholders & Rs. Survey Design Implications for Official Statisticians

47 1 2 Origins and Flows of Response Burden in Business Surveys 3 Management considerations Respondent(s) Authority Capacity Motivation 4 Mismatches Burden flows back to survey organization, and the data are reviewed and edited.

Respondent(s) Authority Capacity Motivation 48 1 2 4 Origins and Flows of Response Burden in Business Surveys 3 Management considerations Through data collection and processing, burden flows from the Survey Organization back to stakeholders and data users for any (remaining) differences between final data set and the underlying concepts associated with the research question.

49 1. Theoretical world: Conceptualization Research Needs: Questions, Issues, Problems, Decisions Empirical world: Physical Observations Attributes Data Concepts Measurements Question(s) Questionnaire Response Specification errors Operationalization errors Measurement & Non-response errors Processing errors Official Statistics in the Context of Total Survey Error Data Mismatch Implications for Official Statistics: Final data retain survey errors. Consequences of survey error for the underlying concepts associated with the research question?

Respondent(s) Authority Capacity Motivation Enter BIG DATA 50

Respondent(s) Authority Capacity Motivation 51 Origins and Flows of Response Burden in Business Surveys The correct and least burdensome data for a business are data found in records.

Respondent(s) Authority Capacity Motivation 52 Origins and Flows of Response Burden in Business Surveys The correct and least burdensome data for a business are data found in records. BURDEN Data Processing

Respondent(s) Authority Capacity Motivation 53 The correct and least burdensome data for a business are data found in records. BURDEN Data Processing Implications for Official Statisticians Role of the National Statistical Institute is to: Bridge the gap between Stakeholder requirements & data in business records. Use IT and statistical methodologies to translate data from recorded data to requested data, despite variations across company Records/sources. Bears burden for Stakeholders. Survey Design Edits

Concluding Thoughts The role of Official Statisticians is to bridge the conceptual and definitional gaps between Stakeholder needs and the Respondents response process via survey design, data collection and processing. 54

Concluding Remarks What is unique about Business Surveys with respect to Burden? Survey response is a non-productive cost to the business. Complex response process Organizationally determined R selection controlled by the business, not the survey organization Mismatch between data in business records and data needed to support Official Statistics 55

Concluding Challenge Implications of the Business Survey Response Process and Burden for Official Statistics: Data obtained directly from or derived based on Business Records retain Burden, Error, Costs. Burden flows from the Stakeholder to the Survey Organization to the Respondent, and, ultimately, back to the Stakeholder. Who has responsibility for associated Errors and Costs??? 56

Thank You! Diane K. Willimack [email protected] 57

Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or...

Documents

Transcript of Diane K. Willimack U.S. Census Bureau Any views expressed on statistical, methodological, or...