WERS the Validity: What’s The Current State of Play? Dr Andrew R Timming Reader in Management...
-
Upload
kellie-wilkinson -
Category
Documents
-
view
215 -
download
0
Transcript of WERS the Validity: What’s The Current State of Play? Dr Andrew R Timming Reader in Management...
WERS the Validity: What’s The Current State of Play?Dr Andrew R Timming
Reader in Management
University of St Andrews
Preface
• WERS is the single greatest source of data on work and employment in the history of not just the United Kingdom, but also the world
• No other dataset comes close
• But scientific advancement depends on critique
Background
• From WIRS (1980, 1984, 1990) to WERS (1998, 2004, 2011)
• WERS 2011: (i) management questionnaire, (ii) financial performance questionnaire, (iii) worker representative questionnaire, (iv) survey of employees and (v) employee profile questionnaire
• 2,680 face-to-face interviews with managers; 1,002 interviews with employee reps; 21,981 respondents completed the survey of employees
Structure of Presentation
• Look at my top 10 “best practices” for survey design
• Evaluate how the WERS instruments stand in relation to those top 10 practices
• Focus mainly on survey of employees
• Summarize key points to draw from the presentation
Tip 1
• Surveys should be relevant and timely
• Relevant according to whose judgement?
• Evidence of timeliness is clear: workplaces in the shadow of recession
• But we still have to ask how much value is added by some of the questions.
Timeliness
Relevance
Tip 2
• The layout should be clear, not cluttered
• If the survey is too spread out, it reduces response rate; too cluttered, it reduces the quality of the data
• On this point, the Survey of Employees gets an A+
• Nice aesthetics, clear instructions, inviting, good boundaries and delineation between items, clear flow
Clear Layout
Tip 3
• Strike a balance on length of the survey
• MQ response rate=46.3%; WR response rate=63.9%; Survey of Employees=54.3%; financial performance questionnaire= 31.8%
• Face-to-face administration is usually much higher than postal, but in WERS 2011 we see the opposite
• Was the length of the instruments driving away respondents?
Tip 4
• Items should be short and simple
• Principle of parsimony
• Express what you need to express in as few words as possible and as clearly as possible
• On this point, WERS does a pretty good job, but not perfect . . .
Tip 5
• Avoid leading questions
• ‘Do you agree that Scotland should be an independent country?’ vs ‘Should Scotland be an independent country?’
• How we ask a question can result an an expected response
Overall, how satisfied or dissatisfied are you with …
Tip 6
• Response categories should be exhaustive and mutually exclusive
• Exhaustive: all possibly choices are listed
• Mutually exclusive: no two choices overlap
• WERS is generally good on this point
Mutually exclusive?
Tip 7
• Respondents should be knowledgeable about the topic or topics.
• Albanian foreign policy towards China . . .
• No problem with survey of employees, but there may be concerns with management and employee rep questionnaires
Are workers’ reps qualified to speculate?
Can a manager really know the answer to this question?
Tip 8
• Items must not be double-barreled
• ‘Double-barreled’ implies that a single item is asking two questions in one
• Double-barreled items lead to serious questions about whether we’re measuring what we say we’re measuring
• On this point, WERS could be improved
Double-barreled items
Tip 9
• Complex constructs require multiple items
• Multiple items allow for the construction of latent variables
• On this point, WERS does a really good job
Tip 10
• Survey instruments should always be piloted and reformed according to results
• Obviously, the fact that there are different iterations of the survey implies sufficient piloting
• But on the question of reform, we need to continue to refine the instrument, even though this may lead to a break in continuity from previous versions
Summary
• On the whole, WERS 2011 is a fantastic resource and a very well designed set of instruments . . . but it’s not perfect
• What is included and excluded from surveys is a political choice and influenced by power
• Do we tolerate imperfection for the sake of continuity?
Counter-Arguments
• ‘We shouldn’t change or reduce the amount of data gathered because they are always relevant to someone’.
• ‘Other surveys have used these same measures, so why shouldn’t we as well’?
• ‘Our response rates are, in fact, much better than other comparable surveys’.
• ‘Effecting changes now will make it difficult to do comparable longitudinal analyses’.
• ‘You’ve cherry picked a few examples to make the whole survey look bad’.
• ‘Making changes would cost too much money without an effective return’.