Methods of Digital Analysis: Examples from BC Stats€¦ · Digital Analysis: Examples from BC...
42
Methods of Digital Analysis: Examples from BC Stats Carleton University – History 5702w 2016-01-23
Transcript of Methods of Digital Analysis: Examples from BC Stats€¦ · Digital Analysis: Examples from BC...
Methods of Digital Analysis: Examples from BC Stats Carleton University – History 5702w 2016-01-23
Martin Monkman
1
• Provincial Statistician & Director, BC Stats • B.Sc. and M.A. in Geography (UVic) • been getting paid to do data analysis in
one form or another since the mid-1980s • member of SABR
• bayesball.blogspot.ca
Presenter
Presentation Notes
As I started to think about this presentation, I realized that this is not a topic for the time we’ve got – one could spend a lifetime researching, thinking, and practicing this topic. I have been actively engaged in this topic – in one capacity or another – for (gads) 30 years. And I’m still learning! To use the metaphor of this library, we’ll visit a three of the stacks and pull one book from each, and skim it. There’s a wealth of information on each of the research projects we’ll covert … some links to further reading are in the notes on each slide. =-=-=-=-=-=-=-= Image: Cincinnati-Old Main Library that was demolished in 1955, Source: https://www.flickr.com/photos/80868612@N03/11983800566/in/photolist-aiP4WA-aSvCV2-bu7Ctj-3AEWz-aSv5ZP-bjtFJV-bhkodF-bhkoBD-jfY7p9-bujoxS-bhknQt-4VJnag-4TZ8fJ-5Av8e4-5kfMPH-59oHHj-5GzcXQ-5CowXv Creative Commons
3
Good decisions start with good data W
ISDO
M Synthesize
with Other Knowledge
KN
OW
LEDG
E
Greater Certainty &
Acceptability
INTELLIG
ENC
E
Judgment (Inference)
INFO
RM
ATION
DATA
Context (Decision
Framework)
Domain of the Information Provider Domain of the Information User Source: Modified from V.P. Barabba, “Through a Glass Less Darkly”, Presidential Address at the Annual Meeting of the American Statistical Association, Aug. 7, 1990.
Wise Decisions
Presenter
Presentation Notes
Source: Modified from V.P. Barabba, “Through a Glass Less Darkly”, Presidential Address at the Annual Meeting of the American Statistical Association, Aug. 7, 1990. The Value of Statistical Analysis All organizations – government, non-profits, businesses – need better data in order to make better decisions. They are on the right—the “information users” These decisions will lead to better outcomes and more effective use of resources BC Stats is on the left side of the diagram – the “information provider. We collect data, compile it into information, and add analysis that turns it into intelligence
Presenter
Presentation Notes
So today I want to touch on four topics. Image source: https://www.flickr.com/photos/duncan/3782076384 Creative commons
The BC Public Service Work Environment Survey
5
Biennial census survey of the entire B.C. Public Service • 25,000 people • 80% response rate • ~75 quantitative questions
• Detailed statistical modeling • One qualitative open-ended question
• ~2/3 of respondents provide answer • 2013: ~13,500 responses
Presenter
Presentation Notes
BC Public Service Work Environment Survey: http://www.bcstats.gov.bc.ca/StatisticsBySubject/EmployeeResearch/WES.aspx
“What one thing would you like your organization to focus on to improve your work environment?”
Presenter
Presentation Notes
B.C. Public Service Work Environment Survey The single qualitative question is specific: One thing Your organization (not work unit or government overall) Improvements in the work environment And yet we get multi-faceted answers, e.g. “Communication, and by that I mean a) b) and c)” “Executive management communicated vision of the organization to be better at managing stress & workload”
Presenter
Presentation Notes
And one wag even used this as their comment … an ASCII version of the classic Captain Picard face palm.
• Topic Modeling • 2006 – 2011 this coding was done
manually – a team of coders read each comment and assigned it to the appropriate code
• Labour intensive / time consuming / costly
• 2013: use of SPSS Text Analytics
Analytic Technique
Ministry of Technology, Innovation & Citizens’ Services, 2013
“Executive have developed very strong mission, vision and objectives...too much focus on some of these generic goals and themes and priority seems to be misaligned rather than focusing on the implications to operational activity, projects and [the organization]’s long term future.”
BC Public Service Agency, 2013
Presenter
Presentation Notes
Comments like these two are common across the BCPS. We’ve coded both of these to the topic theme of Vision, Mission and Goals But you can see elements of other themes, as well as some important sub-themes
“The organization] should follow through on the findings of different prioritizations that have occurred regarding programs and services that it can stop doing/delivering either permanently or temporarily. This would help bring a better balance to peoples' workloads especially in light of the staffing freeze.”
BC Public Service Agency, 2013
Presenter
Presentation Notes
Comments like these two are common across the BCPS. We’ve coded both of these to the topic theme of Vision, Mission and Goals But you can see elements of other themes, as well as some important sub-themes
Top Themes: • Vision, Mission & Goals (59%) • Stress & Workload (47%) • Tools, Equipment & Physical Environment (41%) Leading sub-themes:
– Assess plans, priorities and strategies for the organization
– Review funding or budget allocation to program
– Pay attention to the public interest
Comments from Within: Vision, Mission & Goals
BC Public Service Agency, 2013
Presenter
Presentation Notes
PROJECTS Image source: https://www.flickr.com/photos/holeymoon/2098042797/in/photostream/ Creative commons
“Graphics are instruments for reasoning about quantitative information.” (Edward R. Tufte)
Presenter
Presentation Notes
Edward R. Tufte, “The Visual Display of Quantitative Information” (and his other books) Charts summarize data to make it more understandable That summary is often to compare different sets of data Exploratory Data Analysis - Graphics as a way to understand the data - Tukey, Cleveland Narrative - “data journalism”: telling a story with the data, answering the 5 W questions - Alberto Cairo - storytellingwithdata.com -
“The greatest value of a picture is when it forces us to notice what we never expected to see.” – John Tukey
Presenter
Presentation Notes
John Tukey, author of “Exploratory Data Analysis” (1977)
Data Are Messy
Presenter
Presentation Notes
After the 2009 B.C. general election, BC Stats got information from Elections BC about the voting behaviour of everyone registered to vote in the 2009 election. The data looked like this: the voter’s date of birth, and for the three elections in 2001, 2005, and 2009 whether the individual voted or didn’t (represented as TRUE or FALSE), or was not eligible (this would be due to age, residency, or citizenship) We also received a field showing the electoral district the individual voted in See: / Who heads to the polls? Exploring the demographics of voters in British Columbia (March 2010) http://www.elections.bc.ca/docs/stats/Who-heads-to-the-polls.pdf
Data Are Messy
Presenter
Presentation Notes
We gave each voter a “Pattern” category, based whether they voted in 2009 or not, and how that compared to their status in the two previous events Consistent did the same thing (vote or not vote) in all three election events. Inconsistent voters voted in 2009, were eligible in at least one of the two previous elections, but didn’t vote (or were not eligible) in at least one of the two previous Inconsistent non-voters did not vote in 2009, were eligible in at least one of the two previous elections, but voted (or were not eligible) in at least one of the two previous Newly Eligible appeared on the voter list for the first time in 2009 We also took the date of birth and binned them into 5 year categories (note that some people had clearly incorrect DOBs, such as 2009) Used the postal code to assign the individual to the appropriate Census Division and Census Sub-Division
Data Are Structured…But Still Not Clear
Presenter
Presentation Notes
This chart and the next two are based on individual voter records from the 2001, 2005, and 2009 B.C. general elections. The voter / non-voter split is based on the 2009 election. Consistent did the same thing (vote or not vote) in all three election events. Inconsistent voters voted in 2009, were eligible in at least one of the two previous elections, but didn’t vote (or were not eligible) in at least one of the two previous Inconsistent non-voters did not vote in 2009, were eligible in at least one of the two previous elections, but voted (or were not eligible) in at least one of the two previous Newly Eligible appeared on the voter list for the first time in 2009 Consistent: Voter = 32% / Non-voter = 6% Inconsistent: Voter = 10% / Non-voter = 14% Newly eligible: Voter = 14% / Non-voter = 25% See: / Who heads to the polls? Exploring the demographics of voters in British Columbia (March 2010) http://www.elections.bc.ca/docs/stats/Who-heads-to-the-polls.pdf
-200,000
-150,000
-100,000
-50,000
0
50,000
100,000
150,000
200,000# of Registered Voters
Age Group
VOTERS
NON-VOTERS
Newly eligible
Inconsistent
Consistent
Newly eligible
Inconsistent
Consistent
Presenter
Presentation Notes
From: Who heads to the polls? Exploring the demographics of voters in British Columbia (March 2010) http://www.elections.bc.ca/docs/stats/Who-heads-to-the-polls.pdf Consistent: Voter = 32% Non-voter = 6% Inconsistent: Voter = 10% Non-voter = 14% Newly eligible: Voter = 14% Non-voter = 25%
-100%
-75%
-50%
-25%
0%
25%
50%
75%
100%% of Registered Voters
Age Group
VOTERS
NON-VOTERS
Newly eligible
Inconsistent
Consistent
Newly eligible
Inconsistent
Consistent
Presenter
Presentation Notes
From: Who heads to the polls? Exploring the demographics of voters in British Columbia (March 2010) http://www.elections.bc.ca/docs/stats/Who-heads-to-the-polls.pdf See also: Motivations and barriers: Exploring voting behaviour in British Columbia http://www.elections.bc.ca/docs/stats/Motivations-and-barriers-2010.pdf
B.C. Provincial Election 2009 Voting pattern & behaviour by age group
Presenter
Presentation Notes
This chart and the next two are based on individual voter records from the 2001, 2005, and 2009 B.C. general elections. The voter / non-voter split is based on the 2009 election. Consistent did the same thing (vote or not vote) in all three election events. Inconsistent voters voted in 2009, were eligible in at least one of the two previous elections, but didn’t vote (or were not eligible) in at least one of the two previous Inconsistent non-voters did not vote in 2009, were eligible in at least one of the two previous elections, but voted (or were not eligible) in at least one of the two previous Newly Eligible appeared on the voter list for the first time in 2009 Consistent: Voter = 32% / Non-voter = 6% Inconsistent: Voter = 10% / Non-voter = 14% Newly eligible: Voter = 14% / Non-voter = 25% See: / Who heads to the polls? Exploring the demographics of voters in British Columbia (March 2010) http://www.elections.bc.ca/docs/stats/Who-heads-to-the-polls.pdf
Presenter
Presentation Notes
Data Analysis Image source: https://www.flickr.com/photos/holeymoon/1813639142 Creative commons
Presenter
Presentation Notes
Big data has changed the information that gets captured. In days of yore, there was no central record of who sent and received letters. In addition to the original, perhaps the only other evidence would be a carbon-copy kept at the sender’s location. But emails are an electronic media, and logs can and are kept of the transmission and receipt of those emails. Such a log system exists for the B.C. government’s email server. In 2014, a freedom of information request was submitted, asking for all of the information in the logs for the six month period January 1, 2013 to June 30, 2013. (The fact that the provincial general election was held in May 2013 may or may not have been coincidental.) This was a total of nearly 500,000,000 (half a billion) records. The response from the government’s privacy and records management office was that the work necessary to ensure that the subject lines were devoid of sensitive information (both confidential cabinet correspondence and personal information) was onerous, so that was removed from the request. What remained was the time stamp, receipient(s), and sender. Source: New York Public Library Digital Collections http://digitalcollections.nypl.org/items/510d47e1-275c-a3d9-e040-e00a18064a99
Data Are Messy
Presenter
Presentation Notes
This is a screen capture of the data … it happens to be some of the email I sent. I was asked by the Privacy Officer working on the request if I saw any risk associated with this request. I thought about it and answered “Yes, and let me show you.” I was provided a sample of the records in the file, from a 24 hour period in October 2013. These are the first 5 emails I sent in that period, but there were nearly 200,000 emails logged in that 24 hour span. In order to make a network diagram, I needed to clean this up. Step 1 was to filter it so that it had only those emails with me as the sender, then me as the recipient. Step 2 was to take those people who had sent me and/or received email from me, and then filter on them as sender and receiver (so now we’ve got 2 degrees of Martin) Step 3 was to atomize the multipe recipient emails, so that each recipient had their own line. But first I gave each record a unique ID number, so that it was still possible to show that eg that second email in the list had 3 recipients Step 4 was to write the code in R to create the visualization
Presenter
Presentation Notes
The network diagram created from the email data of 2013-10-07, with Martin at the hub. Reporting relationships are starting to be visible (e.g. Jill’s supervisory relationship with Julie, Sonja and Linda; Julie’s management of contracted service providers [the two Ext accounts]) But also non-work personal items appear; Linda was a member of the BC Govt Employee Fitness Centre, and had sent the Centre’s admin account an email
Order of the Privacy Commissioner
• The email logs contain and reveal personal information of B.C. government employees
• The logs disclose personal information based on who they emailed (e.g. union representatives)
Presenter
Presentation Notes
Orders of the Privacy Commissioner, British Columbia https://www.oipc.bc.ca/rulings/orders/ The specific order: https://www.oipc.bc.ca/orders/1887 Para 33: “...the Logs contain and reveal personal information of BC Government employees. Because of the presence of employee email addresses in the Logs, the Logs reveal the work activities of identifiable BC Government employees, which is the employee’s personal information. The Logs will also disclose personal information about employees based on who they emailed, for example emails to union representatives. The Logs will also reveal personal information about employee’s work hours and leave. Finally, the Logs will contain personal emails. The patterns of personal (non-work) email use could disclose employee personal relationships. “
www.bcstats.gov.bc.ca @BCStats
Appendix 1 – BC Stats
27
What We Do Who We Are
About BC Stats
• The central statistical agency for the Province of British Columbia
• Function in legislation since 1894
• Operate under the legal authority of the Statistics Act, which provides a legislative mandate to …
• Collect, analyze, and distribute statistical information on B.C.
• Assist government with their statistical activities
• Co-ordinate statistical activities between B.C. and Statistics Canada
28
The Act also provides statutory protection of the privacy and security of data we collect
Presenter
Presentation Notes
British Columbia Statistics Act http://www.bclaws.ca/civix/document/id/complete/statreg/96439_01
The Canadian National Statistical System
29
• In Canada we have a very centralized statistical system, with Statistics Canada playing the lead role
• Under the British North America Act of 1867, the federal government was given non-exclusive responsibility for the census and for collecting statistics.
• Since Confederation, each province and territory has developed its own, independent statistical agency.
The Canadian National Statistical System
30
• In B.C., the formal establishment of a central statistical organization took place 1894 with the enactment of the Legislative Library and Bureau of Statistics Act
• The British Columbia Statistics Act, modeled on the federal legislation, was passed in 1979
• We work closely with Statistics Canada through a number of federal-provincial-territorial committees to coordinate statistical activities.
BC Stats: Who Are Our Clients?
31
Primary client groups: • BC government ministries • Statistics Canada
Other client groups: • crown corporations • public agencies and education institutions • business community & academics • public
BC Stats: budget
32
Presenter
Presentation Notes
The dark bars represent the base funding allocation (the “blue book” voted appropriation). The light portion of the bar is the amount of recoveries – in other words, projects paid for by client ministries and public agencies. The black line is the net expenditures – that is to say, the amount of the light bar that’s above the line is what BC Stats spent on contracted services, primarily data collection services associated with large-scale surveys. In the mid-nineties, recoveries represented about 10% of our expenditures. As time has passed, the share of recoveries has grown significantly – it started with the Forest Renewal and high tech sector projects in the late nineties, and then the addition and growth of the post-secondary student outcomes research was a big boost, the growth of public sector research, and other recoveries pushing our total recoveries over the $6 million mark in 2009.
BC Stats: staff
33
• Professional researchers • Academic qualifications:
• Social sciences (sociology, psychology, economics) • Natural sciences (geography, biology) • Statistics and Computer Science • and others!
• Experience in quantitative analysis, data collection • Always learning!
BC Stats: data sources
34
• Statistics Canada • Administrative records • Surveys
• Web: E-mail invitation with internet software for data collection • Phone: under contract by private companies. • Paper, by mail: BC Government’s mail service.
Presenter
Presentation Notes
Statistics Canada has a wealth of data already available that provides insights and information that can inform policy decisions. We will be looking deeper into that topic later in the presentation. BC Stats also has information sharing agreements with various public agencies that allows us access to administrative records that can be used for statistical purposes. Two examples: Tax revenue information from hotels and other accommodation industry businesses provides a proxy for spending by tourists on hotel rooms Information from BC Hydro on the number of residential accounts provides a proxy for the number of households in an area, which is used for population estimation Where data don’t already exist, BC Stats carries out surveys. We do this in three modes – web, phone, and on paper. Surveys are most effective where we have access to underlying administrative records. These records provide us with valuable information, starting with names of program participants and their contact information, which facilitate contacting people. And those admin records can also contain demographic information that we can use for analysis, without having to add more questions for the respondent to answer (reducing respondent burden)
BC Stats: analysis
35
• Techniques • Driven by information needs • Examples:
• Counts and percentages • Time series analysis • Multivariate regression • Statistical modelling
• Software tools • Excel, SPSS, STATA, eViews, R
Presenter
Presentation Notes
The statistical techniques we use in a particular case depends on what the information needs are. Sometimes it’s as simple as counts and percentages. In other cases we’re doing time series analysis, including techniques such as trend analysis and seasonal adjustment And sometimes we’re deep into multivariate methods, such as regression, factor analysis, and modelling techniques (econometrics, structural equation modeling)
Appendix 2 – Work Environment Survey
36
BC Public Service Work Environment Survey 2013
Presenter
Presentation Notes
This bubble chart shows the distribution of the BC Public Service by 6 different engagement groups. Over half of the BCPS – 56% -- are Engaged or Moderately Engaged. But just over 1 in 4 (26%) are Disengaged or Minimally Engaged.
BC Public Service Work Environment Survey 2013 Distribution of engagement scores by work unit
Presenter
Presentation Notes
Because the survey is a census of the BC Public Service (BCPS) and there is a high response rate (~80% in 2013) we are able to generate reports for each work unit in the BCPS. (A work unit has a minimum of 20 staff, with at least a 50% response rate, so always 10 or more respondents.) This chart shows the distribution of engagement scores across the 629 work units of the BC Public Service. While the majority are clustered in the 55-75 point range, there are some high-performing work units with scores above 80 – and there are some work units with scores below 40. In the high performing extremes, this tells us that everybody in the work unit is answering all the engagement questions with a 4 out of 5, except for some folks who are answering 5 out of 5. And vice versa for the work units with low engagement.
BC Public Service Work Environment Survey 2013
Presenter
Presentation Notes
This is the “house model” of the Work Environment Survey. Each of the boxes represents a driver, with the elements of engagement in the roof, the workplace functions as the building blocks of the walls, and executive and supervisory level management as the foundation. Each of the drivers is made up of between one and four questions, each asked on a five point agreement scale (from “strongly disagree” to “strongly agree”). A house model is created in each ministry and work unit report—there were over 800 reports generated for the 2013 WES. See http://www.bcstats.gov.bc.ca/StatisticsBySubject/EmployeeResearch/WES/WESPublicResources.aspx
The Engagement Model
Presenter
Presentation Notes
This diagram shows the statistical relationships between all of the drivers in the WES house model. The weight of the line indicates the strength of the statistical relationship between the drivers. It’s a complicated web, so we’ve summarized the main routes from Management through to Engagement as pathways.
Presenter
Presentation Notes
Here’s one pathway. The “Vision” path through the model links Executive Level Management to Vision, Mission & Goals to Organization Satisfaction. See the report “Tracing the Top Engagement Pathways for the BC Public Service: Pathways Guide” (BC Stats, 2014) http://www.bcstats.gov.bc.ca/StatisticsBySubject/EmployeeResearch/WES/WESPublicResources/ModelPathwaysMethodologies.aspx
