Post on 22-Feb-2019
Big Data, can you tell me where I want to go from how? to delivering
ESS Modernisation Workshop Bucharest, 16-17 Mar 2016
Eurostat
Big data at Eurostat
ESS (European Statistical System) Scheveningen Memorandum Sep 2013
Task Force Big Data Big Data Roadmap and Action Plan June 2014
May 2015 Big Data Business Cases May 2015 BIGD Project Nov 2015 Big Data ESSnet Jan 2016 Contract on supporting services
ESS Vision 2020
European Commission Communication Data4Policy Networking Group
International cooperation (UNSD, UNECE, etc.) UN/ECE project Big data in official statistics (Sandbox) UNSD Global WG on Big Data
2
Scheveningen
http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/0_DOCS/estat/SCHEVENINGEN_MEMORANDUM Final version_0.pdfhttp://www.cros-portal.eu/content/ess-big-data-action-plan-and-roadmap-10http://www1.unece.org/stat/platform/display/bigdata/Big+Data+Project
Eurostat
Key areas
Modernisation: ESS Vision exploiting the potential on new data sources establishing alliances and partnership with data owners new IT tools and methodological development organisational changes improving existing data collection methods
Eurostat
Eight enabling projects + supporting frameworks
Supporting Frameworks: Enterprise Architecture Quality Cooperation Models
Eurostat
What is the BIGD project? What?
Implement ESS Big Data Action Plan Why?
Enable the ESS to gradually integrate big data sources into the production of European and national statistics
How? Concrete Big Data Pilots In parallel work on horizontal issues
Three time ranges in roadmap Short term 2015-2016
Analysis of legislation, Strategy, Ethics, Communication Medium term 2016-2020
Pilots, Partnerships, IT Architecture, Skills Long term >2020
Full integration into official statistics (?!)
Eurostat
and the ESSnet Big Data?
Supports ESS Big Data Action Plan / BIGD project Preparation starts 2015 based on FPA/SGA construction
Core of work: 3+3 pilots 2016-2017 / 2017-2018 learning by doing
Exploration of specific data sources e.g. mobile communication data, social media, internet sites, .
Issue of multisource Statistics Contributes to generic frameworks and guidelines
e.g. methodology, quality, legislation, IT, communication, architecture,
Eurostat
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics / Communication
Partnerships
Pilots
Action Plan Themes
Eurostat
Experience sharing
Big Data project Approach
Policy Legislation
Quality Skills Methodology IT Infrastructure
Ethics / Communication
P
I
L
O
T
S
Eurostat
Actions Official Statistics Big Data Strategy
Roadmap and Action Plan
European Commission Communication "Towards a thriving data driven economy"
Private Public Partnership on big data
Data4Policy Initiative
Data Revolution at UN level
Challenges Datafication Impact on Official Statistics Answer of Statistical System Reaction of Government
10
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Eurostat
Actions Big Data ESSnet: Pilot projects
2015 2019 (FPA/SGA construction) Exploring different big data sources
Characteristics of data, potential outputs, Methodology and Quality, IT requirements
Partnerships with data providers, academia, users Cooperation with UN
Big Data Competition Big Data Workshop in Oct 2016
Challenges Exploration & tentative implementation Scientific Approach Cooperation, sharing of know-how Access to data, lack of skills, new user
demands
11
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Pilots Communication
Mobile Communication
Social Media
WWW
Web Searches
Analysis of websites' contents Job
Advertisements Businesses'
Websites
E-Commerce Real estate
Internet Traffic
Sensors
Traffic loops
Smart meters
Automatic Vessel
Identification System
Satellite Images
Webcams
Process generated data
Reservation Systems Flight Booking
transactions
Trains Hotels
Supermarket Cashier Data
Loyalty Cards
Financial transactions
Mobile Payments
eGovernment
Crowd sourcing
Voluntary Geographic Information
websites (OpenStreetMap)
Voluntary Information
websites (Wikipedia)
Community pictures collection
Eurostat
List of pilot projects (Specific Grant Agreement) Web scraping
job vacancies ; enterprise characteristics
Smart meters electricity consumption ; temporary vacant dwellings
Automatic Identification System (Ships) vessel identification data
Mobile phone data Preparing for Access to data
Scenario for using multiple inputs
13
ESS Big Data Pilots
Eurostat
Actions Cooperation with UN on quality and methodological framework
for big data Transversal topics in ESSnet pilots projects Generalisation to frameworks
Challenges Transversal challenges to all big data
activities: quality, methodology Multiple sources for multiple outputs Sound methodology ("from design-
based to model-based approach") Big data vs. statistics : "goodness of
fit" (concepts, representativeness,)
14
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Eurostat
Actions Training Strategy
Competency based approach Program for European statisticians (ESTP)
In the next years: dedicated courses on big data Focus on big data sources and on big data tools Acquiring the skills needed to assess sources and their
quality, the skills to use tools and to explore big data sources
Challenges New skills for staff: statisticians
vs. data scientists ? Relations with external data
providers Fast changing conditions and
situations
15
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Eurostat
ESTP courses supporting big data (2016)
16
Introduction to big data and its
tools
Hands-on immersion on big
data tools
Big data sources - Web, Social media and text analytics
Advanced big data sources - Mobile phone and other
sensors
Big data courses
Can a statistician become a data
scientist?
The use of R in official statistics:
model based estimates
Time-series econometrics
Methodology courses
Nowcasting
Activity
29 Feb 2 Mar 21 24 Jun
7 10 Nov
12 15 Sep
5 7 Apr 8 10 Jun 24 26 Feb
Eurostat
Actions Contract
2016-2017 (22 months) Selected big data sources
Analysis of legislative situation in EU Review of ethical principles, Elaboration of Communication Strategy See also the Feasibility study on the use of mobile positioning data for tourism
statistics (report on feasibility of access)
Challenges Access to data & continuity of
access Data security & privacy concerns Use of data only for statistical
purposes Impact on the public opinion of
privacy and security concerns ?
17
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Eurostat
Actions Sandbox IT infrastructure for experimenting
UNECE, Eurostat, EU
New Software (Hadoop, Pig, Python, ) Pilots for gaining experience
Challenges Computing capacity, hardware ? Analytical tools, software? Storage ?
18
Policy Quality Skills
Experience sharing Legislation
IT Infrastructures
Methods Ethics /
Communication
Partnerships
Pilots
Eurostat
Thank you for your Attention!
To be honest, Mr Wirthmann, I don't care how many yottabytes you have
Big Data,can you tell me where I want to gofrom how? to deliveringBig data at EurostatModernisation: ESS VisionNmero de diapositiva 5What is the BIGD project? and the ESSnet Big Data?Nmero de diapositiva 8Big Data project ApproachNmero de diapositiva 10Nmero de diapositiva 11PilotsESS Big Data PilotsNmero de diapositiva 14Nmero de diapositiva 15ESTP courses supporting big data (2016)Nmero de diapositiva 17Nmero de diapositiva 18Thank you for your Attention!