ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.
-
Upload
gavin-byrd -
Category
Documents
-
view
218 -
download
0
description
Transcript of ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.
![Page 1: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/1.jpg)
ESS-net DWH
ESSnet on microdata linking and data warehousing in statistical production
![Page 2: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/2.jpg)
ESS-net DWH 2
Background ESS-net
Challenges
Explaining the statistical data warehouse (S-DWH)
Elements of the S-DWH
- Business architecture
- GSBPM mapping
Meta data
Organisational aspects
Content
![Page 3: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/3.jpg)
ESS-net DWH 3
ESS-net coordinator:
Statistics Netherlands (CBS)
Co-partners:
Estonia, Italy, Lithuania, Portugal, Sweden, UK
Starting date:
4 October 2010
SGA 1: first year, till 3 October 2011
SGA 2: last 2 years, till 3 October 2013
ESSnet Partnership
![Page 4: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/4.jpg)
ESS-net DWH 4
Provide assistance in: the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture
Results in daily statistical practice: increase the efficiency of data processing
in statistical production systems maximize the reuse of already collected data
a 'data warehouse' approach to statistics
General Objectives ESSnet DWH
![Page 5: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/5.jpg)
ESS-net DWH 5
Conclusions Data Warehousing in statistics is ‘hot’
Metadata is found important…..but also often neglected !
S-DWH is very difficult to compare with common commercial DWH
Visiting NSIs has proven very effective for gathering information AND for sharing knowledge and expertise
Great need for knowledge & expertise
Start SGA2
![Page 6: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/6.jpg)
ESS-net DWH 6
Decrease of costs & administrative burden versusincrease of efficiency & flexibility
Rapidly changing demand for information:- growing need for more information on more topics- decreasing lifecycle of policymakers, quicker delivery
Disclosure of all kinds of new data sources Need for integrated production systems
Make optimal use of all available data sources (existing & new)
The Challenges
![Page 7: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/7.jpg)
ESS-net DWH 7
The Statistical Data Warehouse
A central ‘statistical data store’ for managingall available data of interest, regardles of its source, enabling the NSI to produce necessary information (= statistics !)
and to (re)use available data to create new data / new outputs.
A central data hub to connect and integrate all available data sources, supporting statistical production AND data collection processes by providing:
a detailed and correct overview/insight of all available data sources a framework for adequate data governance, including metadata management, confidentiality aspects and data authorisation flexible data storage and data exchange between processes access to registers sampling frames (BR, etc);
![Page 8: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/8.jpg)
ESS-net DWH 8
AggregateStatistics
AggregateStatistics
Microdata
Dataextracts
Dataextracts
Dataextracts
Dataset
Dataset
Dataset
Backbones(BR eg.)
Selectedsample
Selectedsample
Admin datasource
Admin datasource
BBsnapshots
Storage, combination
OutputsInput dataInput reference frame
Sta
ging
are
a
Wor
king
dat
a
Rules for generating samples etc.
Rules for updating BB
![Page 9: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/9.jpg)
ESS-net DWH 9
A system or set of integrated systems, designed to handle the processing of statistical data in the production of statistics, comprimising: technical facilities for storing and processing data, receiving data in and producing outputs in a flexible way rules for updating the sources for the DWH definitions necessary to achieve those samples / sources
The S-DWH is a concept that provides an architectural model of the statistical data flow, from data collection to statistical output
Explaining the S-DWH
![Page 10: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/10.jpg)
ESS-net DWH 10
The S-DWH Business Architecture
Conceptualisation of how to build up a S-DWH A common model for the total statistical process
and data flow Provide optimal organisation of all structured data,
enabling re-use, creation of new data etc. 4 Layers, covering all statistical activities
‒ Sources‒ Integration‒ Interpretation & Analysis‒ Data Access / Output
![Page 11: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/11.jpg)
ESS-net DWH 11
The layered architectureof the S-DWH, with focus on the data sources used in each layer
Specific for S-DWH
![Page 12: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/12.jpg)
ESS-net DWH 12
Use the GSBPM as common language to identify and locatethe various phases on the 4 S-DWH layers
Mapping the S-DWH on the GSBPM
![Page 13: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/13.jpg)
ESS-net DWH 13
The S-DWH is a logically coherent central data store, not necessarily one single physical unit.
Metadata is vital in the governance, satisfying 2 essential needs:
to guide statisticians in processing and controlling the statistical data
to inform users by giving insight in the exact meaningof the statistical data
The vertical metadata layer enables to search all (meta)data in the 4 layers and, if permitted, give access to the data.
Managing the S-DWH
![Page 14: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/14.jpg)
ESS-net DWH 14
Meta data layer
Source Layer
Integration Layer
Interpretation and Data Analysis Layer
Data Access Layer
![Page 15: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/15.jpg)
ESS-net DWH 15
Framework:
General metadata definitions
Metadata for the S-DWH
Use of metadata models
Metadata standards & norms
Metadata quality & governance
Categories & subsets
Minimum requirements
Metadata - the DNA of the S-DWH
![Page 16: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/16.jpg)
ESS-net DWH 16
S-DWH meta data requirements
Subsets Standards & Norms
ISO 11179
Internal rulesGuidelines
Mata data model S-DWH Gatekeeper
![Page 17: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/17.jpg)
ESS-net DWH 17
Defining and implementing business modell:
Organisational aspects
- Experts from partners and other ESS members
- Research on actual topics
- Seminar / workshop
Financial aspects covered
Roll out for more fields of expertise
Centre of knowledge & expertise
![Page 18: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/18.jpg)
ESS-net DWH 18
Implementation of a S-DWH has huge organisational impact:
It means: moving from single operations to integrated, generic processes
It needs: a redesign of the statistical process
It asks: new IT systems, tools, high investments
It is: a new way of working
Only changing systems will not do the trick,changing people is the key to success
Organisational aspects
![Page 19: ESS-net DWH ESSnet on microdata linking and data warehousing in statistical production.](https://reader033.fdocuments.us/reader033/viewer/2022051319/5a4d1b5b7f8b9ab0599ab25b/html5/thumbnails/19.jpg)
ESS-net DWH
Thank you !
ESSnet on data warehousing