ESCWA SDMX Workshop

© Metadata Technology

ESCWA SDMX Workshop

Session: SDMX Introduction


What is SDMX?

• The problem space (2001):– Statistical collection, processing, and

exchange is time-consuming and resource-intensive

– Various international and national organisations have individual approaches for their constituencies

– Uncertainties about how to proceed with new technologies (XML, web services …)


What is SDMX?

The Statistical Data and Metadata Exchange (SDMX) initiative addresses these challenges and opportunities that have just been mentioned:– By focusing on business practices in the field

of statistical information– By identifying more efficient processes for

exchange and sharing of data and metadata using modern technology


Who is SDMX?

• SDMX is an initiative made up of seven international organizations:– Bank for International Settlements– European Central Bank– Eurostat – International Monetary Fund– Organisation for Economic Cooperation and

Development– United Nations– World Bank

• The initiative was launched in 2001


Official Recommendations

• SDMX has been officially recommended:– February 2007: SDMX endorsed by the

European Union’s Statistical Programme Committee (now ESS Committee)

– March 2008: UN Statistical Commission declares SDMX to be the preferred standard for data and metadata

SDMX: Benefits

• For Data Consumers– Search for, locate, and query data and related metadata– Query directly the database and metadata repositories of

the data and metadata producers– View data and related metadata in table, charts, graphs,

maps using SDMX visual components

• For Data Collectors– Share the data collection burden– Eliminate many and varied formats for data and reference metadata– Automate the database and metadata repository loading process to

assist validation

SDMX: Benefits

• For Data Reporters– Common reporting format (including coding schemes) for

all reporting needs– Ability to open databases and metadata repositories to

direct query for data collectors

• For Data Publishers– Automated web dissemination system for data and

reference metadata– As soon as data is published it is available for query, and

linked to the reference metadata


Exchange Patterns

• Bilateral: Institutions exchange data according to bilateral agreements regarding format, timing, protocols, etc.

• Gateway: Institutions share the data they collect with their peers, in agreed formats among counterparty communities

• Data-sharing: standard exchange of data using standard formats and protocols


Bilateral Exchange


Gateway Exchange


Data/Metadata-Sharing Exchange

Registration of Data Source

query/retrieval

RSS or SDMX Notification

Data/Metadata Reporters

Data/Metadata Consumers


Adopters/Interest• The following are known adopters (or planning to adopt):

– US Federal Reserve Board and Bank of New York– European Central Bank– Joint External Debt Hub (WB, IMF, OECD, BIS)– UN/TRADECOM at UN Statistical Division– NAAWE (National Accounts from OECD/Eurostat)– European Statistical System (Eurostat and National Statistical Institutes)– Mexican Federal System– IMF (BOP, SNA, SDDS/GDDS)– Food and Agriculture Organization– Millennium Development Goals (UN System, others)– International Labor Organization– Bank for International Settlements– OECD– World Bank– Marchioness Islands (Spanish/Portuguese Statistical Region)– UNESCO (Education)– Australian Bureau of Statistics– International Energy Agency– World Health Organization– There are many others


SDMX Status and Future

• Revised version 2.1 standards have been released (end April)– Increasing availability of SDMX tools

• In May 2011 the IMF and World Bank hosted a 3-day SDMX Global Conference– 250 people attended– Many implementation scenarios– Focus on web services– There was a vendor demonstration area on day 3

• Commitment of the Sponsors– 2011 – Web “Sandbox” for users to play with SDMX– 2012 - Global SDMX Registry

• Retrieval of common structural metadata by local registries• Sharing of structural metadata• Portal to find data and SDMX web services

– 2015 - 20 International Data Structure Definitions– World Bank and IMF asked to help with training and tools for

developing nations


SDMX and Domain Standards

• SDMX are keen that organisations prominent in a specific domain take the lead to define the domain standards– Structural definitions

• Code lists• Concepts• Data structure definitions• Metadata structure definitions

– Quality metadata and quality frameworks• Based on the SDMX content oriented guidelines


Dataor Metadata

Structure Definition

SDMX Version 2.0/2.1: High Level Overview

Category Scheme

CategoryData or Metadata

Flow

Data Provider

Provision Agreement

Data Set or Metadata

Set

Content

Constraint

Structure and Item Scheme

Maps

Registered Data Source or

Metadata Source

Attachment

Constraint

Categorisation

Process


Dataor Metadata

Structure Definition

Data or Metadata

Flow

Data Provider

Registered Data Source or

Metadata Source

Register

Data or Metadata Source

references

Category Scheme

Category Categorisation

Provision Agreement

Data Provider Scheme

High Level Overview: Flow


Existing Websites Using SDMX

• BIS– http://stats.bis.org

• ECB– http://www.ecb.int/stats/exchange/eurofxref/html/index.en.html

– http://stats.ecb.europa.eu/stats/sdmx/visualisation/icp/dashboard/rc1/

Code available at (http://flex-cb.googlecode.com/ )

• OECD– http://stats.oecd.org/WBOS/index.aspx– http://stats.oecd.org/OECDregionalstatistics/


SDMX: Course Introduction

• At the end of the Sessions on SDMX you will have– Understood the scope of SDMX– Gained a high level understanding of the SDMX

Information Model– Know the key SDMX structures that can be used to

create, report, store, maintain, visualise, MDG data and structural metadata


SDMX: Course Introduction

• In order to do this you will learn about– SDMX Structural Metadata

• DSD and related Concepts and Code lists• MSD and related Concepts and Code lists• Category Scheme and Classification• Data flow and Metadata flow• Provision Agreement• Registration• Subscription and Notification• Registry• Database• Query

– SDMX Data and Metadata• Data set formats• Metadata set

– SDMX Web Services


Tools Used on the Course

SDMX Registry/Repository


Tools Demonstrated

SDMX Structure and Data Converter

SDMX Relational Database

SDMX Data Viewer


Now – The Hard Work

ESCWA SDMX Workshop

Documents

Transcript of ESCWA SDMX Workshop