Met a-data Resources in Europe: within NSIs and from Dosis Projects

Meta-data Resources in Europe: within NSIs and from

Dosis Projects

Wilfried Grossmann

Department of Statistics and Decision Support Systems

University Vienna

29.3.2000 Metadata Resources in Europe 2

Contents

Introduction

Contents of Meta-data

IT- Structures for Meta-data

Processing Meta-data

Conclusions


Introduction

Continuing hot topics in the meta-data discussion

Content-orientation versus IT-orientation

There is a lack of communication between these two groups


Introduction

Meta-data providers versus meta-data users

Who provides which type of information for whom?



What kind of objects should be documented?

Basic statistical structures Variables Values Data sets

____________________

Statistical output Statistical Systems Statistical Processing



Approaches towards meta-data content

The template oriented approach

The data warehouse approach

The process oriented approach


Contents of Meta-data The template oriented approach

Templates defined by a number of working groups

For micro data and data setsDDI, Dublin Core

For (economic) macrodata

OECD, IMF, ECE (Internet)


Contents of Meta-dataThe template oriented approach

The OECD Template:

Concepts and sources

Data Collection

Data manipulation by national source

Data quality

Data Transmission

International Standards

Data Storage and Manipulation by OECD

Output preparation and delivery by OECD



The IMF Template:

Coverage

Periodicity

Timeliness

Quality of disseminated data

Integrity of disseminated data

Access by the public



Although the OECD approach seems more

reliable from statistical point of view, IMF is

favoured at the moment by international

organisations (EUROSTAT)


Contents of Meta-dataThe warehouse approach

Integration of the data inside the NSIs in a data warehouse

Output and dissemination as first step

Meta-data are oriented towards the needs of the

data warehouse


Contents of Meta-dataThe warehouse approach

Projects in this direction in many NSI

Best documentation: Australian Office

Definitional meta-data

Procedural meta-data

Operational meta-data

Systems meta-data

Datasets meta-data


Contents of Meta-dataThe process oriented approach

Combines statistical and IT considerations

Statistical data are considered not as final products but as the result of a process chain

More detailed consideration of statistical terminology



Starting point was the SCB-DOC model

(Rosen and Sundgren, 1991)

• A sequence of templates accompanying the statistical production process

• Ongoing activities at Statistics Sweden

• A number of NSIs want to adopt the model



The IDARESA model

Object oriented representation based on

SCB-DOC with emphasis on possible semi-automatic processing



The US-Bureau of census model

(Gillman, Appel et al. running project):

Statistical system defined as an identifiable process .... to produce one or more deliverables


Contents of Meta-dataSummary

Process oriented approach seems

to be favourable for a number of reasons

Two Examples:

Classification servers

Data Quality


Contents of Meta-dataSummary: Classification server

A classification server should

Support unified use of terminology inside NSIs or international organisations

Support harmonisation between (international) standard classifications and locally defined (adapted) classifications


Contents of Meta-data Summary: Classification server

Requirements for a classification server

• A data base supporting easy and user friendly manipulation of hierarchy trees

• A mapping tool supporting the definition of correspondence tables between classifications

• A management strategy for implementation


Contents of Meta-data Summary: Classification server

Up to now only few successful implementations

for partial solutions

EUROSTAT (SIMONE-Server)

New Zealand,


Contents of Meta-data Summary: Data Quality

Data Quality Criteria for quality of statistics are well known

(Relevance, accuracy, timeliness, accessibility, comparability, coherence, completeness)

The problem

• Achieve quality in the production process

• Document quality by appropriate meta-data


Contents of Meta-data Summary: Data Quality

Experience shows that documentation

quality is rather poor as soon as it is

separated from the production process

Example for an integration project

SIDI-approach by ISTAT


IT Structures for Meta-data

Internet and data warehouse offer new opportunities for

Meta-data and data repositories

Meta-data access and exchange

Lead towards a more open policy in data dissemination


IT Structures for Meta-dataMeta-data repositories

Approaches towards repositories

The thesaurus approach

The template oriented approach

The Data Warehouse oriented approach



Example for a thesaurus oriented approach

EUROSTAT servers for concepts and

definitions

• Advantage: available on the Internet

• Problem: Navigation not so easy



• Contents

– Descriptions (dictionaries)

– Semantic (coverage, standard classifications coherence of information)

– Administration (responsible persons)

– Selection (keywords, search facilities)



Example for the template oriented approach

StatBase: supporting access to meta-data as well as data and reports

• Meets quite well the requirements of OECD data template

• No direct connection between data and meta-data



Example for the warehouse oriented approach

StatLine(CBS): Based on data access from multidimensional tables (cubes)

• Accompanying meta-information is only in Dutch

• Extraction of special meta-data items is not so easy as in StatBase


IT Structures for Meta-dataMeta-data access and exchange

Ongoing work in access and exchange

New Standards for access and exchange

Accessing distributed sources

Combination of information



Actual trends in standardization

• Traditional standards for data and meta-data exchange like GESMES or CLASET will probably switch to XML-platform.

• New standards from the Object Management Group (OMG)



Example MOF (Meta Object Facility)

– Extensible Framework for meta-data model definition

– Programming interface for storage and access of meta-data

– Integration facilities across domains

But note: This is a general approach for warehouses not necessarily tied with statistics



Example for Accessing and processing distributed sources

ADDSIA: Accessing and processing distributed sources for analysis purposes

• Minimum requirements for standardisation in advance

• Orientation towards statistical problems



Goal Data and meta-data are processed

together

<OldDataSets, OldMetadataSets>

<NewData, NewMetadata>



Advantages Reduction of documentation effort

More consistency in meta-data

Requirements Software tools supporting this view

Operational models for meta-data



Up to know only prototypes with emphasis

on different aspects of processing

The planning approach

The throughput approach

The transformation approach


Processing Meta-dataThe planning approach

Develop software tools (workbench) for setting up meta-data documentation

BRIDGE/IMIM: A desktop for planning surveys and statistical

production Meta-data generated in the planning phase are

managed by the system No data are processed


Processing Meta-dataThe planning approach

Improvement and adaptation of meta-data models for new tasks like quality and use of administrative sources

SIDI (Statistics Italy) Integration of quality in the statistical

production process Standardization of the production process


Processing Meta-dataThe throughput approach

Use as much meta-data as possible from OldMeta-data to obtain NewMeta-data

CBS (ongoing work):

Use BLAISE meta-data as input Produce StatLine meta-data as output


Processing Meta-dataThe transformation approach

Define meta-data algorithms for all types of data algorithms

Throughput meta-data Modified meta-data New meta-data Meta-data summarization



IDARESA project

Meta-data algorithms for elementary data base operations

ISMIS

Identification of added value in meta-data (new meta-data)

Pursuit of the production process inside EUROSTAT

41Metadata Resources in Europe29.3.2000


In p u t d a ta 1In p u t M eta -d a ta 1


In te rim d a ta 1In te rim M eta-d a ta 1


In te rim d a ta 4In te rim m eta-d a ta 4




In te rim d a ta 2In te rim M eta-d a ta2


In te rim d a ta 3In te rim M eta-d a ta 3

In te rim d a ta 5In te rim m eta-d a ta 5

O u tp u t d a taO u tp u t M eta-d a ta


Conclusions

Is there progress in meta-data research and development?

Yes, but rather slow because There is a lack of co-ordination in research

(Probably improved by a forthcoming meta-data working group)

There is an information gap between meta-data research groups and NSIs

NSIs seem to prefer their own solutions

Met a-data Resources in Europe: within NSIs and from Dosis Projects

Documents

Transcript of Met a-data Resources in Europe: within NSIs and from Dosis Projects