Eurostat

THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION

Spatial information and maps in statistics

Eurostat

• Day 2 – afternoon

• INSPIRE Directive

• Spatial Data Infrastructure

• Open Standards / Web Services

• Metadata

Eurostat

http://inspire.ec.europa.eu/ INSPIRE Directive

Eurostat

European Directive: Infrastructure for Spatial Information in Europe

European Union (EU) Spatial Data Infrastructure (SDI)

sharing of environmental spatial information among public sector organisations

to better facilitate public access to spatial information across Europe

addresses 34 spatial data themes needed for environmental applications

• Annex I: basic themes (e.g. administrative boundaries)

• Annex II: supporting themes (e.g. ortho-imagery)

• Annex III: other themes (population, energy resources, habitats, industrial and agricultural facilities,…)

INSPIRE Directive

Eurostat

INSPIRE Directive – Data themes

Eurostat

• Architecture suggested by INSPIRE:

• Interoperable services (web services)

• To produce, publish, discover, access and use geographical information

• Via the internet

• For the whole of Europe

• On local, national and European level

INSPIRE Directive

Eurostat

• Based on a number of common principles: • Data should be collected only once and kept where it can be maintained most

effectively

• It should be possible to combine seamless spatial information from different

sources across Europe and share it with many users and applications

• It should be possible for information collected at one level/scale to be shared

with all levels/scales; detailed for thorough investigations, general for

strategic purposes

• Geographic information needed for good governance at all levels should be

readily and transparently available

• Easy to find what geo-information is available, how it can be used to meet a

particular need, and under which conditions it can be acquired and used

INSPIRE Directive

Eurostat

• Adoption

• Preparation phase (2004-2006) • "co-decision procedure”

• Preparation of implementing rules

The INSPIRE Directive entered into force on the 15th May 2007

• Transition phase (2007-2008) • The European directive is operational

• Continuation of preparation implementing rules

• Conversion to national/regional legislation

• Implementation phase (2009-2021) • Phased implementation of rules

INSPIRE Directive

Eurostat

Eurostat

• Implementing Rules

• Accepted by EC and published in EU statutes (binding!)

• Supported by "Guidance document", with real technical specifications

• Available for: 1. Metadata

2. Data Specifications

3. Network Services

4. Data and Service Sharing

5. Monitoring & reporting

6. Spatial Data Services (new)

Technical

Organisational/Policy

INSPIRE Directive

Eurostat

INSPIRE Directive - Components

1. Metadata

• Data about data – information that describes geographical data and services

• To discover data and evaluate use

2. Data Specifications

• Interoperability: Consistent combination of geographical information and services from different

sources

Give access to geographical data via netwerk services

• Harmonization: Adapt and align definitions of spatial objects and models and the way they are

encoded, defined in data specifications in specific domain or application area defined by data model

Eurostat

3. Network services • Discovery services

Searching for geographical data and services based on content of their metadata, showing the metadata

• View services Visualize, navigate, zoom in/out, overlay of geographical data via maps, show

legend, show accompanying metadata

• Download services Enable to download part or whole data sets or enable to question/query data

set directly

• Transformation network services Transform data sets to reach interoperability

e.g. coordinate transformation, data model transformation

INSPIRE Directive - Components

Eurostat

4. Data and Service sharing • Access to geographical data and services under harmonized

conditions

Scope:

By member states and their administrations

By agencies of the EC

To execute tasks for the common interest

Where these tasks have an impact on the environment

INSPIRE Directive - Components

Eurostat

5. Monitoring and reporting

• Followup member states in their INSPIRE implementation and how it is converted into national legislation

• Monitoring of a set of indicators (measurable)

• Reporting on qualitative information about national SDI’s

6. Spatial Data Services

• A more general class of services than the “Network Services”

• ‘Free’

INSPIRE Directive - Components

Eurostat

EU community geoportal and the Member States shall provide access to their infrastructures through the geoportal as well as through any access points they themselves decide to operate

INSPIRE Directive - GeoPortal

http://inspire-geoportal.ec.europa.eu/

Eurostat

• Day 2 – afternoon

• INSPIRE Directive

• Spatial Data Infrastructure

• Open Standards / Web Services

• Metadata

Eurostat

SDI: Why?

Geographical information is different from other information

• Multifunctional and domain crossing Example: Parcel (source: commune, Federal Public Service Finance, different

application domains)

• Complex compared with other data Structure: geometry, topology

Quality: accuracy, precision, scale dependency

• Big data and expensive Images, raster data

Data collection requires huge effort and expertise

• Competences needed for maintenance of geodata Specialized software – powerful hardware

IT infrastructure is different from "standard" IT-infrastructure

Eurostat

SDI: why?

Evolution of the SDI

• From: “Closed" GIS for small application domain Software-own databases and format

Difficult: only for GI-experts

Data only for dedicated goals – several data gathering

“Island GIS" – not integrated in company processes and –architecture

• To: "Open" and modular SDI Follows overall IT-trend

Based on services and open standards

Maintaining geodata at the source

Using geodata in as many application domains as possible

Offer geodata as ‘geographical services’

Eurostat

SDI: principles

Basic principles of an SDI

• Gather once Gather once and maintain where most efficient (at the source)

• Seamless Combine data from different sources seamlessly

Use for wide range of applications

• Share Gathered by one organization

Used by many

• Instantly available and transparent Easy to discover and evaluate

Terms of use are simple and transparent

Eurostat

SDI: principles

Advantages

• Enhanced creation, maintenance and use of geographical data

• Organizations enhance their processes and efficiency

• Organizations get more value for their investments

• Users (intern/extern) receive better services

• Lot of new application possibilities

Eurostat

SDI: principles

Example: real estate information (inlichtingenformulier vastgoedinformatie)

• Real estate/civilian/company wants to know status of property:

Size, location, destination, permits, environment, …

• Local governments gather information > takes more than 30 days!

Problems: finding the access to actual information in different sources, no automation, not simple for functionary

Eurostat

SDI: components

• Infrastructure

• Durable, supporting environment

• Can be compared with road- or telecom infrastructure

• Consists out of components that interact (interoperability):

Harmonized geodata

Metadata

Geographical services (web services)

Policy for geodata

Coordination and monitoring

More info: SDI Cookbook: http://www.gsdidocs.org/GSDIWiki/index.php?title=Main_Page

Eurostat

SDI: components

• Harmonized geodata

• Agreed, multi-functional data models and specifications

• Data exchangable between organizations

• Data can be combined between organizations, domains, countries, regions

Eurostat

SDI: components

• Harmonized geodata

• Example : Hydrographical network Multi-functional: navigation, maintanance waterways, drinking water,

inundation risks, recreation, infrastructures, …

Exchangeable : MOW, VMM, VMW, WatLab, Toerisme Vlaanderen, Waterwegen & Zeekanaal, de Scheepvaart, …

Combinable: spoiling or inundations in water basin: between countries (NL/BE), between regions (Flanders, Brussels, Wallonia), between government/private (industrial zones, insurance companies)

See INSPIRE Hydrografie – data specificaties

Eurostat

SDI: components

• Metadata

• Information that describes geographical data Inventarisation, discover and find, evaluation of use

• Standardized metadata model Everybody describes metadata based on the same set of elements

Title, goal, data, contact, location, use restrictions, availability …

To use in Catalogue applications

Eurostat

SDI: components

• Geographical services (web services)

• Discovery services

• Viewing services

• Download services

• Transformation services

• Central access via GeoPortal

> Services based on Open Standards!

Eurostat

SDI: components

• Policy for use/sharing geodata

• Transparent, clear and easy to apply rules to access and use geodata

• E-commerce

• Coordination and infrastructure

• Technical and organizational guidelines

• Monitoring and reporting

• Supporting environment

Eurostat

• Day 2 – afternoon

• INSPIRE Directive

• Spatial Data Infrastructure

• Open Standards / Web Services

• Metadata

Eurostat

Online mapping tools

But how and where are these data coming from?

Eurostat

What are open standards?

• Standard maintained by not-for-profit organisation with open decision making procedure

• Standard is published and the document is offered freely

• Intellectual property of the standard is provided irrevocable and royalty-free

• No constraints with respect to reusability of the standard

Source: European Interoperability Framework (EIF) published by EC IDABC Programme

Eurostat

What are open standards?

In-company Standards

Open Specifications

Free Specifications

Open Standards

GIF

Doc

PDF

RTF

XML TCP/IP

Source: Advies 86 “Open Source Software” Vlaamse Raad voor Wetenschapsbeleid

Eurostat

Open standards for geo information

• ISO TC 211 Geographic information/Geomatics www.iso.org

http://isotc.iso.org/livelink/livelink/open/tc211

• Open GeoSpatial Consortium (OGC) International organization

>350 active members (government, private, academic)

Mission : “deliver spatial interface specifications that are openly available for global use”

http://www.opengeospatial.org/

• Formal co-operation between ISO and OGC

Eurostat

La directive INSPIRE en pratique © GIM

Oracle Spatial

GeoTIFF/MrSID JP2000/ECW

Interoperability

Data

Client Software

DB xml

WFS WMS WCS CSW

WMS: maps WFS: features WCS: coverages SOS:Observations CSW: catalog

PostGIS PostGreSQL

SOS

MIF/MID Shape/GML

WWW

Data sources

Eurostat

Open standards for geo information

• Abstract specifications (19): Feature, ISO Metadata

• Implementation specifications (30): GML, WMS,...

• Specification Profiles: GML Simple Features

• Best Practices (20): GeoPDF

• Discussion papers (93): WCTS, WTS

• Deprecated documents (>100): Old Versions

• Requests for Information/comments

• Reference Model (2): Overview

• White papers

Eurostat

Open standards for geo information

OGC Implementation Specifications

•Technical and detailed

• Interface between software components ▶ Simple Feature (4 specs) (SFS)

▶ Geography Markup Language (GML)

▶ Web Map Service (WMS)

▶ Web Feature Service (WFS)

▶ Web Coverage Service (WCS)

▶ Web Map Context Documents (WMC)

▶ Filter Encoding (FES)

▶ Styled Layer Descriptor (SLD)

▶ Symbology Encoding (SE)

▶ Catalog Service (CAT - CSW2.0)

▶ Coordinate Transformation Services (CT)

▶ Grid Coverage Services (GC)

▶ Location Services (OpenLS)

▶ Geographic Objects (GO)

▶ Web Serice Common (Common)

▶ GML in JPEG2000 (GMLJP2)

▶ GeoSpatial eXtensible Access Control

Markup Language (GeoXACML)

▶ OGC netCDF encoding

Eurostat

Open standards for geo information

WMS – Web Map Service • The OGC WMS specification is an interface specification,

which allows the client software to access maps produced by the map services - in a standardized way.

• Different interactions:

• Q/A with respect to the offered map layers

• Question to offer map layer

• Simple requests

• It is possible to overlay different layers from different

"Map servers"

• Map= visual presentation of underlying data

Eurostat

Open standards for geo information

WFS – Web Feature Service • The OGC WFS specification is an interface specification, which allows the

client software to access the actual vector data to download and edit - in a standardized way.

• Different interactions :

• Which datalayers are available, which functionality does the server offer (read, transactions, …)

• What is the content of the specific data layer

• Execute a spatial query

• Transaction: edit objects

• Vector data are mostly offered in GML

Eurostat

Open standards for geo information

GML – Geography Markup Language (GML)

• XML language for describing vector data

• For geographic (spatial) component and for attribute data

• GML describes the data, not the presentation of the data

Eurostat

Open standards for geo information

CS-W – Catalog Services for the Web • The OGC CS-W specification is an interface specification, which allows the

client software to access catalogues of geographical data and services - in a standardized way.

• Searching for data and services based on metadata

• Catalogues can be linked:

• "Harvesting"

• Remote access

• Catalogues can be accessed via

• a GeoPortal (client application)

Eurostat

When to use open standards?

• In organizations with a wide variety on geographical

information platforms

ArcGIS, MapInfo, AutoCAD, modelling software, ETL, Oracle, …

• For applications that use geodata from different suppliers

• Real estate information, touristic applications …

• Organizations that offer geodata to a wide variety of users

and applications

• To build an SDI according to "Best Practices"

• To be conform with the INSPIRE directive, GDI-decreet,…

• To build future, sustainable information systems

Eurostat

Example: GeoServer • GeoServer is open source (and free)

• Provides geographic information in the form of interoperable services

• Popular software used by companies / governments as a basic building block for their SDI

• Advantages: Easy setup

Intuitive

Faster and more effective than its commercial counterparts

OGC compliant

Perfectly scalable

Future ensured with the support of a stable and growing community of users and developers

• Supports WMS, WCS, WFS, …

More info: http://geoserver.org/

How to provide/set up Web Services?

Eurostat

Open Standards in practice: “GIM GeoCMS”

• Client application which connects to OGC WMS, WFS, WFS and WCS Services

• Easy configuration of counters

• Based on a Content Management System

• Export of GML, KML, Shape, GeoTIFF, PDF

• Symbology using SLD

• Configuration based on WMC

• Chart functionality

• Editing

• Tested using the most popular servers

Eurostat

Open Standards in practice: “GIM GeoServices”

• Offer maps and online Geo-Web Services Web Map Services

- VHRSI

- TomTom Streetmaps

- …

Geocoding

Reverse geocoding

Address generalization

• Hit-based cost model (monitoring and control)

• Web application to monitor hits

Eurostat

• Day 2 – afternoon

• INSPIRE Directive

• Spatial Data Infrastructure

• Open Standards / Web Services

• Metadata

Eurostat

What are metadata

• Data about data - e.g.: data creation date

• what, who, where, when, how, why, …

• Data describing a series of geodata, a set of series of geodata or geodata service

• Allow research, inventory, and facilitates the use of geodata

• Within the organization and between different organizations

Eurostat

Standardisation organisms Metadata

• OGC (Open Geospatial Consortium)

• ISO (International Organisation for Standards) (ISO/TC 211)

• CEN (European Committee for Standardization) (CEN/TC 287)

• DWIWG (Digital Geographic Information Working Group)

NATO’s Digital Geographic Information Exchange Standards (DIGEST)

• FGDC (Federal Geographic Data Committee) (FGDC-STD-001-1998) -National Spatial Data Infrastructure (NSDI) in USA

• New Zealand Government

Geospatial metadata standard for New Zealand Government agencies

• DCMI (Dublin Core Metadata Initiative) (ISO 15836)

Cross-domain information resource description: limited content but widely usable

Eurostat

Metadata - ISO Series

• ISO 19106: Profiles

• ISO 19109: Rules for application schema

• ISO 19110: Methodology for feature cataloguing

• ISO 19113: Quality principles

• ISO 19115: Metadata

• ISO 19118: Encoding

• ISO 19119: Service Metadata

• ISO 19133: Location-based services (Tracking and navigation)

• ISO 19139: Metadata -XML schema implementation

GeoNetwork 2014_01 © GIM

Eurostat

ISO 19115

• ISO 19115: “norm” versus standard

• Some 400 mandatory, conditional or optional metadata sections, entities and elements

• Minimum set of metadata required to serve full range of metadata applications

• Method for extending metadata

• A community can use these to build its own standard (application profile) according to ISO 19139

GeoNetwork 2014_01 © GIM

Eurostat

ISO 19115 Contents • Introduction

• References

• Terms (not defined elsewhere in the ISO 191** series)

• UML models – including codelists

• Core metadata list

• Data Dictionary

• Descriptions of extensions and profiles

• Abstract test suite

• Dataset Metadata application profile

• Extension methodology (informative only)

• Metadata implementation (informative only)

• Hierarchical levels of metadata (informative only)

• Implementation examples (informative only)

• Multilingual support for free text metadata elements (informative only)

Eurostat

GeoNetwork 2014_01 © GIM

UML Package Entity (UML Classes)

Metadata entity set information MD_Metadata

Identification information MD_Identification

Constraint information MD_Constraints

Data quality information DQ_DataQuality

Maintenance information MD_MaintenanceInformation

Spatial representation information MD_SpatialRepresentation

Reference system information MD_ReferenceSystem

Content information MD_ContentInformation

Portrayal catalogue information MD_PortrayalCatalogueReference

Distribution information MD_Distribution

Metadata extension information MD_MetadataExtensionInformation

Application schema information MD_ApplicationSchemaInformation

Extent information EX_Extent

Citation and responsible party

information

CI_Citation

CI_ResponsibleParty

Eurostat

Example ISO 19115: MD_Identification

Eurostat

ISO 19115 Core Metadata for Geographic Datasets

• Mandatory and recommended optional elements

• Answers What, Where, When and Who questions

Eurostat

ISO 19115 Core Metadata for Geographic Datasets

Eurostat

GeoNetwork as Metadata Catalogue

• Web-based Catalogue software for metadata of geographic datasets, dataset series, Feature Catalogues, Services, applications, …

• Metadata catalogue forms the glue in a Spatial Data Infrastructure

• Central access points that allows to

Create Metadata and make it available – Publish

Discover Metadata – Find

Access Datasets, Services, … described by these metadata – Bind

• Facilitates the exchange of data between organizations and their audience

• Helps in reducing duplication of datasets

• Enhances information consistency and quality

Eurostat

Main Features End Users

• Instant search on local and distributed geospatial catalogues

• Search on multiple catalogues through one website

• Downloading of metadata data, documents, PDF's and any other content

• An interactive Web map viewer

Metadata Editors

• Online editing of metadata with a powerful template system

• Support for different metadata standards

• Data upload

Administrators

• Scheduled harvesting and synchronization of metadata between distributed catalogues

• Groups and users management

• Fine grained access control

• Active user and developer community

Eurostat

Searching metadata

• Default search

• Allows to perform a free text search, a search on the Resource Type and/or on the geographic location

(A) Free text search

• Type a search term in Search Field.

• Terms will be auto-suggested after typing 3 letters

• Perform Search

• Fields that are discoverable can be configured

(B) Resource types

• By clicking on the dropdown, a list with possible values will appear. You can choose one or more items from the list.

• The values that are supported are

Series for geographic dataset series (collections)

Service for geographic services

Dataset for geographic datasets

Eurostat

• Default search (C) Geographic search:

• A small overview map is shown on the screen that allows you to draw a Bounding Box

• Search options

• SortBy : Different ways to order the results • Relevance (based on lucene score)

• Title

• Metadata changeDate

• Rating

• Popularity

• Scale Descending

• Scale Ascending

• hitsPerpage: Number of results per parge (default = 20)

Searching metadata

Eurostat

• Advanced search • Works similar to the default search

• You can however be more specific in your search criteria as it offers different metadata elements to query upon.

• Search directly on the Keywords, organization or category (maps, applications, audio, video, …)

• Lists of possible values are determined on the basis of the contents of the Catalog

• By clicking on the arrows, lists of possible values will appear

• You can then select one or more items

• Remove previously selected entries by clicking on the “X”

Searching metadata

Eurostat

• Advanced search

• When – section: • Limits your search to a particular

temporal extent

• Either Anytime

• Either by using the metadata change date

• Either by using the temporal extent

• Spatial representation type: (e.g. grid, vector, tin,…)

• Scale: use the slider to enter the min and max scale denominator

• Catalogue: select the local and/or remote /harvested catalogue instances

Searching metadata

Eurostat

• Output of search:

• List of metadata records that match your request

• For each record the result page shows a summary of the metadata consisting by default of the title, the abstract, the thumbnails and the keywords and a set of icons

Searching metadata

Eurostat

• Analyzing search results

• Different Result views are offered • Title and keywords

• Title, thumbnail and keywords

• Title, abstract, thumbnail and keywords

• Results can be sorted using a variety of criteria

Searching metadata

Eurostat

• Analyzing search results

• Search results can be selected to allow operations (deletion, changing privileges, exporting, …) on multiple records at once

• (De)select one by one using checkbox

• Select all or all in the current page

• Possible actions on selected results Download & printing

Delete

Change

Adding new metadata

(Link to administration page)

Searching metadata

Eurostat

• Metadata description • After completing your search, you can view details by clicking on

the metadata title.

• Or by clicking view on the Action menu (appears on mouse over)

Searching metadata

Eurostat

• Metadata description

• Metadata descriptions can be viewed in different View modes : Simple, Default, By Group, By Package or XML

• Simple View Mode is organized in the following sections:

• Title and Abstract

• Information about the dataset

• Contact information

• Metadata contact

• Technical information

Searching metadata

Eurostat

• Metadata description

• The “Default” View mode is organized in 6 sections that are shown below each other and that can be opened and closed:

• Identification info

• Distribution info

• Reference system info

• Quality info

• Metadata

• Content Information

Searching metadata

Eurostat

• Metadata description

• The “INSPIRE” view is organized in 10 sections and addresses all the metadata elements relevant for INSPIRE

• Identification

• Classification of spatial data and services

• Keywords & INSPIRE themes

• Geographic location

• Temporal reference

• Quality and validity

• Conformity

• Constraints related to access and use

• Organizations responsible for the establishment, management, maintenance and distribution

• Metadata

• Needs to be enabled in the system configuration

Searching metadata

Eurostat

• Metadata description

• The “By Group” View is organized in 3 tabs:

• Iso Minimum has 2 sections

- Data identification

- Metadata

• Iso Core has 5 sections

- Data identification

- Lineage

- Reference system info

- Distribution info

- Metadata

• Iso All has all sections for which metadata is present

Searching metadata

Eurostat

• Metadata description

• “By Package” View mode • The “Per Package” is organized in 12 tabs

• One per ISO19115 package except for Extent and Citation and Responsible Party Information Packages

• The “XML view” shows the metadata in XML format

Searching metadata

Eurostat

• Actions on Metadata records • View: shows the metadata

• Zoom To: zooms to data on the map

• Save as XML: shows the metadata as XML

• Export (PDF): shows the metadata as pdf

• Export (MEF) : saves the metadata in zip format and includes the thumbnails

• Additional actions for administrators/editors • Edit: opens metadata record in edit mode

• Delete: removes metadata record

• Other actions:

- Duplicate: copy metadata record

- Create Child metadata record

- Privileges: assign specific rights to specific groups

- Status: change metadata status: draft, approved, submitted, retired, rejected, …. 69

Searching metadata

Eurostat

• To produce a good metadata record, always try to gather as much details as possible on the resource that you want to describe

• Login as administrator or editor (or reviewer)

• Mechanisms for creating a new metadata record - Add new metadata by copying it from a template

- Importing XML formatted metadata

- Duplicate existing metadata

- Bulk metadata import

- Harvesting

Editing metadata

Eurostat

• Metadata templates

• A template is a metadata record that is prefilled with default values

• GeoNetwork comes with templates for several metadata profiles including ISO19115/ISO19119/ISO19110/ ….

• Difference between a normal metadata record and a template can be set in the metadata editor by selecting the “record type”

• Administrators can create new templates that are generally usable

• Editors can create templates for their groups

• Note: to discover templates select kind = “template” in the search menu as by default templates are not shown

Editing metadata

Eurostat

• Metadata editing Views • Simple View: Simplified View that only shows metadata elements that

are present in the template.

Simple: user is not exposed to entire ISO Specification

User can not add metadata elements that are not present in the template

User can remove metadata elements and cannot add them back

• Advanced View: is based on metadata XSD

User has access to all metadata elements and can always add them

Complexity

• XML View

• INSPIRE View

• Custom Views: through development

Editing metadata

Eurostat

• Importing Metadata one XML file at the time • Go to the administration page

• Click on the “Metadata insert ” link

• Browse for a file or Copy/Paste

• Select File type

• Select options

- - apply XSLT stylesheet to transform metadata - UUID management - Assign to a group Assign to a category - validate or not

• Click “Insert”

Editing metadata

Eurostat

• Duplicate existing metadata

• Search and select metadata

• Click on Actions / otherActions / duplicate menu

• In the form that appears select a group

• Click on create

Editing metadata

Eurostat

• Metadata validation

• After Editing Save -and check – the newly created metadata.

• Checks verifies the conformance of the metadata record wrt

- ISO19139 schema

- ISO 19115/19115 schematrons

- GeoNetwork recommendation schematron

- INSPIRE schematron – if installed

- Custom schematrons to check additional constraints

• The validation report is given in the right column

• In case of validation errors correct them and run the check again

Editing metadata

Eurostat

• Day 2 – afternoon

• INSPIRE Directive

• Spatial Data Infrastructure

• Open Standards / Web Services

• Metadata

• Questions?

• Thanks for your attention!