Open Data Trentino - Seminar at Universidad Simon Bolivar - 15th October 2013

11/04/231 Lorenzino Vaccari, Juan Panehttp://dati.trentino.it

Open Government DataSeminar @USB*

*This presentation is taken from the “Open Government Data Tutorial” presented at CLEI2013

Lorenzino Vaccari1, Juan Pane2

1Autonomous Province of Trento, Trento, Italy [email protected]

2University of Trento, Trento, Italy – Universidad Nacional de Asuncion, Asuncion, Paraguay [email protected]

11/04/23 Lorenzino Vaccari, Juan Pane2

Goal of the Seminar• Introduce Open Government Data

• Intro, Issues (Part 1)

• If you need it, how can you organize it?• Real experience (Part 2)

• Methods for opening data• Applications (Part 3)• Semantic Issues (Part 4)

11/04/23 Lorenzino Vaccari, Juan Pane3 15/10/2013Juan Pane, Lorenzino Vaccari3http://www.point-fort.com/index.php?2012/01/25/805-why-how-what

http://www.point-fort.com/index.php?2012/01/25/805-why-how-what


What?

“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to the requirement to attribute and

sharealike.” *

*(Source: )

http://www.opendefinition.org


usereuse

“open” = redistributioncommercial reusederivative works

BUT, may require:- attribution- share alike

http://myfbcovers.com/uploads/covers/2012/06/09/16628a1094aa012f7c6e0025902480d2/watermarked_cover.jpg

J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how


The value is in its use

11/04/23 Lorenzino Vaccari, Juan Pane7 Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q


Is open data useful?

Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q


Open Data Benefits The Open data are the knowledge base to:

Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information

Answer to social needs through the publication of innovative services and applications

Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)

Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities


Principles

Tim Berners-Lee (5-Stars of Linked Open Data)Vs.Tim Davis (5-Stars of Open Data Engagement)

http://5stardata.info/

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/


5 Starts Linked Open DataTim Berners-Lee

http://5stardata.info


5-Stars of Open Data Engagement

* Be demand driven * * Provide context * * * Support conversation * * * * Build capacity & skills* * * * * Collaborate with the community

Tim Davis

http://www.timdavies.org.uk/2012/01/21/5-stars-of-open-data-engagement/


Create Communityhttp://msnbcmedia.msn.com/j/MSNBC/Components/Photo/_new/pb-121007-spain-tarragona-pyramid-nj-02.photoblog900.jpg


Open Government Data


State of the ArtWhat is happening around us?-Globally-Europe-Latin America


Open Data Charter - G8The principles are:Open Data by DefaultQuality and QuantityUseable by AllReleasing Data for Improved GovernanceReleasing Data for Innovation

http://opensource.com/government/13/7/open-data-charter-g8

https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex

11/04/23 Lorenzino Vaccari, Juan Pane17http://opensource.com/government/13/7/open-data-charter-g8

http://census.okfn.org/

OGD around the world

11/04/23 Lorenzino Vaccari, Juan Pane18http://opensource.com/government/13/7/open-data-charter-g8

http://census.okfn.org/country/


OGD in Europe

http://open-data.europa.eu/


OGD in Europescreenshots

http://epsiplatform.eu/content/european-psi-scoreboard


OGD in EuropeInsert table

http://epsiplatform.eu/content/european-psi-scoreboard http://epsiplatform.eu/content/psi-scoreboard-indicator-list


OGD in Italy

http://www.dati.gov.it/content/infografica


OGD in Latin America*

*In Venezuela some OD projects have been started by the USB

11/04/2324 Lorenzino Vaccari, Juan Pane

Questions?

OGD: Part 2 - Issues

11/04/2325 Lorenzino Vaccari, Juan Pane 08/10/2013Juan Pane, Lorenzino Vaccari25http://evian-thesource.com/kids-having-fun/http://evian-thesource.com/kids-having-fun/


Open Data. Oh ohh

08/10/2013Juan Pane, Lorenzino Vaccari26

LegalLegalOrganizationalOrganizational TechnicalTechnicalAdoptionAdoptionBarriersBarriers

ContextualContextual

http://www.wallpapermania.eu/wallpaper/trick-or-treat-cute-pumpkins-lanterns-halloween-wallpaper

11/04/23 Lorenzino Vaccari, Juan Pane27http://de.straba.us/wp-content/uploads/2012/08/barrieres_for_implementation_of_ogd.png


Organizational Barriers

Not readyLack of resources

ITHuman

Don’t want to be ready

http://montcomediation.org/images/MCMC_MyWayYourWay.jpg


Legal barriersOpen the Data

All the data that was produced using public money has to be made publicly available (with exceptions)

vs PrivacyYou cannot open data that could allow

correlation of private personal data

Or the complete lack of legislation!


Adoption barriersData is not contextualizedPeople are not informedOpening data is a complex task, opening

cleaned data is even more complex.Unclear licenses


Technical BarriersAccess to data:

OrganizationalTechnical, Downtimes, logins, Payment fees

Fragmentation, incomplete data, scattered

FormatCataloging, indexing, searchLack of explicit semantics, metadataData is not reliableConflicting standards, models,

ontologies


BarriersZuiderwijk et al 2010

Listed 118 socio-technical impediments for opening data in the literature.FindabilityUsabilityUnderstandablityQualityLinkingComparability and compatibilityMetadata….

http://www.ejeg.com/issue/download.html?idArticle=255


Context Barriers

Privileged access to dataOther companies what to avoid legislation

of privacy.Transparency is bad for fraudulent business

http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg

11/04/23 Lorenzino Vaccari, Juan Pane34http://netdna.webdesignerdepot.com/uploads/photo_manipulation/manipulation-9.jpg


Preguntas?

Part 3 - Real Experience

11/04/23 Lorenzino Vaccari, Juan Pane36http://goo.gl/T2Xp80


The “Open Data in Trentino” project

• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model

• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity


Workplan - Steps


Nome (Acronimo) Descrizione

Tipo di Dato Estensione del file

Comma Separated Value (CSV) Formato testuale per l'interscambio testuale di tabelle, le cui righe corrispondono a linee e i cui valori delle singole colonne sono separati da una virgola (o punto e virgola)

Dato tabellare .csv

Geographic Markup Language (GML) Formato XML utile allo scambio di dati territoriali di tipo vettoriale

Dato geografico vettoriale

.gml

Keyhole Markup Language (KML) Formato basato su XML creato per gestire dati territoriali in tre dimensioni nei programmi Google Earth, Google Maps


.kml

Open Document Format (ODF) Formato per l'archiviazione e lo scambio di documenti di testo, fogli di calcolo, diagrammi e presentazioni

Dato tabellare .odc

Resource Description Framework (RDF) Basato su XML, e' lo strumento base proposto da World Wide Web Consortium (W3C) per la codifica, lo scambio e il riutilizzo di metadati strutturati e consente l'interoperabilità tra applicazioni che si scambiano informazioni sul Web

Dato strutturato .rdf

ESRI Shapefile (SHP) Lo Shapefile ESRI è un popolare formato vettoriale per sistemi informativi geografici. Il dato geografico viene distribuito normalmente attraverso tre o quattro files (se indicato il sistema di riferimento delle coordinate). Il formato è stato rilasciato da ESRI come formato (quasi) aperto


.shp, .shx, .dbf,

.prj

Extensible Markup Language (XML) E' un formato di markup, ovvero basato su un meccanismo che consente di definire e controllare il significato degli elementi contenuti in un documento o in un testo attraverso delle etichette (markup)

Dato strutturato .xml


…MeteoMeteo GeoDatiGeoDati StatisticaStatistica Comune

TrentoComuneTrento TrasportiTrasporti Etc…Etc……

Tecnological platform


Catalog

The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.

(2004)

http://okfn.org


http://dati.trentino.it*

Analysis: http://dati.trentino.it/stats Admin: http://dati.trentino.it/admin Harvesting: http://dati.trentino.it/harvest

* Available for all the data providers of Trentino


Services


Also Trentino is going to launch a challenge to build software applications and creative products (multimedia, audiovisual products, posters, illustrations) based on the datasets published on the http://dati.trentino.it open data catalog.

#ODTChallenge will be the official hashtag for our first open data challenge in Trentino!


7 months until now68.555 visits 7.988 unique visits2.516 downloads

37,36% returning visitors

62,64% new visitors

NOW- ALL the departmnets demand to be involved- Plus other local actors

AgricultureCultureGeographical DataWelfareWeather ForecastSocial policiesStatisticsTransports…MUNICIPALITY OF TRENTO, and

INFORMATICA TRENTINA

567 datasetsprovided by 10 departments of PAT…

20 reporting errors15 asking for new data10 new suggestions6 OD Applications

100% ENTHUSIASTIC REACTIONS


Want to Know & Learn more?

11/04/23 Lorenzino Vaccari, Juan Pane48http://www.theodi.org/

11/04/23 Lorenzino Vaccari, Juan Pane49http://schoolofdata.org/

11/04/23 Lorenzino Vaccari, Juan Pane50http://opendatahandbook.org/pt_BR/

11/04/23 Lorenzino Vaccari, Juan Pane51 http://www.od4d.org/category/open-data/how-to/

11/04/23 Lorenzino Vaccari, Juan Pane52http://schoolofdata.org/online-resources/


Thanks to the project team !!!!• General Manager: Isabella Bressan

• Project coordinator: Lorenzino Vaccari• Organizational/Communication issues: Francesca Gleria,

Roberto Cibin • Data gatherer: Luca Paolazzi • Catalog: Maurizio Napolitano, Samuele Santi• Semantics: Juan Pane, David Leoni, Alberto Zanella• Legal issues: Eleonora Bassi, Stefano Leucci• Communities: Maurizio Napolitano, Francesca De Chiara• System integration: Marco Combetto, Lorenzo Dallapè• Statistical Linked Data: Pavel Shvaiko


Questions?

OGD: Part 4 - Applications


Apps4Italy


Best Application: http://parlamento17.openpolis.it/


Open Bilancio

Best Idea: http://opendata.comune.fi.it/open_bilancio/


What?

DAL America Latina (2012): http://desarrollandoamerica.org/aplicaciones-2012/

DAL America Latina (2013): http://2013.desarrollandoamerica.org/appschallenge/


http://limaio.innovacion.pe/ http://www.limaio.com/demo

11/04/23 Lorenzino Vaccari, Juan Pane60http://www.mysociety.org/2007/more-travel-maps/morehousing


Johann MITTHEISZ (CIO der Stadt Wien)

http://www.slideshare.net/BrigitteLutz/keynote-mittheisz-cio-stadt-wien/16

Total hours to develop 38 applications:around 2.600

City of Wien saved around 208.000 Euro


The Open Data Ecosystem(and the OpenStreetMap case)


OpenStreetMap

~

OpenStreetMap project creates and provides geographical data, such as road maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use or availability of map information across much of the world and the advent of inexpensive portable satellite navigation devices.

OpenStreetMap is a free map of theworld, created by someone like you

11/04/23 Lorenzino Vaccari, Juan Pane65http://tools.geofabrik.de/mc/?mt0=mapnik&mt1=googlemap&lon=11.12042&lat=46.07224&zoom=18

11/04/23 Lorenzino Vaccari, Juan Pane66http://haiti.ushahidi.com


Watercolor maps

http://content.stamen.com/files/cartography/index_watercolor.html#18.00/46.07204/11.12097


From maps to blankets…

http://softcities.net


Sharing Data Globally(the eHabitat example)


21th Century Challenges

Source: http://www.slideshare.net/angeled/geoss © GEO secretariat


The Group of Earth Observation

Source: http://www.slideshare.net/angeled/geoss © GEO secretariat84 GEO members and 61 Participating organizations


GEOSS Data Sharing Principles

• Full and Open Exchange of Data, recognizing Relevant International Instruments and National Policies

• Data and Products at Minimum Time delay and Minimum Cost

• Free of Charge or minimal Cost for Research and Education

http://www.geoportal.org/web/guest/geo_home


“Venezuela is considered a state with extremely high biodiversity, with habitats ranging from the Andes mountains in the west to the Amazon Basin rainforest in the south, via extensive llanos plains and Caribbean coast in the center and the Orinoco River Delta in the east."

Source: Wikipedia


GEOSS for biodiversity

http://www.eurogeoss-broker.eu/


The eHabitat Model

http://ehabitat-wps.jrc.ec.europa.eu/ehabitat/


Questions?

OGD: Part 5 - Semantics


Available

Structured

Open formats

Redefenceable

Linked

Linked Open Data

The best data is an open data

Vs.

All data must be perfect


Lack of explicit semanticsThe real meaning of the data was kept in the developers mind when creating the data

78http://goo.gl/npEHKr


Lack of explicit semanticsCan lead to things like:


Semantic heterogeneityDifference in the meaning of local data


Issues when Opening Trentino Data

Each department has authority on only some part of the data.

Dataset originally created for internal use only.Dataset created for a specific need.Dataset created with custom format:

For structure (some exceptions)For data

Lack of reuse -> duplication.Lack of programmers.We cannot TELL them what/how to do (always).Data changes


Available

Structured

Open formats

Redefenceable

Linked

Entity CentricSemantic Layer

Data Catalog

Data Catalog


Entity centric: Added valueAggregated dataAccurate data, manually curatedUnique identifiers, distributed perspectives

Re-think identifiersSemantified values

E1

name Juan Pane

nationality italian

lives in Trento

affiliation Univ. Trento

E2

name Ignacio P. F.

born in Paraguay

date of birth 1980

affiliation PF-UNA


EntitiesReal world: is something that has a distinct,

separate existence, although it need not be a material (physical) existence. Has a set of properties, which evolve over time. Example:

Mental: personal (local) model created and maintained by a person that references and describes a real world entity.

Digital: capture the semantics of real world entities, provided by people.


Entity Centric Semantic Layer:• Address the integration problems due to

semantic heterogeneity:• Different formats• Different identifiers• Implicit semantics• Homonyms, synonyms, aliases• Partial knowledge• Knowledge evolution

http://www.webfoundation.org/2011/11/5-star-open-data-initiatives/


Entity-based Integration• Focus on entities as first class citizens

• Entities are objects which are so important in our everyday life to be referred with a name

• Each entity has its own metadata (e.g. name, latitude, longitude, …)• Each entity is in relation with many other entities (e.g. Einstein was

born in Ulm, his affiliation was Charles University, Ulm is a city in Germany)

• There are relatively “few” commonsense entity types (person, …, event)

• There are many domain specific entities (bus stops, cycling paths, ..)• All components have explicit semantics: schema, entities, attributes,

values


Importing pipeline, Macro Steps1. Domain analysis

Study the needed entity types, adapt the knowledge base accordingly. First time bootstrapping

2. Import entities Semi-automatic tool.

Domain experts are expensive. Human attention is a scarce resource. Incremental enrichment and aggregation of

entities.


Open Data PeculiaritiesAll data comes from a CKAN repository

(DCAT).Process one data file at a time.Each data file can be represented as a

table.Each row in the table represents a (partial)

entity.The format of the values might not be

enforced in the data files.Not all data is relevant.


Importing tool process


1. Source SelectionImport one data file at a time


2. Schema MatchingSelect a target type of entity -> correspondences between the input columns and the output attributes

nome provincia descrizione funivie lat long

Andalo (1047) Provincia di Trento

Sorge su un'ampia sella prativa al centro...

3 654463 712857

Canazei (1450) Trento Prov. Situato all'estremità settentrionale della...

2 511504 147444


3. Data ValidationApplies format and structure validation and possible automatic transformations needed to have the input data in the expected format.


4. Semantic Enrichment (1/2)Entity disambiguation: Transform text references into links to existing entities.


4. Semantic Enrichment (2/2)Natural Language Processing: Extract concepts and entity references from free-text.


5. ReconciliationRun Identity Management Algorithms to identify each row as a new or existing entity.

Result•No Match•Match•Multiple Matches

Action:•Use ID•New ID•Ignore Row


6. ExportingAt this point:We know what to export.All values for target attributes conform to the expected format.All text has been semantified (NLP).All textual references to entities are converted to linksEach row has an identifier

i i+1v0


7. PublishingPut back the semantified entities into CKAN so that the entities can be Open Data and can be found in the same catalog as the original data.Developers and find the data files of the cleaned, aggregated entitiesBut can also interact with the entities via the Entitypedia APIs

8. VisualizationSearch and Navigation


Semantic Layer: ServicesTool for aiding the “semantification” of the datasets in the catalog based on:

• Schema matching services• Identity Management services

• Entity Matching services• Global Unique Identifier services

• Semantic search and indexing services• Natural Language Processing• Entity store


Our Goal

TN

UK

BEES

11/04/23 Lorenzino Vaccari, Juan Pane100 http://www.youtube.com/watch?v=Bq_ZWl1ZXA0

BEYOND


Gracias!

Grazie!

Mercy!

Gràcies!Gratias!

Thanks!

Danke!

Dank u!

Kiitos!

ευχαριστώ

We thank in particular CLEI 2013, Autonomous Province of Trento, TrentoRise association, Universidad Nacional de Asuncion, Universidad Simon Bolivar and University of Trento

Lorenzino Vaccari1, Juan Pane2

1Autonomous Province of Trento, Trento, Italy [email protected]

2University of Trento, Trento, Italy – Universidad Nacional de Asuncion, Asuncion, Paraguay

[email protected]

Open Data Trentino - Seminar at Universidad Simon Bolivar - 15th October 2013

Technology

Transcript of Open Data Trentino - Seminar at Universidad Simon Bolivar - 15th October 2013