BIG DATA EUROPE'S INTEGRATOR PLATFORM

A ONE-STOP SOLUTION FOR BIG AND SMART DATA MANAGEMENT

EuroPro Workshop @ EDBT/ICDT Conference21.03.2017

Supporting the Societal Domains with Big Data Technology

BigDataEurope Project

2 mai 2023www.big-data-europe.eu

BigDataEurope Action EC Horizon 2020 Coordination & Support

Action o ~5mio €, 2015-2017

Show societal value of Big Datao Across all societal challenges addressed by H2020

Lower barrier for using big data technologieso Effort to setup and deploy use-case workflowso Lack of skills & expertise

Help establish data value chains across domains & orgs. 2 mai 2023www.big-data-europe.eu

Consortium

NCSRDEMOKRITOS

Data Value Chain Evolution

2 mai 2023

Extraction, Curation Quality, Linking, Integration

Publication, Visualization, Analysis

Extraction, Curation, Quality, Linking, Integration, Publication,

Visualization, Analysis

HealthTransport

Security

Extraction Curation Quality Linking Integration Publication Visualization Analysis

Data Repositori

es

Linked Open Data

TIME

Food SocietiesClimate EnergyProprietary, ‘locked-in’solutions

OS Solutions,Big Data Stacks

www.big-data-europe.eu

A flexible, generic platform for (Big) Data Value Chain Deployment2. Architecture

Big Data Integrator

2 mai 2023www.big-data-europe.eu

Big Data Integrator: Architecture

Stacks Open Source solutions (Free)

Dockerization Facilitates integration and

deployment Plug-and-play BD Platform Key BDE additions

o Support layer: integrated UIo Semantification layer

Final BDI Release:o (3rd) May 2017] Final

2 mai 2023www.big-data-europe.eu

Big Data Integrator: In-Use

Big Data Integrator:https://github.com/big-data-europe

WIKI : extensive documentation, information on supported components, instructions, etc. 2 mai 2023www.big-data-europe.eu

BDE vs Hadoop distributions

Hortonworks Cloudera MapR Bigtop BDE

File System HDFS HDFS NFS HDFS HDFSInstallation Native Native Native Native lightweight

virtualizationPlug & play components (no rigid schema)

no no no no yes

High Availability Single failure recovery (yarn)

Single failure recovery (yarn)

Self healing, mult. failure rec.

Single failure recovery (yarn)

Multiple Failure recovery

Cost Commercial Commercial Commercial Free Free

Scaling Freemium Freemium Freemium Free FreeAddition of custom components

Not easy No No No Yes

Integration testing yes yes yes yes --Operating systems Linux Linux Linux Linux AllManagement tool Ambari Cloudera

managerMapR Control system

- Docker swarm UI+ Custom

9

Data Acquisition: Apache Flume Data Storage: Hue, Apache Cassandra, ScyllaDB,

Apache Hive, Postgis Search/Indexing: Apache Solr Message Passing: Apache Kafka Data Processing: Spark, Flink Semantic Components: Sansa, Silk, Strabon,

Sextant, GeoTriples, Semagrow, Limes, 4Store, Openlink Virtuoso

BDI Docker Containers (..and counting)

2 mai 2023www.big-data-europe.eu

Semantic Layer

www.big-data-europe.eu

Semantic Data Lakes o Minimal

ingestion pre-processing

o Semantic layer maintains metadata

o Add meaning when retrieving/processing

Data Lake: scalable unstructured data store

Relationship definitions and metadata

JSON-LD CSVW R2RMLXML2RDF

Ongoing Research for Semantic Big Data & Analytics Knowledge

Graphs

Semantic Layer tools

2 mai 2023www.big-data-europe.eu

BDE tooling for Semantic Data Lake:o Swagger: Semantics of RESTful

APIso Semantic Analytics Stack

(SANSA): Distributed data processing over large-scale Knowledge Graphs

o Semagrow: SPARQL over Big Data stores

o Ontario: Querying over Semantic Data Lakes

Demonstrating the Societal Value through 7 Pilot ‘Real-world’ use-cases

1. Overview

BigDataEurope Pilots

2 mai 2023www.big-data-europe.eu

7 Pilots◎ BDI Platform Instantiations

o Allow end-users to easily deploy functionality in own system environment

o Modularized Docker approach - easier to replace components

o Reduces effort to keep 3rd party software updated & integrated

◎ 7 Societal Challenge Pilots o Aligned with 7 European Commision H2020 Societal

Challengeso Real-world use-cases (Data, Objectives, Solutions)o Some pilots have different data & objectives but a similar

solution◎More info:

2 mai 2023www.big-data-europe.eu

www.big-data-europe.eu/pilot/

7 BDI Instances

2 mai 2023www.big-data-europe.eu

Free Workshops, Hangouts & WebinarsBigDataEurope Activities

2 mai 2023www.big-data-europe.eu

3rd round of Societal Workshops

2 mai 2023www.big-data-europe.eu

Health 11/12 May 2017 Malta Collocated with EC eHealth Week

Food&Agri

31 March 2017 Brussels

Co-organized with e-ROSA H2020 project

Energy Autumn 2017 Brussels

(Details to Follow)

Transport

12/13 September 2017

Collocated with Big Data for Transport, Tisa workshop, organised by Ertico

Climate 21/22 September 2017

Brussels

Collocated with iScape Project Workshop – Improving the Smart Control of Air Pollution in Europe by EC JRC Ispra.

Societies

Autumn 2017 T.B.C. (Details to Follow)

Security Autumn 2017 Brussels

Standalone Event

PLUS! Online

Hangouts & Technical Webinar

BIG DATA INTEGRATOR www.github.com/big-data-europe

SEVEN PILOT DESCRIPTIONSwww.big-data-europe.eu/pilot/

PROJECT COORDINATION (Fraunhofer IAIS)Prof. Sören Auer, auer © cs.uni-bonn · de > Dr. Simon Scerri, scerri © cs.uni-bonn · deEIS Department/Group,Fraunhofer IAIS & CS Department Uni-Bonn, Bonn, Germany

Questions & Contactswww.big-data-europe.eu

2 mai 2023#BigDataEurope

leads the Fraunhofer Big Data Alliance