Robert Isele | eccenca CorporateMemory - Semantically integrated Enterprise Data Lakes

WWW.LEDS-PROJEKT.DE

ECCENCA CORPORATE MEMORY

SEMANTICALLY INTEGRATED ENTERPRISE DATA LAKES

May 2, 20231

http://2016.semantics.cc/robert-isele

May 2, 20232

MOTIVATION

Enterprise Data Management Objective:“Ensure all data is aligned to a common meaning in order to achieve automation in performing complex analytics and generating trusted reports.”

Source: 2015 Data Management Industry Benchmark - EDM Council

In 2015 only 7% of respondents claim to already be using shared and unambiguous definitions of data across the firm and have it accessible as operational metadata.

7%

May 2, 20233

ARCHITECTURE

ManagementAccounting

Risk ManagementRegulatory Reporting

Treasury MarketingAccounting

Corporate Memory

Inbound

Data Sources

Outbound and Consumption

Inbound Raw Data Store

Knowledge Graph for Meta Data, KPI Definition and Data Models

Frontend to Access Relationship and KPI Definition / Documentation Frontend to Access (ad hoc) Reports Outbound Data Delivery to Target Systems

Big Data DWH-Infrastructure

ARCHITECTURE






Frontend to Access Relationship and KPI Definition / Documentation Frontend to Access (ad hoc) Reports Outbound Data Delivery to

Target Systems


Data Ingestion• Files in the data lake (CSV, XML, Excel)• (relational) Databases

ARCHITECTURE







Target Systems

Big Data

DWH-Infrastructure

Data Lake• Emerging approach to handle large amounts

of data• Cost-effective storage• Data is held in their native formats GoodDoes not force an up-front integration of the ingested data sets BadRetaining an overview of disparate data silos in the lake without having a coherent shared view is a challenging issue

ARCHITECTURE







Target Systems


Data Warehouses• Existing infrastucture• Typically relational databases

ARCHITECTURE







Target Systems


Metadata Layer• Dataset Metadata• Ontologies• Integration Rules

ARCHITECTURE







Target Systems


Graphical User Interface

Customer Applications

May 2, 20239

INTEGRATION PROCESS

Dataset Management• Catalog Datasets• Catalog Ontologies• Manage Metadata

Dataset Discovery• Data Profiling• Dataset Exploration

Dataset Integration• Dataset Lifting• Dataset Linking• Data Quality

Validation

Data Access• Domain Specific

Consolidated Views• Execution on Hadoop

May 2, 202310

DATASET MANAGEMENT




Validation



May 2, 202311

DATASET CATALOG

• Enables the user to explore and manage datasets in the data lake• Files in the data lake (CSV, XML, Excel)• Databases (Apache Hive or external databases)

May 2, 202312

MANAGING METADATA

• Exploring and editing dataset metadata • Semantic content information, like

textual descriptions, tags and related Persons

• Technical information and parameters, like formats, data model and encoding

• Access information, like access path or URL, source system or API call

• Organizational provenance, like organizational units owning or maintaining the dataset

DATASET DISCOVERY




Validation



May 2, 202313

May 2, 202314

DATASET DISCOVERY

• Goal: Augment a dataset with data from related datasets• Automatic discovery of dataset with overlapping information• Explorative interface• Discovery is based on two data parts

• Business meta data• Profiling summary

May 2, 202315

DISCOVERY VIEW

• Datasets are matched based on their metadata (profiling + business data)

May 2, 202316

DATASET PROFILING

• Datasets often contain implicit and explicit schema information• Column names, data formats, enumerated values etc.• Example: column contains formatted dates

• Idea: Extract a dataset summary• For each column / property the summary contains:

1. Data type (e.g., number, date, industry classification)2. Data format (e.g., date format)3. Data statistics (e.g., range, distribution, most frequent values)

• Materialized as RDF with UI view

May 2, 202317

DETECTING DATA TYPES

• Detecting common datatypes as well as user-defined types• Common datatypes

• Numbers• Dates / Times• Geographic locations (geo-coordinates, states, countries)

• User-defined data types can be integrated by adding an ontology / taxonomy• Usually a SKOS taxonomy• Managed as another dataset in the dataset management• Example: Industry taxonomy

• Standard taxonomy (NACE, SIC, NAICS) or company specific

May 2, 202318

FORMATS AND STATISTICS

• For some types, the data format is detected• Example: Dates are formatted in DD-MM-YYYY

• Two functions are generated:1. Parser that is able to read the detected representation2. Normalizer that converts the parsed values into a configurable,

organization-wide target representation• Statistics summarize the values:

• Value range and distribution• Most frequent values• Data selectivity

DISCOVERY VIEW

• Datasets are matched based on their metadata (profiling + business data)

May 2, 202319

May 2, 202320

INTEGRATION PROCESS




Validation



May 2, 202321

DATA INTEGRATION

• The integration process is driven by a set of rules• Lifting Rules map the source datasets to a ontology• Linking Rules connect different datasets to a knowledge graph

• Rules are operator trees, consisting of four types of operators• Data Access Operators• Transformation Operators• Similarity Operators• Aggregation Operators

• Rules can be learned using genetic programming algorithms• Rules are human understandable and can be edited

May 2, 202322

DATASET LIFTING

• Objective: Map the datasets in the data lake to a consistent vocabulary.• A lifting rule consists of a number of mappings

• Each mapping assigns a term in the original data set (such as a column for tabular data) to a term in the target ontology (such as a property provided by an ontology).

• Multiple mappings for each dataset can be managed to allow different views on the same data.• Initial mappings are generated automatically based on the

profiling results from where the user can continue to build on.

May 2, 202323

LIFTING EXAMPLE

Bond ISIN Country Industry

NEDWBK CAD 5,2%25 CA639832AA25 Canada Banking

SIEMENSF1.50%03/20 DE000A1G85B4 Germany Electrical Equipment

Electricite de France (EDF), 6,5% 26jan2019

USF2893TAB29 France Utilities

NEDWBK CAD 5,2%25

fibo:hasSecurityIdentifier

Utilities

Industry Ontology

Banking

France

Country Ontology

Germany

EMEA

“CA639832AA25”

fibo:legallyRecordedIn

fibo:industrySector

May 2, 202324

LINKING

• Goal: Connect individual datasets to a knowledge graph• Identify related entities in different datasets and link them

• Either entities describing the same real world object or another relationNEDWBK CAD 5,2%25

ratingScore

Industry OntologyCountry Ontology

EMEA“AAA”


fibo:industrySector

Rating CAD 5,2%25hasRating

fibo:industrySector


May 2, 202325

LINKAGE RULES

• Linking is based on domain-specific rules• Specify the conditions that must hold true for two entities to be

linked

May 2, 202326

LEARNING LINKAGE RULES

Problem: Manually writing rules is time-consuming and requires expertiseApproach: Interactive machine learning algorithm for generating rules• Generates a rule based on a number of user-confirmed link candidates.• Link candidates are actively selected by the learning algorithm to include link

candidates that yield a high information gain.• The user does not need any knowledge of the characteristics of the dataset or any particular similarity computation techniques.

INTEGRATION PROCESS




Validation



May 2, 202328

VIEW GENERATION

• The user selects a set of lifted and linked datasets

May 2, 202329

Hadoop Data Lake

DATA ACCESS

• Generate data flows based on Apache Spark• The data flows utilize Resilient

Distributed Datasets (RDDs)• RDDs derive new data sets from

existing data sets by applying a chain of transformations• A derived data set can either

• be recomputed on-the-fly • persisted on stable storage

• Data flows can be executed efficiently on Hadoop clusters. Corporate

Bonds

Data Lifting 1(Apache Spark

RDD)

Data Linking(Apache Spark RDD)

Internal Ratings


RDD)

External Ratings


RDD)

eccenca Corporate

Memory

Data Consumer

SQL CSVExcel

SparkAPI

ContactDr. Robert IseleTel: +49 151 17238616email: [email protected]

eccencaCommand your Data!

Robert Isele | eccenca CorporateMemory - Semantically integrated Enterprise Data Lakes

Technology

Transcript of Robert Isele | eccenca CorporateMemory - Semantically integrated Enterprise Data Lakes