Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Andreas Buckenhofer

Data Warehouse (Datenbanken II)

Daimler TSS GmbH

Overview of the lecture

Data Warehouse / DHBW / Fall 2016 / Page 2

1. Introduction to DWH, DWH Architectures - 20.10.2016

2. Data Modeling, OLAP 1 - 27.10.2016

3. OLAP 2, ETL - 03.11.2016

4. Metadata, DWH Projects, Advanced Topics - 10.11.2016

Daimler TSS GmbH

What you will learn today

• After the end of this lecture you will be able to

• Understand the necessity for metadata

• Understand lifecycle of DWH projects

• Advanced topics like Operational BI, DWH Appliances, Cloud BI

Daimler TSS GmbH

Metadata

Daimler TSS GmbH

What is metadata?

other data

Daimler TSS GmbH

Types of metadata

• Business Metadata

• Definition of business vocabulary and relationships

• Definition of the value range

• Linkage to physical representation

Daimler TSS GmbH

Types of metadata

• Report metadata

• Report definitions

• Data sources

• Column definitions

• Computations

• Logical and physical metadata of data model

• Table structure

• Definition of columns

• Relationships between tables and columns

• Dimension hierarchy

Daimler TSS GmbH

Types of metadata

• ETL metadata

• Job design

• Input-/output tables

• computations

• Mappings / transformations

• Operational meta data of ETL jobs

• Start time and duration

• Return code

Daimler TSS GmbH

The Areas of Metadata

Daimler TSS GmbH

The Areas of Metadata Connected

Daimler TSS GmbH

Why a common metadata repository?

• Components of a data warehouse system are interconnected

• BI report user has to know

• the meaning, definitions of the shown measures, „KPIs“ (key performance

indicators)

• BI report designer has to know

• the table definitions

• the meaning of the column values

• ETL job designer has to know

• the table definitions

• the exact definition of the measures

• Database administrator has to know

• Which tables are used by ETL jobs, reports

Daimler TSS GmbH

Why a common metadata repository?

• Metadata driven ETL development

• Generate parts of ETL code

• increasing interest for Data Vault development projects

• Tools e.g. MID Innovator, Quipu, AnalytiX DS, Talend, Pentaho, Wherescape, and

others

• Common metadata repository ensures consistency across all components

• Many tools involved (DB, ETL, Frontend, …)

• Enables cross component metadata analysis

• Data Lineage

• Impact Analysis

Daimler TSS GmbH

Where does a field of data in this report come from?

• “Data lineage”

• Import & Browse Full BI Report Metadata

• Navigate through report attributes

• Visually navigate through data lineage across tools

• Combines

operational &

design viewpoint

Daimler TSS GmbH

What happens if I change this column?

• “Impact Analysis”

• Show complete change impact in graphical or list form

• Includes impact on reports in BI tools

• Visually navigate through impacted objects across tools

• Allows impact analysis on any object type

Daimler TSS GmbH

What does this field mean?

• Show relationships between business terms, data model entities, and technical and

report fields

• Requires cross-tool mapping of business terms

• Allows field meaning to be understood

• Allows business term relationships to be understood

Daimler TSS GmbH

What objects does this user own?

• Shows objects that user manages

• Shows stewardship relationships on business terms

• Shows user group associations

Daimler TSS GmbH

What happened on the last job run?

• Navigation through complete job details

• Navigation of complete operational metadata

Daimler TSS GmbH

Data Warehousing Projects

Daimler TSS GmbH

Data Warehouse

FrontendBackend

External data

sources

Internal data

sources

Top-Down vs Bottom-Up Approach

Staging Layer

(Input Layer)

Core Warehouse

(Storage Layer)

Reporting Layer

(Output Layer)

(Mart Layer)

Top Down (Inmon)

Bottom Up (Kimball)

Daimler TSS GmbH

Top-Down vs Bottom-Up Approach

• Top-Down (Inmon)

• Design Core Warehouse Layer = integrated data model first

• Design data marts afterwards

• Bottom-Up (Kimball)

• Design data marts first

• Combine data Marts together

• DWH Bus architecture

• conformed dimensions to integrate different data marts / fact tables

Daimler TSS GmbH

Think big, start local

• Both approaches have their down-sides

• Top-Down takes enormous initial effort to build data model for Core Warehouse

• Bottom-Up is risky as central / integrated focus is lost

�Think big, start local

• Small iterations

• Waterfall approach taking 8-12 months or longer often fails or does not deliver in

• Always think about how to achieve flexible data integration in Core Warehouse Layer

• Data Marts can be dropped and reloaded from Data in the Core Warehouse Layer

• Dropping the Core Warehouse Layer not possible. Data loss (history)

Daimler TSS GmbH

Why do DWH projects fail?

Daimler TSS GmbH

Critical success factors for building a data warehouse

• Answer most important questions of participating business units

• Provide high-quality data

• Introduction in time

• Usage of modern technology

• Business orientation

• Easy to use

• Executive sponsor

• Patience – user acceptance evolves over time

• “Quick wins”

Daimler TSS GmbH

DWH project phases

Daimler TSS GmbH

1. Project start

• Describe future situations and scenarios

• No technical details

• Develop multiple solutions and discuss their advantages and disadvantages

• Maybe start with a Proof of Concept (PoC)

• Estimate expected amount of data

Daimler TSS GmbH

2. Analysis/Technical Concept

• Information requirements analysis

• Logical modeling of data / information

• Transform knowledge from interviews into logical data schemas (represented by

Multidimensional or Star Schemas)

• Define transformation and unification rules (from data in operative systems to the

data warehouse)

• Identify Frontend requirements

• Define dimensions and measures

• Define reports (layout, prompts, output fields, filter, etc)

• Analyze operative data sources

• Very important task to get an understanding of source data, structures of the data,

data quality

Daimler TSS GmbH

2. Analysis/Technical Concept

• Data and Architectural Concept

• Important: Scalability

• Top-down design

• Transform abstract data model into the world of hardware (e.g. separate servers for

DB, ETL, Frontend), software, scalability, return times, etc.

• Ensure that data warehouse works together with other IT systems

• Tool Selection / Evaluation

• Choose tools: ETL tool, database, Frontend tools

• Has to know own tool-requirements very detailed

• Aspects: performance, availability and uniformness (interfaces, query languages,

Daimler TSS GmbH

3. System Design

• Transition from business view to technical view

• Transform requirements into actual solutions

• Describe how to implement the system

• Create catalog of actual technical and other requirements

Daimler TSS GmbH

Possible DWH Analysis and Design work products

• How to document / identify requirements?

• Must be easy to understand from non-technical users during Analysis/Technical

concept phase

• Must provide sufficient information for System Design phase

• The following slides provide some example work products that are produced during

Analysis/Technical concept phase and may be refine during System Design phase

Daimler TSS GmbH

Source: Lawrence Corr: Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema

BEAM = Business Event

Analysis and Modeling

Daimler TSS GmbH

SK = Surrogate Key

BK = Business Key

CV = Current Value (SCD1)

GD = Granular Dimension

NA = Nonadditive fact

FA = Fully Additive fact

SA = Semiadditive fact

PS = Periodic Snapshot

RP = Role-playing

HV = One Historic value (SCD2)

Daimler TSS GmbH

4. Implementation/Realization

• Data storage

• Install and configure database system

• Create physical data schema for all DWH layers

• Usage of database design tools

Daimler TSS GmbH

4. Implementation/Realization

• Data Integration, ETL

• Transfer data from company-internal and -external sources into the data warehouse

• Connect data sources

• Eliminate mistakes / inconsistencies in data / possible error origins

• Transform data to unique coding

• Aggregate data

• Frontend

• Set up front ends, OLAP tools

• Connect to Data Mart Layer

• Create reports or other visualizations

Daimler TSS GmbH

5. Test & Rollout

• Authorization concept

• Access control

• Not static

• Enable administration

• Production concept

• Concept for initial load and incremental/delta loads

• Concepts to keep the system running, even if amount of data and users increases

exponentially

• Define responsibilities

• Educate users

• Classes for different types of users

Daimler TSS GmbH

BICC: BI Center of Excellence

• Organizational teams that coordinate and standardize DWH activities within an (end

user) organization

• Define standards and create BI portfolio (e.g. which tools/products to use)

• Create DWH architecture and govern BI activities

• Establish processes for business and IT interaction DWH application development

• Monitor DWH/BI market for new trends

• Determine skills and experience of Business users

Daimler TSS GmbH

Exercises

• List 3 reasons why common metadata is important in the context of warehousing

• Define 3-5 criteria for the evaluation of an ETL tool

• How does a relational DBMS (like Oracle, DB2, MS SQL Server) meet these

requirements?

Daimler TSS GmbH

Exercises

• List 3 reasons why common metadata is important in the context of warehousing

• Components of a data warehouse system are interconnected (high complexity!)

• Metadata driven ETL development

• Common metadata repository ensures consistency across all components

• Enables cross component metadata analysis

Daimler TSS GmbH

Exercises

• Define 5 criteria for the evaluation of an ETL tool

• Supplier profile

• Support

• HW/SW requirements

• Costs

• Usability

• Reliability

• Performance and scalability

• Multi-tenant

• Interfaces

• Scheduling

Daimler TSS GmbH

Exercises

• How does a relational DBMS meet these requirements?

• RDBMS provide many of the functionalities but additional programming required

• RDBMS are often used for ETL/ELT by programming with SQL, PL/SQL, SQLT, etc

ETL Tool Manual ETL

Informatica, Talend, Oracle ODI, etc. SQL, PL/SQL, SQLT, etc.

Separate license No additional license

Workflow, error handling, and restart/recovery

functionality included

Workflow, error handling, and restart/recovery

functionality must be implemented manually

Impact analysis and where-used (lineage)

functionality available

Impact analysis and where-used (lineage)

functionality difficult

Faster development, easier maintenance Slower development, more difficult maintenance

Additional (Tool-) Know How required Know How often available

Daimler TSS GmbH

Frontend

Daimler TSS GmbH

Interface to the end user

• Reporting (Standard, ad-hoc)

• OLAP

• Dashboards, Scorecards

• Advanced Analytics / Data Mining / Text Mining

• Search & Discovery

Daimler TSS GmbH

Reporting (Standard, ad-hoc)

• Standard Reports

• Prepared static reports that can be executed at request by end users

• Are executed at the end of an ETL process and e.g. send by email to end users

• Normally based on fact tables and its dimensions

• Reports are often lists similar to Excel-Sheets but can also contain graphics (e.g. line

charts)

• Ad-hoc Reports

• End users create their own reports („Self service“)

Daimler TSS GmbH

• ROLAP / MOLAP Client Frontend

• Prepared cubes (multidimensional or relational fact tables)

• User can perform interactive analysis of data

Daimler TSS GmbH

Dashboards, Scorecards

• „Progress reports“

• Provide an overall view of KPIs (Key Performance Indicators)

• Combination of several elements from Reporting and/or OLAP (e.g. line charts) into an

overall view (like a „cockpit“)

• Dashboard is more focused on operational goals

• High-level overview what is happening

• Scorecard is more focused on strategic goals

• Plan a strategy and identify why something happens

Daimler TSS GmbH

Advanced Analytics / Data Mining / Text Mining

• See Mr. Bollinger‘s lecture

Daimler TSS GmbH

Search & Discovery

• Not just numerical data

• Analysis of new data types gets more and more important

• Text

• GPS coordinates

• Pictures

• Videos

• Data can be available in RDBMS (e.g. text modules/indexes available), Hadoop or SQL

Daimler TSS GmbH

Many graphical elements to use in reports

Daimler TSS GmbH

Many graphical elements to use in reports

Source: https://github.com/d3/d3/wiki/Gallery

Daimler TSS GmbH

Many graphical elements … chamber of horror

Source: Hichert / Faisst, http://www.backup-page.hichert.com/

Daimler TSS GmbH

Information Design

• Information design is the practice of presenting information in a way that fosters

efficient and effective understanding of it.

(source: Wikipedia, https://en.wikipedia.org/wiki/Information_design )

• Some authors are well known for their criticism of many graphical representations -

they provide rules for good information design

• Edward Tufte

• Stephen Few

• Rolf Hichert

• Define standards, e.g.

• use always the same colors and with care (red = negative, green = positive)

• pie charts are rarely useful and should be avoided (better use bar chart or line chart)

• No 3D elements as these elements don’t enhance information but introduce clutter

Daimler TSS GmbH

Table with integrated bar charts

Source: Hichert, http://www.hichert.com/de/resource/table-template-02/

Daimler TSS GmbH

BI end user roles

• Consumers / BI Users

• use reports and dashboards to obtain information

• Power Users

• Use reports and dashboards to obtain information

• Create new reports and dashboards

• Data Scientists

• Statistical / mathematical geeks

• Analyze / explore data

• Need to analyze raw (non-cleansed, non-transformed) data

Daimler TSS GmbH

Visual data discovery and automatic data analysis

Source: Kohlhammer, J., Proff, D.U., Wiener, A.: Visual Business Analytics – Effektiver Zugang zu Daten und Informationen. dpunkt Verlag GmbH, Heidelberg (2013b)

Daimler TSS GmbH

Newer / Advanced Topics

Daimler TSS GmbH

Newer / advanced Topics

1. Operational Data Warehousing

2. Data Warehouse Appliances

3. Cloud BI

Daimler TSS GmbH

Operational Data Warehousing

• Classical“ Data Warehouses

• Information in the warehouse used to support strategic business decisions

• Kept separate from operational systems

• Load of new data only in larger intervals (mostly weekly or monthly)

• Shorter intervals not required by users

• Huge system resources of the ETL process made it necessary to run it in low

usage periods of the warehouse (like night or weekend)

• Near/Real Time Operational Data Warehousing

• Information in the warehouse used for tactical business decisions as well

• Low latency of information in data warehouse therefore needed

• Not only mathematical aggregations

Daimler TSS GmbH

Why operational Data Warehousing?

• With classical data warehouses users have to access two types of systems to get a

complete image of a customer (for instance for CRM applications or in call centers)

• the data warehouse to see what happened in the past

• the OLTP systems to get the most current information

• With an operational data warehouse

• all this information is in one system

• tighter integration with operational systems is easier

• for instance personalized offers � „closing the loop“

Daimler TSS GmbH

Examples of Operational Data Warehousing

New applications and data sources

Increase demand for an

Operational DWH, e.g.

• Industry 4.0 / Smart Factory

• Internet Of Things

• Internet of medical things

• Connected Cars

Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 277

Replace pen

& paper with

electronic

workflows

Decision support for

each end user and not

only management

Increasing demand to

publish same content

on different devices

Daimler TSS GmbH

SmartFactory Service Platform

Source: Gluchowski: Analytische Informationssysteme, 5.Aufl., p. 279

Workers

getting

alarms

Containing

displaying

complex

manuals,

e.g. during

repair

New data

source

sending lots

of data with

high speed

Real-Time

required for

automated

actions

Daimler TSS GmbH

Challenges for Operational Data Warehousing

• Real time ETL

• Triggered by business transactions in the operational systems

• Executed asynchronously

• Incremental real-time load

• Tighter integration of operational and data warehouse systems

• DWHs become „mission critical“

• Higher requirements on availability and performance

• Higher „transactional“ system load on data warehouse system

• Data warehouse DB has to deal with typical DWH system load and transactional

• Not just aggregations on high amount of data rows

Daimler TSS GmbH

Comparison classical DWH – Operational DWH

Classical DWH Operational DWH

Strategic

• Passive

• Historical trends

Tactical

• Execution of strategy

• Prediction

• E.g. daily batch

Real-Time

• Up-to-data view

Availability

• System can be down for maintenance and

longer response times for some reports are

accepted

Availability

• System becomes critical and must fulfill high

availability and performance requirements

Daimler TSS GmbH

Data Warehouse Appliances

• Setting up and configuring a data warehouse system is a complex task

• Hardware

• Servers

• Storage

• Network

• Connectivity to source systems

• Software

• Database management system

• ETL software

• Reporting and analytics software

• ...

• An optimal performance of the whole system is difficult to achieve

Daimler TSS GmbH

Data Warehouse Appliances

• Data Warehouse Appliances are

• Pre-configured and pre-tested hard- and software configurations developed for

running a data warehouse

• Optimized for data warehousing workload

• They are ready to be used after they are delivered to the customer

• Only suited for running OLAP

• In contrast RDBMS: one size fits all: RDBMS are suited for OLTP, OLAP and mixed

workloads

• Products, e.g. Teradata, IBM Netezza (IBM PureData System for Analytics), HP Vertica,

Exasol, Oracle Exadata, MS Analytic Platform System

Daimler TSS GmbH

Simplicity (e.g. Netezza)

Daimler TSS GmbH

Typical enhancements

• Move as many operations as possible to storage cell instead of moving data to the DB

server

• E.g. filter data already at storage cell and not at DB server

• Avoid transferring unnecessary data

• Column-oriented In-memory storage with high compression

• Many appliances are based on shared nothing architecture

• Each node is independent

• Each node has its own storage or memory

• Parallel processing simpler and faster as no overhead due to contention

Daimler TSS GmbH

Cloud BI

• BI applications (database, ETL tools, Frontend) are hosted in a public cloud, e.g.

• AWS (Amazon Web Services)

• Microsoft Azure

• …

• Many tools nowadays are available in the cloud first

• Vendors try to force customers to use clouds

• Or even available in the cloud only

• E.g. Microsoft Power BI

• Security concerns for sensitive data

• But new data source coming from Internet. Storing the data in a (public) cloud can

make sense, e.g.

• Connected Cars, IOT in general

Daimler TSS GmbH

Cloud BI architecture

Source: Lang: Business Intelligence erfolgreich umsetzen, 5.Aufl., p. 185

Daimler TSS GmbH

Cloud BI architecture

• Analytics as a service

• Provide complete BI (Analytics) SW stack including

• data storage

• data integration (ETL)

• data visualization and/or data modeling (Frontend)

• Meta data management

• Data as a service

• Provide quality data for further usage

• Data marketplace

Daimler TSS GmbH

Cloud BI – Data Warehousing services

Source: http://db-engines.com/en/system/Amazon+Redshift%3BSnowflake

Daimler TSS GmbH

Snowflake Architecture

Don‘t confuse

Snowflake product

with Snowflake

dimensional model

from session 2

Daimler TSS GmbH

• Snowflake Storage

• Snowflake loads data into its internal optimized, compressed, columnar format

• Snowflake itself uses (!) Amazon Web Service’s S3 (Simple Storage Service) cloud

storage

• Query Processing

• Each virtual warehouse is an MPP (Multi Parallel Processing) compute cluster

composed of multiple compute nodes allocated by Snowflake from Amazon EC2

• Each virtual warehouse is an independent compute cluster that does not share

compute resources with other virtual warehouses

Daimler TSS GmbH

• Cloud Services

• Authentication and access control

• Infrastructure management

• Metadata management

• Query parsing and optimization

• Security

Daimler TSS GmbH

Exercise

• For one of the following companies

• Bank

• Telecommunication company

• Online book store (like Amazon.com)

• Discount furniture store (like IKEA)

• Airline

• Car manufacturer

sketch an application based on a classical and

another based on a (near) real time operational data warehouse

Daimler TSS GmbH

Exercise

• Compare lecture 1. Possible solutions

• Standard Data Warehouse Architecture

• Data Vault 2.0 Architecture (Dan Linstedt) including log-based discovery (CDC) or

replication for Data extraction

Thank you!

Daimler TSS GmbH

Wilhelm-Runge-Straße 11, 89081 Ulm, Germany / Phone +49 731 505-06 / Fax +49 731 505-65 99

tss@daimler.com / Internet: www.daimler-tss.com / Intranet portal code: @TSS

Domicile and Court of Registry: Ulm / Commercial Register No.: 3844 / Management: Christoph Röger (Vorsitzender), Steffen Bäuerle

Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Data & Analytics

Transcript of Part 4 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

MBM Accounting, Controlling und Steuern - DHBW CAS · Modul- und Dozentenübersicht - Dualer Master am DHBW Center for Advanced Studies Seite 1/43 MBM Accounting, Controlling und

DHBW BWL-Versicherung | Kick-Off Integrations-Workshop 03/14

SQLScript: Efﬁciently Analyzing Big Enterprise … SQLScript: Efﬁciently Analyzing Big Enterprise Data in SAP HANA Carsten Binnig, DHBW Mannheim carsten.binnig@dhbw-mannheim.de

What’s New with SAP BW/4HANA 2.0, SP00 · 2020. 5. 28. · SAP BW/4HANA 2.0 is… SAP's vision for a modern packaged data warehousing solution highly optimized for SAP HANA simplifying

LECTURE @DHBW: DATA WAREHOUSE 03 COMMON DWH …buckenhofer/20182DWH/...I'm regularly giving a full lecture on Data Warehousing and a seminar on ... version of the truth” • Integrates

Introduction to Data Warehousing (DWH) › ~hvoellin › Introduction_DWH.pdf · Date: 01.10.2019 Page: 1 Introduction to Data Warehousing (DWH) DHBW –Fakultät Technik-Informatik,

Code-Generierung Semantik Optimierung - DHBW Stuttgartsschulz/TEACHING/CB2015/Compilerbau_ho.pdf · Compilerbau Stephan Schulz stephan.schulz@dhbw-stuttgart.de Scanner Parser AST

SAP Data Warehousing - MIBCON · 2020. 10. 27. · SAP Data Warehousing approaches Application driven approach = SAP BW/4HANA All required data warehousing services in one integrated

[Arshad Khan] SAP and BW Data Warehousing How to (BookSee.org)

Part 2 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

MBM Marketing - DHBW CAS · Modul- und Dozentenübersicht - Dualer Master am DHBW Center for Advanced Studies Seite 2/18 W2M10701 General Marketing-Management

Enterprise Wide Data Warehousing with SAP BW...©SAP AG 2004, Enterprise Wide BW, J. Haupt / 3 Enterprise Wide BW Background zContinuously increasing number of BW installations zSteady

Green Technology Staffing Presentation dhbw 2013

SAP Learning Hub: Learning Rooms - hs-furtwangen.de · SAP Business Warehouse (SAP BW) Data Warehousing with SAP BW Learning Room This learning room will help you to prepare for the

Dmm300 – mixed scenarios for sap hana data warehousing and BW: overview and experiences

Enterprise Data Warehousing With SAP BW an Overview

Lecture @DHBW: Data Warehouse

CONTACT US Semester at DhBW Stuttgart · Semester at DhBW Stuttgart ... cycle of information technology infrastructure library V3 (ITIL ... human-computer-interaction / usability

Part 1 - Data Warehousing Lecture at BW Cooperative State University (DHBW)

Orientierungsseminar August 2014 DHBW Mosbach, Campus Bad Mergentheim