Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and...
Transcript of Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and...
![Page 1: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/1.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
News / Outlook / VisionsPerformance und Scalability
Ralf NörenbergDirector
![Page 2: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/2.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
2
Performance und ScalabilityTopics
1. Sketching Tomorrow2. Big Data Project A: „ODS as a Master“3. Big Data Project B: „ODS as a Slave“4. Expectations and Combination of Results
![Page 3: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/3.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Physical Storage
Todays ODS Infrastructure and Architecture View
3
ODS Server
Application
![Page 4: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/4.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Today’s ODS What are the questions?
4
Give me all measurements of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”.
“Meta data” result based on a “meta data” query.
![Page 5: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/5.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS What are the questions?
5
Give me all <vehicles, sensors, …> of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”
“Meta data” result based on a combined “meta data & mass data” query.
where Fuel Consumption > 10 literswhere Speed Peak > 150 km/hwhere Avg. Temp > 35 °Cwhere Diesel Pipe Burning = yes
![Page 6: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/6.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
6
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
![Page 7: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/7.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Applications
7
Applications have the benefit of predefined questions:boundaries to limit possible questions / queries that can be askedStreamlining of information flow
provision of pre-cached result setsProvision of pre-configured information about mass-data
![Page 8: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/8.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Applications
8
Applications have the benefit of predefined questions:boundaries to limit possible questions / queries that can be askedStreamlining of information flow
provision of pre-cached result setsprovision of pre-configured information about mass-data
Give me all <vehicles, sensors, …> of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”
No Big Data required.
where Fuel Consumption > 10 liters = AVG ODSwhere Speed Peak > 150 km/h = MAX ODSwhere Avg. Temp > 35 °C = AVG ODSwhere Diesel Pipe Burning = yes = pre-configured ODS
![Page 9: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/9.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS What are the questions?
9
Give me all <vehicles, sensors, …> of vehicle “Ford”, type “Focus”, year “2015” and engine “3.0”
We need new technologies (“big data”)!
where Fuel Consumption > 10 literswhere Speed Peak > 150 km/hwhere Avg. Temp > 35 °Cwhere Diesel Pipe Burning = yesin continuous time window of “30 min” within measurement
![Page 10: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/10.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Big Data Queries
10
Applications may but “correlation queries” surely require new technologies forAnalysis of mass dataReturn of result sets
For “random” queries, there are consequences on the solution architectureStreamlining of information flow is not possible
How many experts are using the system?How often are the same / likewise queries used?How will the provision of “cached” information work (for performance)?
![Page 11: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/11.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
11
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
Being “smart” is key! /Discussion currently focus on the very last part of the process
Customer Requirements /
Solution Specification
Customer Requirements /
Solution Specification
“open”
![Page 12: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/12.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
12
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
HQL Implementationavailable
![Page 13: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/13.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
13
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
Scalability available / Avalon Distributor
![Page 14: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/14.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
14
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
Merlin 2G ImplementationAvailable / Architecture Design analogous to Avalon Scalability (to be shown)
![Page 15: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/15.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Tomorrow’s ODS Sketching the Future
15
Meta-Data
ODS Server
Applications„Big Data“ Query
(Correlations)
Query Interface
Reduction / Filtering of Information
Big Data Technology Adapter A
„Cache“ - Level
Mass-Data
„Cache“ - Level
Analysis Server
Routing of Information
Big Data Technology Adapter B
Big Data Research
![Page 16: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/16.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
16
Performance und ScalabilityTopics
1. Sketching Tomorrow2. Big Data Project A: „ODS as a Master“3. Big Data Project B: „ODS as a Slave“4. Expectations and Combination of Results
![Page 17: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/17.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
17
Big Data Project A: „ODS as a Master“Project Facts
Project started April 2015 and is runningOur partner is in Ingolstadt (~10.000 employees Tier 1 supplier)
Why partner up?
a big partner is required for provision of scalable work environments
“ODS technology” and “customer IT infrastructure” are regarded to still be existent (now: Avalon and Oracle)
there is an existing challenge with measurements of big size (~50GB each)
there is an existing big data cluster running
![Page 18: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/18.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
18
Big Data Project A: „ODS as a Master“Objectives
1) Performance analysis of regular ODS system with big data use cases (bottlenecks?)
2) Identification of “BIG ODS” system designs and architectures
3) Performance analysis of “BIG ODS”
openMDM application data access
Avalon Server data management
Merlin Server integration / analysis
Frequent Updates within ASAM US Big Data Workshop
Working Prototype for ASAM Big Data Conference
![Page 19: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/19.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
19
Big Data Project A: „ODS as a Master“Planned Steps
1) Setting up a “SPARK”-Cluster and writing mass-data into it (partly done)
2) Retrieving channel data from cluster (driver development) (working on)
3) Performance Analysis considering “huge tables” / Solution Identification (to start soon)
4) Integration of measurement meta data into SPARK (purple elements) (planned)
Solve the challenge of “Joints” between Oracle and SPARK
5) Retrieval data out of BIG ODS system (planned)
![Page 20: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/20.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
20
Big Data Project A: „ODS as a Master“System Architecture
Spark Cloud / Cluster
Hadoop
Files
SQL
XY
1
![Page 21: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/21.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
21
Big Data Project A: „ODS as a Master“System Architecture
Spark Cloud / Cluster
Files
Avalon Server
Driver
OracleJOB
Driver submits “orders” (to be generic)
Jobs accepts “orders” (specific to physical storage and ODS data model)
Master / Worker enable physical storage access
Master / Worker
2
3
4
4
5
![Page 22: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/22.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
22
Big Data Project A: „ODS as a Master“Sustantiated Big Data Statements
Spark Cloud / Cluster
Files
Avalon Server
Driver
OracleJOB
Technology Boundary: ODS Server needs to run outside of cluster
Scalability: Multiple (local) clusters may run next to data generators and enable scalability
Confirmation: ODS Server / Oracle remain as structuring entities
Master / Worker
JOB
![Page 23: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/23.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
23
Performance und ScalabilityTopics
1. Sketching Tomorrow2. Big Data Project A: „ODS as a Master“3. Big Data Project B: „ODS as a Slave“4. Expectations and Combination of Results
![Page 24: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/24.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
24
Big Data Project A: „ODS as a Slave“Project Facts
Project started April 2015 and is runningOur partner is
Why partner up?
a big partner is required for provision of a scalable middle-ware solution
Know-how of integration systems to middle-ware
Objectives
General Case Study and Prototype
Specific Project Realization
![Page 25: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/25.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
25
Big Data Project A: „ODS as a Slave“Today’s Setup
Third Party Analysis Tool
Avalon Server+ Physical StorageMoMa
![Page 26: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/26.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
26
Big Data Project A: „ODS as a Slave“Introduction of a Middleware
Third Party Analysis Tool
BUS - Middleware
Avalon Server+ Physical StorageMoMa
Log /Admin
![Page 27: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/27.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
27
Big Data Project A: „ODS as a Slave“Use-Case A: Connecting ODS and ODS
Third Party Analysis Tool
BUS - Middleware
Avalon Server+ Physical Storage
Avalon Server+ Physical Storage
Combined ODSWeb
![Page 28: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/28.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
28
Big Data Project A: „ODS as a Slave“Use-Case B: Connecting ODS and None-ODS
Third Party Analysis Tool
BUS - Middleware
Avalon Server+ Physical Storage
None-ODSThird Party Tool
None-ODS
![Page 29: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/29.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Combined ODSWeb
29
Big Data Project A: „ODS as a Slave“Use-Case C1: Scalability of ODS Systems by Outsourcing (“Copy”)
Third Party Analysis Tool
Avalon Server+ Physical Storage None-ODSAvalon Server
+ Physical StorageAvalon Server
+ Physical Storage
BUS - Middleware
Avalon Server+ Physical Storage
![Page 30: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/30.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
30
Big Data Project A: „ODS as a Slave“Use-Case C2: Scalability of ODS Systems by Outsourcing (“Move”)
Avalon Server+ Physical Storage
Avalon Server+ Physical Storage
BUS - Middleware
Avalon Server+ Physical Storage
Tool XYThird Party Analysis Tool
![Page 31: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/31.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Combined ODSWeb
31
Big Data Project A: „ODS as a Slave“Use-Case D: Connecting Locations
Third Party Analysis Tool
Avalon Server+ Physical Storage None-ODS
BUS - Middleware
Tool XY Tool XYThird Party Analysis Tool
![Page 32: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/32.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
Combined ODSWeb
32
Big Data Project A: „ODS as a Slave“Use-Case E: Scalability / Decentralization of indifferent Set-Ups
Third Party Analysis Tool
Avalon Server+ Physical Storage None-ODSAvalon Server
+ Physical StorageAvalon Server
+ Physical Storage
BUS - Middleware
Avalon Server+ Physical Storage
Tool XY Tool XYThird Party Analysis Tool
![Page 33: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/33.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
33
Performance und ScalabilityTopics
1. Sketching Tomorrow2. Big Data Project A: „ODS as a Master“3. Big Data Project B: „ODS as a Slave“4. Expectations and Combination of Results
![Page 34: Performance und Scalability - HighQSoft€¦ · Performance und Scalability Expectations and Combination of Results ODS and Big Data will supplement each other If technologies like](https://reader034.fdocuments.us/reader034/viewer/2022042604/5f60588d9d877135c57ad732/html5/thumbnails/34.jpg)
HighQSoft GmbH | www.highqsoft.de | 21.05.2015
34
Performance und ScalabilityExpectations and Combination of Results
ODS and Big Data will supplement each otherIf technologies like SPARK are sustainable, the actual physical storage is of secondary interest
Both project designs (master & slave) supplement each otherRealization of successful BIG ODS systems is realistic
This presentation shall encourage discussions throughout the day! Many topics of the overall picture will be discussed and presented today!
Our outlook is optimistic!