Building a Big Data Warehouse
-
Upload
mids106 -
Category
Technology
-
view
102 -
download
0
description
Transcript of Building a Big Data Warehouse
![Page 1: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/1.jpg)
GoDataDrivenPROUDLY PART OF THE XEBIA GROUP
Building a Big Data Warehouse
Joris BontjeBig Data Hacker
![Page 2: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/2.jpg)
GoDataDriven
About MeBig Data HackerData Driven Solution ArchitectHadoop Trainer
![Page 3: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/3.jpg)
![Page 4: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/4.jpg)
GoDataDriven
About GoDataDriven
![Page 5: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/5.jpg)
Data Warehouse Evolution
![Page 6: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/6.jpg)
http://en.wikipedia.org/wiki/Data_warehouse
In computing, a data warehouse is a database used for reporting and data analysis.
![Page 7: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/7.jpg)
GoDataDriven
Database Architecture (1.0)
Products)Customers)Orders)
Inventory)Sales)DB)
![Page 8: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/8.jpg)
GoDataDriven
Analytical Database (2.0)
Sales&
Inventory&Customers&
Products&Orders&
![Page 9: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/9.jpg)
GoDataDriven
Basic DWH Architecture
TX#DB#
Analy+cal#DB#
BI#ETL
![Page 10: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/10.jpg)
GoDataDriven
Data Marts
TX#DB# DW#
Sales#
Mktg#
Prch#
BI#
![Page 11: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/11.jpg)
GoDataDriven
Multiple Data-Sources
other&
Files&
TX&DB&
DW&
Sales&
Mktg&
Prch&
BI&
![Page 12: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/12.jpg)
GoDataDriven
Operational Data Store
DW#ODS#
other#
Files#
TX#DB# Sales#
Mktg#
Prch#
BI#
![Page 13: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/13.jpg)
Hadoop
![Page 14: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/14.jpg)
GoDataDriven
No Hadoop
DW#ODS#
other#
Files#
TX#DB# Sales#
Mktg#
Prch#
BI#
![Page 15: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/15.jpg)
GoDataDriven
ETL Engine
other&
Files&
TX&DB& Sales&
Mktg&
Prch&
DW BI&
![Page 16: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/16.jpg)
GoDataDriven
Tiered Data Warehouse
other&
Files&
TX&DB& Sales&
Mktg&
Prch&
BI&
![Page 17: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/17.jpg)
GoDataDriven
Analytical Query Engine
other&
Files&
TX&DB&
BI&
![Page 18: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/18.jpg)
Tools
![Page 19: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/19.jpg)
GoDataDriven
Tools
![Page 20: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/20.jpg)
Tools Applied
![Page 21: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/21.jpg)
GoDataDriven
Tools Applied
![Page 22: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/22.jpg)
Considerations
![Page 23: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/23.jpg)
GoDataDriven
ConsiderationsBig Data is dirtyAutomate everythingMonitoring and QA become the same thing
![Page 24: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/24.jpg)
My Past TrendsBig Data Forum 2012
![Page 25: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/25.jpg)
GoDataDriven
My Past Trends
Cloud / On-demand
![Page 26: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/26.jpg)
GoDataDriven
My Past Trends
Hadoop Hardware
![Page 27: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/27.jpg)
GoDataDriven
My Past Trends
Batch → Real-Time
![Page 28: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/28.jpg)
New TrendsXebiCon 2013
![Page 29: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/29.jpg)
GoDataDriven
TrendsImpala
Open Source, Real-time Query enginefor Hadoop
![Page 30: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/30.jpg)
GoDataDriven
Trends
Defacto standard for Hadoop metadata
![Page 31: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/31.jpg)
GoDataDriven
Simple Database Architecture
Products)Customers)Orders)
Inventory)Sales)DB)
![Page 32: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/32.jpg)
GoDataDriven
The future?
Products)Customers)Orders)
Inventory)Sales)
![Page 33: Building a Big Data Warehouse](https://reader034.fdocuments.us/reader034/viewer/2022051819/54c9559e4a7959dd078b4584/html5/thumbnails/33.jpg)
GoDataDriven
We’re hiring / Questions? / Thank you!
Joris BontjeBig Data Hacker