B3 - Business intelligence apps on aws
-
Upload
amazon-web-services -
Category
Technology
-
view
536 -
download
1
description
Transcript of B3 - Business intelligence apps on aws
![Page 1: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/1.jpg)
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Business Intelligence Applications on AWS Steffen Krause, Amazon Web Services
@sk_bln
![Page 2: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/2.jpg)
Overview
Designing BI & big data solutions in the cloud Not the only way to do it (but one that we have seen)
![Page 3: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/3.jpg)
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
![Page 4: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/4.jpg)
![Page 5: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/5.jpg)
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
![Page 6: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/6.jpg)
Data App App
h(p://blog.mccrory.me/2010/12/07/data-‐gravity-‐in-‐the-‐clouds/
Data has gravity
Compute Storage Big Data
![Page 7: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/7.jpg)
Data App App
h(p://blog.mccrory.me/2010/12/07/data-‐gravity-‐in-‐the-‐clouds/
latency Throughput
…and iner0a at volume…
Compute Storage Big Data
![Page 8: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/8.jpg)
Data
h(p://blog.mccrory.me/2010/12/07/data-‐gravity-‐in-‐the-‐clouds/
…easier to move applica0ons to the data
Compute Storage Big Data
![Page 9: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/9.jpg)
Courtesy http://techblog.netflix.com/2013/01/hadoop-platform-as-service-in-cloud.html
S3 as a “single source of truth”
S3
![Page 10: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/10.jpg)
Getting your Data into AWS
Amazon S3
Corporate Data Center
• Console Upload
• FTP
• AWS Import Export
• S3 API
• Direct Connect
• Storage Gateway
• 3rd Party Commercial Apps
• Tsunami UDP
![Page 11: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/11.jpg)
Write directly to a data source
Your applica+on Amazon S3
DynamoDB
Any other data store
Amazon S3
Amazon EC2
![Page 12: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/12.jpg)
Queue, pre-process and then write
Amazon Simple Queue Service (SQS)
Amazon S3
DynamoDB
Any other data store
![Page 13: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/13.jpg)
Amazon SQS
Amazon S3
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
Choose depending upon design
![Page 14: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/14.jpg)
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
![Page 15: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/15.jpg)
Hadoop based Analysis
Amazon S3 Amazon EMR
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 16: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/16.jpg)
EMR is Hadoop in the Cloud
Amazon Elastic MapReduce (EMR)?
![Page 17: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/17.jpg)
EMR Cluster
S3
Put the data into S3
Choose: Hadoop distribuGon, # of nodes, types of nodes, custom configs, Hive/Pig/etc.
Get the output from S3
Launch the cluster using the EMR console, CLI, SDK, or APIs
You can also store everything in HDFS
How does EMR work ?
![Page 18: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/18.jpg)
Resize Nodes
EMR Cluster
You can easily add and remove nodes
![Page 19: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/19.jpg)
1 instance for 100 hours =
100 instances for 1 hour
![Page 20: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/20.jpg)
Small instance = $5.50 (including EMR – without: $4.40)
![Page 21: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/21.jpg)
1 instance for 1000 hours =
1000 instances for 1 hour
![Page 22: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/22.jpg)
Small instance = $55 (including EMR – without: $44)
![Page 23: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/23.jpg)
When you turn off your cloud resources, you actually stop paying for them
![Page 24: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/24.jpg)
SQL based processing
Amazon S3 Amazon EMR
Amazon Redshift
Pre-processing framework
Petabyte scale Columnar Data -warehouse
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 25: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/25.jpg)
Amazon Redshift is a fast and powerful, fully managed, petabyte-scale data warehouse service in the AWS cloud
What is Amazon Redshift ?
Easy to provision and scale
No upfront costs, pay as you go
High performance at a low price
Open and flexible with support for popular BI tools
![Page 26: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/26.jpg)
Demo: Amazon Redshift
![Page 27: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/27.jpg)
![Page 28: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/28.jpg)
Generation
Collection & storage
Analytics & computation
Collaboration & sharing
![Page 29: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/29.jpg)
Your choice of BI Tools
Amazon S3 Amazon EMR
Amazon Redshift
Pre-processing framework
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 30: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/30.jpg)
![Page 31: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/31.jpg)
Demo Jaspersoft as a BI Frontend
![Page 32: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/32.jpg)
![Page 33: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/33.jpg)
Sharing results and visualizations
Amazon S3 Amazon EMR
Amazon Redshift
Web App Server Visualization tools
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 34: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/34.jpg)
Sharing results and visualizations
Amazon S3 Amazon EMR
Amazon Redshift Business
Intelligence Tools
Business Intelligence Tools
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 35: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/35.jpg)
Geospatial Visualizations
Amazon S3 Amazon EMR
Amazon Redshift Business
Intelligence Tools
Business Intelligence Tools
GIS tools on hadoop
GIS tools
Visualization tools
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 36: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/36.jpg)
Rinse and Repeat
Amazon S3 Amazon EMR
Amazon Redshift
Visualization tools
Business Intelligence Tools
Business Intelligence Tools
GIS tools on hadoop
GIS tools
Amazon data pipeline
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 37: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/37.jpg)
The complete architecture
Amazon S3 Amazon EMR
Amazon Redshift
Visualization tools
Business Intelligence Tools
Business Intelligence Tools
GIS tools on hadoop
GIS tools
Amazon data pipeline
Amazon SQS
DynamoDB
Any SQL or NoSQL Store
Log Aggrega+on tools
![Page 38: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/38.jpg)
Real Time
![Page 39: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/39.jpg)
Amazon Kinesis • Real-time processing • Massive scale • Integrated • Use cases:
• Real-time log analysis • Real-time data analytics • Social media monitoring • Financial transactions • Online machine learning
![Page 40: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/40.jpg)
Amazon Kinesis Data Flow Data Sources
App.4 [Machine Learning]
AWS En
dpoint
App.1 [Aggregate & De-‐Duplicate]
Data Sources
Data Sources
Data Sources
App.2 [Metric ExtracGon]
S3
DynamoDB
Redshift
App.3 [Sliding Window Analysis]
Data Sources
Availability Zone
Shard 1 Shard 2 Shard N
Availability Zone Availability Zone
![Page 41: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/41.jpg)
Use cases
![Page 42: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/42.jpg)
SkillPages
Customer Use Case
Everyone Needs Skilled People
At Home At Work In Life
Repeatedly
![Page 43: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/43.jpg)
Who they are
What they can do
Your real life connections to them
Examples of what they can do
![Page 44: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/44.jpg)
Data Architecture
Data Analyst
Raw Data
Get Data
Join via Facebook
Add a Skill Page
Invite Friends
Web Servers Amazon S3 User Action Trace Events
EMR Hive Scripts Process Content
• Process log files with regular expressions to parse out the info we need.
• Processes cookies into useful searchable data such as Session, UserId, API Security token.
• Filters surplus info like internal varnish logging.
Amazon S3
Aggregated Data
Raw Events
Internal Web
Excel Tableau
Amazon Redshift
![Page 45: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/45.jpg)
We found that Amazon Redshi^ offers the performance we needed while freeing us from the licensing costs of our previous soluGon With Amazon Redshi^ and Tableau, anyone in the company can set up any queries they like—from how users are reacGng to a feature, to growth by demographic or geography, to the impact sales efforts have had in different areas. It’s very flexible
Jon Hoffman, So<ware Engineer, Foursquare
0
0.2
0.4
0.6
Female Male
Gender
0 20 40 60 80
Age
Foursquare
Gorilla Coffee
Gray's Papaya
Amorino
When do people go to a place?
![Page 46: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/46.jpg)
Stack – analysis and sharing
App
licat
ion
Sta
ck
Scala/Liftweb API Machines WWW Machines Batch Jobs
Scala Application code
Mongo/Postgres/Flat Files Databases Logs
Dat
a S
tack
Amazon S3 Database Dumps Log Files
Hadoop Elastic Map Reduce
Hive/Ruby/Mahout Analytics Dashboard Map Reduce Jobs
mongoexport postgres dump Flume
![Page 47: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/47.jpg)
Everything that was a limited resource
is now a programmable resource
![Page 48: B3 - Business intelligence apps on aws](https://reader034.fdocuments.us/reader034/viewer/2022051818/54b6c74e4a79599f728b45bb/html5/thumbnails/48.jpg)
• Hadoop Technology and Use Cases: http://www.powerof60.com/
• http://aws.amazon.com/de • Start with the Free Tier:
http://aws.amazon.com/de/free/ • 25 US$ credits for new German customers:
http://aws.amazon.com/de/campaigns/account/ • Twitter: @AWS_Aktuell • Facebook:
http://www.facebook.com/awsaktuell • Webinars: http://aws.amazon.com/de/about-aws/events/
Resources