SnapLogic Best Practices: Sub Pipelines and Guaranteed Delivery
Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth...
-
Upload
snaplogic-inc -
Category
Technology
-
view
358 -
download
2
Transcript of Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth...
![Page 1: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/1.jpg)
Weathering the Data Storm: How SnapLogic and Amazon Web Services
Deliver Analytics in the Cloud for Earth Networks
![Page 2: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/2.jpg)
Today’s Agenda
Eddie Dingels
Architect, Earth NetworksErin Curtis
Product Marketing, SnapLogic
• Amazon Web Services
• SnapLogic and Redshift
• Earth Networks: Moving Data Analytics Into The Cloud
• Discussion
Kyle Lichtenburg
Solutions Architect, Amazon Web Services
![Page 3: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/3.jpg)
![Page 4: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/4.jpg)
2011
82
159
2012
280
2013
516
2014
AWS’ Rapid Pace of Innova3on AWS has launched a total of 1,515 new features and/or services since incep:on in 2006.
2015
+342*
* As of July 9, 2015
![Page 5: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/5.jpg)
More Func:onality Than Any Other Infrastructure Provider
![Page 6: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/6.jpg)
It’s never been easier and less expensive to collect, store, analyze & share data
![Page 7: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/7.jpg)
Companies will use data more expansively than at any other point in history
![Page 8: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/8.jpg)
Fully Loaded for Big Data
• Sources of Truth • High Performance Databases
• Analysis PlaKorms
Amazon S3 Amazon Glacier
Amazon EFS
Amazon DynamoDB Amazon Aurora
Amazon Redshift Amazon Kinesis Amazon EMR
![Page 9: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/9.jpg)
Amazon Simple Storage Service (S3)
• Storage for the Internet • Store and retrieve any amount of data, at any :me, from anywhere on the web
• Highly scalable, reliable, and secure • Supports encryp:on • Pay only for what you use
![Page 10: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/10.jpg)
Amazon DynamoDB
• Fast, fully-‐managed NoSQL Database Service • Capable of handling any amount of data • Durable and Highly Available • All SSD storage • Simple and Cost Effec:ve
![Page 11: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/11.jpg)
Amazon RedshiX
• Fast, simple, fully-‐managed petabyte-‐scale data warehousing
• Online and func:onal in minutes • SQL based • Con:nuous backup • Less than $1,000/TB/Year • ODBC/JDBC Compliant
![Page 12: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/12.jpg)
Connect Faster
![Page 13: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/13.jpg)
Unified Platform for Data, Apps, Things
Our unified platform significantly speeds up enterprise data access everywhere.
– Gaurav Dhillon, co-founder and CEO, SnapLogic
![Page 14: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/14.jpg)
Why SnapLogic Elastic Integration?
Unified Platform Productive User Experience
Modern Architecture Connected: 300+ Snaps
![Page 15: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/15.jpg)
Productive: UX for Citizen and Advanced Users
We can do more in two hours with SnapLogic than we could in two days
with traditional solutions.
• Integration Cloud: Design, Admin, Monitoring• Drag, Drop, Connect: HTML5 interface built for speed
![Page 16: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/16.jpg)
Modern Architecture: Hybrid and Elastic
Streams: No data is stored/cachedSecure: 100% standards-basedElastic: Scales out & handles data and app integration use cases
Metadata
Data
Databases Enterprise Systems Hadoop
![Page 17: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/17.jpg)
Modern Architecture: Real-Time and Batch
Ultra Pipelines SnapReduce and the Hadooplex
Map Reduce
Certified YARN Execution
![Page 18: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/18.jpg)
Connected: 300+ Snaps
We look at SnapLogic as an opportunity to think differently about integration.
![Page 19: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/19.jpg)
SnapLogic Integration for Amazon Redshift
Customers:
Free, hosted trial of SnapLogic + Redshift: www.snaplogic.com/redshift-trial
The Redshift Snap helps customers rapidly transfer data into and out of Amazon Redshift from multiple sources• Rapidly connect Redshift to database services• Quickly load data into an Amazon S3 bucket and kick off the Redshift
import process in a single step• Easily replicate source tables into their Amazon Redshift clusters and
detect daily changes to keep data synchronized• Take advantage of core REST and SOAP connectivity
![Page 20: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/20.jpg)
Edward Dingels
7/22/2015
AWS & SnapLogic
![Page 21: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/21.jpg)
Company
• Weather Networks
• Schools/Education
• Consumer
• Alerting
• Environmental Network
• Energy
7/22/2015
![Page 22: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/22.jpg)
Operational Environment
• Weather • Dynamic
• Local
• Users • Engaged
• Proximity
• Spikes/Peak Periods
7/22/2015 22
![Page 23: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/23.jpg)
Data Center
• Scale • Weather intersecting users • Increase rapidly
• Capacity • Hardware • Software
7/22/2015 23
![Page 24: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/24.jpg)
Data Center
• Mobile pushed us to the limit • Demanding performance • Feature releases delays due to capacity planning
The data center became a limiting factor in a space where technology should enable
7/22/2015 24
![Page 25: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/25.jpg)
AWS EC2
• EC2 • Dynamic capacity • Automatic capacity
• SQL • Data tier • Horizontal scale
7/22/2015 25
SQL Data Store
API Tier
Ingest
![Page 26: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/26.jpg)
AWS Storage
• Blob • S3
• Transactional • Dynamo • RDS
• Warehouse • Redshift
7/22/2015 26
Cloud Storage
API Tier
Ingest
![Page 27: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/27.jpg)
AWS ETL
• Storage was great
• ETL limiting • SQS
• Kinesis • EMR
• Data Pipeline
• How do we move data between different cloud data stores effectively?
7/22/2015 27
![Page 28: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/28.jpg)
Cloud Integration
• Needed a new tool set
• Criteria • Data stayed in our VPC
• Repeatable building blocks
• Cloud data stores are 1st tier citizens
• Horizontal scale
• Performance
• Price
7/22/2015 28
![Page 29: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/29.jpg)
Project – Data Ingest
• Network of Networks
• Challenge – Providers • Formats
• Delivery • Standardization
• Solution – Pipeline Per Provider
7/22/2015 29
Partner A
Partner B
Pipeline A
Pipeline B
Cloud Data Stores
![Page 30: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/30.jpg)
Project – Data Analysis
• Deriving KPIs for BI
• Challenge – Domains • Unique domain
• Different storage technologies • Varied timeliness requirements
• Solution – Pipeline Per Domain
7/22/2015 30
Redshift
BI Toolset
Pipeline
Domain
![Page 31: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/31.jpg)
Operational – Automated Database Tasks
• Storage Limitation
• Challenge – Automation • Redshift
• MSSQL RDS
• Solution – Scheduled pipeline
7/22/2015 31
Scheduled Pipeline
Cloud Data Stores
![Page 32: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/32.jpg)
AWS + snapLogic
• AWS is the platform
• SnapLogic is the glue
Drives faster implementations with repeatable patterns for more business value
7/22/2015
![Page 33: Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cloud for Earth Networks](https://reader034.fdocuments.us/reader034/viewer/2022042716/55cbb11fbb61ebbd678b46b4/html5/thumbnails/33.jpg)
Thank you Questions?
See SnapLogic in action:
Contact us: [email protected]
http://video.snaplogic.com/
@SnapLogic @awscloud @EarthNetworks