Journey Through the AWS Cloud; Application Services
-
Upload
amazon-web-services -
Category
Technology
-
view
110 -
download
3
description
Transcript of Journey Through the AWS Cloud; Application Services
![Page 1: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/1.jpg)
Journey through the Cloud:
Application Services
Ryan Shuttleworth – Technical Evangelist @ryanAWS
![Page 2: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/2.jpg)
Common use cases & stepping stones into the AWS cloud Learning from customer journeys
Best practices to bootstrap your projects
Journey through the cloud
![Page 3: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/3.jpg)
Build upon services built for the cloud Address common pain points in application architectures
Reduce operational management of software components Focus on application function, not undifferentiated heavy lifting
Application Services
![Page 4: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/4.jpg)
Why AWS for application services Services overview Dive into select services Where to go next
Agenda
![Page 5: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/5.jpg)
Why AWS for application services?
![Page 6: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/6.jpg)
Application Services
Services that wrap software you’d commonly install and manage
yourself
![Page 7: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/7.jpg)
Application Services
Databases
Middleware
Analytics
Frameworks
![Page 8: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/8.jpg)
Application Services
Reliability Scalability Availability
![Page 9: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/9.jpg)
Application Services
Reliability Scalability Availability
Do you focus on scaling frameworks
rather than optimizing your
code?
Does core component
reliability affect your application
track record?
Is middleware ‘glue’ difficult to
make as available as you
need it?
![Page 10: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/10.jpg)
Application Services
Operational Management
![Page 11: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/11.jpg)
Application Services
Operational Management
Do you spend more time managing application services than you do
building and managing assets core to your business?
![Page 12: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/12.jpg)
Your Business
70%
On-Premise Infrastructure
30%
Managing All of the “Undifferentiated Heavy Lifting”
![Page 13: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/13.jpg)
AWS Cloud-Based
Infrastructure
Your Business
More Time to Focus on Your Business
Configuring Your Cloud Assets
70%
30% 70%
On-Premise Infrastructure
30%
Managing All of the “Undifferentiated Heavy Lifting”
![Page 14: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/14.jpg)
Infrastructure & Application Services
Building blocks for applications
Designed and built for the cloud
Available at end of a web service call
Security Scaling
Database
Networking Monitoring
Messaging
Workflow
DNS
Load Balancing
Backup CDN
Compute
Storage
![Page 15: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/15.jpg)
AWS is used in a variety of ways…
Storage & Archive
Saved months of development & architecture time and focused on application development instead
by using Cloud Search
By managing time-consuming database administration tasks,
RDS allows SEGA to focus on business critical applications
Relies on Simple Workflow to orchestrate complex, heterogeneous
scientific workflows
Brought on massive DynamoDB capacity in an operationally short period of time to handle demand
peaks during SuperBowl
![Page 16: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/16.jpg)
You might be able to:
Business & technical drivers
![Page 17: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/17.jpg)
You might be able to:
Business & technical drivers
Reduce costs
Running software on EC2 can be more expensive than consuming
functionality as a service
![Page 18: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/18.jpg)
You might be able to:
Business & technical drivers
Reduce costs
Running software on EC2 can be more expensive than consuming
functionality as a service
Improve reliability
Services built with inherent multi-AZ functionality improve
application reliability
![Page 19: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/19.jpg)
You might be able to:
Business & technical drivers
Reduce costs
Running software on EC2 can be more expensive than consuming
functionality as a service
Improve reliability
Services built with inherent multi-AZ functionality improve
application reliability
Scale easily
Services scale as you grow and need them, without large
investments in infrastructure
![Page 20: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/20.jpg)
You might be able to:
Business & technical drivers
Reduce costs
Running software on EC2 can be more expensive than consuming
functionality as a service
Improve reliability
Services built with inherent multi-AZ functionality improve
application reliability
Scale easily
Services scale as you grow and need them, without large
investments in infrastructure
Re-focus energies
Spend less time doing undifferentiated heavy lifting and
more time on your business
![Page 21: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/21.jpg)
Availability Zone
Region
Q Instance
Decoupled applications on EC2 instances
Reliable message queue on instance
![Page 22: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/22.jpg)
Availability Zone
Region
Availability Zone Availability Zone
Q Instance
Q Instance
Q Instance
Multiple instances
for reliability
Technical implementation
![Page 23: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/23.jpg)
Availability Zone
Region
Availability Zone Availability Zone
Q Instance
Q Instance
Q Instance
Remove software running on
EC2
![Page 24: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/24.jpg)
Availability Zone
Region
Availability Zone Availability Zone
Replace with a regional service
Simplified operations
Amazon Simple
Queue Service (SQS)
Reduced costs
![Page 25: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/25.jpg)
Queuing on AWS
Setup & manage instances
Install & configure queuing middleware
Set up persistent message store
Implement clustering across AZs for HA
Implement queue monitoring systems
Implement queuing in applications
![Page 26: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/26.jpg)
Amazon SQS
Create a queue
HTTP PUT to place messages
HTTP GET (long polling) to pull messages
Queuing on AWS
Setup & manage instances
Install & configure queuing middleware
Set up persistent message store
Implement clustering across AZs for HA
Implement queue monitoring systems
Implement queuing in applications
![Page 27: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/27.jpg)
AWS application services
![Page 28: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/28.jpg)
Relational Database Service
DynamoDB
SimpleDB
Databases Middleware Frameworks Analytics
Simple Queuing Service
Simple Notification Service
Simple Workflow
Elastic MapReduce
Simple Email Service
CloudSearch ElastiCache
CloudWatch
![Page 29: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/29.jpg)
Relational Database Service
DynamoDB
SimpleDB
Databases Middleware Frameworks Analytics
Simple Queuing Service
Simple Notification Service
Simple Workflow
Elastic MapReduce
Simple Email Service
CloudSearch ElastiCache
CloudWatch
![Page 30: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/30.jpg)
Amazon Relational Database Service
![Page 31: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/31.jpg)
Source: Forrester
Backup, recovery
load and unload
Security planning
License training
Script automation
Installation, upgrade,
patching, migration Performance
and tuning
![Page 32: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/32.jpg)
Frequent server upgrades Storage upgrades
Backup and recovery
Software upgrades
Patching
Hardware crash
Query construction
Query optimization
Configuration
Migration
Schema design
![Page 33: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/33.jpg)
Frequent server upgrades Storage upgrades
Backup and recovery
Software upgrades
Patching
Hardware crash
Query construction
Query optimization
Configuration
Migration
Schema design
Focus on these things
![Page 34: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/34.jpg)
Frequent server upgrades Storage upgrades
Backup and recovery
Software upgrades
Patching
Hardware crash
Query construction
Query optimization
Configuration
Migration
Schema design
Instead of these
![Page 35: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/35.jpg)
Near zero administration
Painless patching, automatic upgrades
Cloudwatch monitoring, metric alarms
One click. High Availability.
![Page 36: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/36.jpg)
![Page 37: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/37.jpg)
One click. High availability with Multi-AZ
Automated deployment across multiple AZs
Synchronous replication from master to replica
Automatic fail-over; replica promoted to master
Test fail-over
![Page 38: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/38.jpg)
Push-button scale, high performance
Scale storage from 5Gb to 1Tb of storage
Scale instance from small to 4XL (better I/O)
Add Read Replicas with asynchronous replication
Add ElastiCache for performance
![Page 39: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/39.jpg)
Relational Database Service
Database-as-a-Service
No need to install or manage database instances
Scalable and fault tolerant configurations
Feature Details
Platform support Create MySQL, SQL Server and Oracle RDBMS
Preconfigured Get started instantly with sensible default settings
Automated patching Keep your database platform up to date automatically
Backups Automatic backups and point in time recovery and full DB backups
Backups Volumes can be snapshotted for point in time restore
Failover Automated failover to slave hosts in event of a failure
Replication Easily create read-replicas of your data and seamlessly replicate data across availability zones
![Page 40: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/40.jpg)
RDBMS on AWS
Setup & manage instances
Install & configure database platform
Configure backups
Implement master-slave for HA
Implement read-replicas for performance
Manage maintenance updates
![Page 41: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/41.jpg)
RDS
Create RDS instance
Select Multi-AZ
Choose backup period
Choose maintenance windows
RDBMS on AWS
Setup & manage instances
Install & configure database platform
Configure backups
Implement master-slave for HA
Implement read-replicas for performance
Manage maintenance updates
![Page 42: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/42.jpg)
Amazon Relational Database Service (Amazon RDS) databases stores forum threads, site content, and project configuration
data.
High availability Multi-AZ database deployment to handle live game metadata and user-generated content.
Enterprise-grade fault tolerance for protecting customer data.
By managing time-consuming database administration tasks,
Amazon RDS allows SEGA to focus on business critical applications.
![Page 43: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/43.jpg)
Amazon DynamoDB
![Page 44: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/44.jpg)
Requirement: predictable, consistent
performance
Scalability
Pe
rfo
rma
nce
DynamoDB & NoSQL
Database services
![Page 45: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/45.jpg)
Requirement: predictable, consistent
performance
Reality: performance
degrades with scale
Scalability
Pe
rfo
rma
nce
DynamoDB & NoSQL
Database services
![Page 46: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/46.jpg)
Hardware provisioning
Data sharding
Data caching
Cluster management
Fault management
Requirement: predictable, consistent
performance
Reality: performance
degrades with scale
Scalability
Pe
rfo
rma
nce
DynamoDB & NoSQL
Database services
![Page 47: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/47.jpg)
DynamoDB Provisioned read/write performance per table
Predictable high performance scaled via console
or API
Dial it up
![Page 48: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/48.jpg)
Low provisioned throughput
Table Partition
SSD
Region
Illustrative diagram only
![Page 49: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/49.jpg)
Low provisioned throughput
Table Partition
SSD
Region
Illustrative diagram only
Availability Zone Availability Zone Availability Zone
Replica Partition
SSD
Replica Partition
SSD
![Page 50: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/50.jpg)
Low provisioned throughput
Table Partition
SSD
Region
Illustrative diagram only
![Page 51: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/51.jpg)
Increased provisioned throughput
Illustrative diagram only
Region
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
Table Partition
SS
D
![Page 52: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/52.jpg)
High provisioned throughput
Region
Illustrative diagram only
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
Table Partition
![Page 53: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/53.jpg)
Provisioned throughput NoSQL
database
Fast, predictable performance
Fully distributed, fault tolerant
architecture
Feature Details
Provisioned throughput
Dial up or down provisioned read/write capacity
Predictable performance
Average single digit millisecond latencies from SSD backed infrastructure
Strong consistency Be sure you are reading the most up to date values
Fault tolerant Data replicated across availability zones
Monitoring Integrated to Cloud Watch
Secure Integrates with AWS Identity and Access Management (IAM)
Elastic MapReduce
Integrates with Elastic MapReduce for complex analytics on large datasets
DynamoDB
![Page 54: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/54.jpg)
NoSQL on AWS
Perform capacity planning and determine nodes required
Define PIOPS EBS volumes & instance types
Setup & manage instances
Install NoSQL database software
Configure replica sets
Implement auto-scaling
Define tables
Define key strategy
![Page 55: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/55.jpg)
DynamoDB
Define tables
Define key strategy
Choose performance level
NoSQL on AWS
Perform capacity planning and determine nodes required
Define PIOPS EBS volumes & instance types
Setup & manage instances
Install NoSQL database software
Configure replica sets
Implement auto-scaling
Define tables
Define key strategy
![Page 56: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/56.jpg)
![Page 57: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/57.jpg)
“AWS gave us the flexibility to bring a massive amount of capacity online in a short period of
time and allowed us to do so in an operationally straightforward way.
AWS is now Shazam’s cloud provider of choice,”
Jason Titus,
CTO
DynamoDB: over 500,000 writes per
second
Amazon EMR: more than 1 million writes
per second
![Page 58: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/58.jpg)
Amazon Simple Workflow
![Page 59: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/59.jpg)
Flow framework for simplification of
cross system task coordination
Long running transaction state and task
distribution
Task A
Task B
(Auto-scaling)
Task C
2
3
1
Simple Workflow
![Page 60: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/60.jpg)
New Order
Update
Inventory $ or €?
Update $
account
Update €
account
Ship
order
Send
A typical business workflow…
Multiple steps
Multiple decision points
![Page 61: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/61.jpg)
New Order
Update
Inventory $ or €?
Update $
account
Update €
account
Ship
order
Send
System A System C
System B
System D
System E
A typical business workflow…
Multiple steps
Multiple decision points
Heterogeneous systems
![Page 62: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/62.jpg)
A typical business workflow…
New Order
Update
Inventory $ or €?
Update $
account
Update €
account
Ship
order
Send
System A System C
System B
System D
System E
State
Process logic Middleware
State managed in end systems
Process logic embedded in applications
Complex queuing, message ordering, de-duplication and dependencies
Scaling and failover of tasks troublesome
![Page 63: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/63.jpg)
New Order
Update
Inventory $ or €?
Update $
account
Update €
account
Ship
order
Send
Decider
Worker
Workflow metadata & state Implemented in Simple Workflow…
Implement a ‘Decider’ with simple, linear decision logic
Processes/tasks in a workflow become ‘Workers’
All state & metadata is handled by highly available and durable AWS ‘Workflow’
![Page 64: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/64.jpg)
Amazon Simple Workflow Service
(SWF)
![Page 65: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/65.jpg)
Amazon Simple Workflow Service
(SWF)
Decider
Return Result
Get Decision Task
![Page 66: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/66.jpg)
Amazon Simple Workflow Service
(SWF)
Decider
Worker
Get Activity Task
Return Result
Return Result
Get Decision Task
![Page 67: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/67.jpg)
Workflows on AWS
Setup & manage instances
Install & configure workflow middleware
Architect for high availability
Implement workflows in proprietary/complicated
languages
Implement process audit
![Page 68: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/68.jpg)
Simple Workflow
Setup & manage instances
Install & configure workflow middleware
Architect for high availability
Implement workflows in proprietary/complicated
languages
Implement process audit
Implement ‘decider’ in language of choice
Implement task ‘workers’ to consume work
Workflows on AWS
![Page 69: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/69.jpg)
![Page 70: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/70.jpg)
![Page 71: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/71.jpg)
![Page 72: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/72.jpg)
Amazon CloudSearch
![Page 73: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/73.jpg)
select *
from articles
where content
like ‘%cloud%’
![Page 74: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/74.jpg)
1 million hits a day? 5 million items?
![Page 75: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/75.jpg)
Finding things is trickier than first appears
![Page 76: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/76.jpg)
![Page 77: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/77.jpg)
![Page 78: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/78.jpg)
+
![Page 79: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/79.jpg)
Results & Ranking
Facets
Fielded
Anatomy of a search
Tokenization
![Page 80: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/80.jpg)
search-mydomain.us-east-1.
cloudsearch.amazonaws.com
doc-mydomain.us-east-1.
cloudsearch.amazonaws.com
Endpoints
![Page 81: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/81.jpg)
Facets
Query String
Results
Testing
![Page 82: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/82.jpg)
http://search-toilets-in-oz-m6gko2xujhti47jv5w62eldyoy.us-
east-1.cloudsearch.amazonaws.com/2011-02-01/search
?q=True+Blue
&return-fields= address1
%2Caddressnote
%2Cf__name
%2Clastupdatedate
%2Clatitude
%2Clongitude
%2Cnotes
%2Cpostcode
%2Ctoiletdetails_id
%2Ctoileturl
%2Ctext_relevance
Domain
Query String
Search Options
Integration
![Page 83: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/83.jpg)
{
"rank":"-text_relevance",
"match-expr":"(label 'True Blue')",
"hits":{"found":1,
"start":0,
"hit":[{"id":"toiletdetails_csv_41",
"data":{"address1":["51-59 Samwell Street"],
"addressnote":["3 male and 3 female toilets"],
"f__name":["Croydon True Blue Visitor"],
"lastupdatedate":["2010-03-04Z"],
"latitude":["-18.204249"],
"longitude":["142.244534"],
"notes":[],
"postcode":["4871"],
"text_relevance":["309"],
"toiletdetails_id":["40”]
}]
},
"info”{ "rid":"ccd66a5219f938d295e4326391ce31e1359fb7bc306564",
"time-ms":3,
"cpu-time-ms":0}
}
Results Search
Integration
![Page 84: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/84.jpg)
Scalable
![Page 85: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/85.jpg)
Small Instance Partition 1
Copy 1
Time: 1800h
<80% CPU
Elastic Search
Time
Requests
![Page 86: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/86.jpg)
Small Instance Partition 1
Copy 1
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
>80% CPU
Small Instance Partition 1
Copy 2
Time: 2000h
![Page 87: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/87.jpg)
Small Instance Partition 1
Copy 1
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
>80% CPU
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 3
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Time: 2200h
![Page 88: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/88.jpg)
Small Instance Partition 1
Copy 1
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 3
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2 <30% CPU
Small Instance Partition 1
Copy 1
Time: 0000h
![Page 89: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/89.jpg)
Small Instance Partition 1
Copy 1
Cost savings
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 3
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 1
Traditional required capacity
![Page 90: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/90.jpg)
Small Instance Partition 1
Copy 1
Cost savings
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 3
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 1
Traditional required capacity
Elastic Capacity
![Page 91: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/91.jpg)
Small Instance Partition 1
Copy 1
Cost savings
Elastic Search
Time
Requests
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 3
Small Instance Partition 1
Copy 1
Small Instance Partition 1
Copy 2
Small Instance Partition 1
Copy 1
Traditional required capacity
Elastic Capacity
Savings Savings
![Page 92: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/92.jpg)
Cloud Search
Search Instance Partition 1
Copy 1
Search Instance Partition 1
Copy 2
Search Instance Partition 1
Copy n
Traffic Request
volume & complexity
![Page 93: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/93.jpg)
Data Document quantity & size
Search Instance Partition 1
Copy 1
Search Instance Partition 2
Copy 1
Search Instance Partition n
Copy 1
Cloud Search
![Page 94: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/94.jpg)
Search Instance Partition 1
Copy 1
Search Instance Partition 2
Copy 1
Search Instance Partition n
Copy 1
Search Instance Partition 1
Copy 2
Search Instance Partition 2
Copy 2
Search Instance Partition n
Copy 2
Search Instance Partition 1
Copy n
Search Instance Partition 2
Copy n
Search Instance Partition n
Copy n
Traffic Request
volume & complexity
Data Document quantity & size Cloud
Search
![Page 95: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/95.jpg)
Search on AWS
Perform capacity planning on size of indexes
Setup & manage required number of instances
Install & configure search software such as Solr across
cluster
Manage cluster partitioning and size over time
Implement monitoring
![Page 96: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/96.jpg)
Cloud Search
Perform capacity planning on size of indexes
Setup & manage required number of instances
Install & configure search software such as Solr across
cluster
Manage cluster partitioning and size over time
Implement monitoring
Create search domain in console
Upload documents
Retrieve results
Search on AWS
![Page 97: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/97.jpg)
![Page 98: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/98.jpg)
![Page 99: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/99.jpg)
Amazon Elastic MapReduce
![Page 100: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/100.jpg)
1 instance for 100 hours =
100 instances for 1 hour
![Page 101: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/101.jpg)
Small instance = $8
![Page 102: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/102.jpg)
![Page 103: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/103.jpg)
Managed, elastic Hadoop cluster
Integrates with S3 & DynamoDB
Leverage Hive & Pig analytics scripts
Integrates with instance types such as spot
Elastic MapReduce
Feature Details
Scalable Use as many or as few compute instances running Hadoop as you want. Modify the number of instances while your job flow is running
Integrated with other services
Works seamlessly with S3 as origin and output. Integrates with DynamoDB
Comprehensive Supports languages such as Hive and Pig for defining analytics, and allows complex definitions in Cascading, Java, Ruby, Perl, Python, PHP, R, or C++
Cost effective Works with Spot instance types
Monitoring Monitor job flows from with the management console
![Page 104: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/104.jpg)
But what is it?
![Page 105: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/105.jpg)
A framework Splits data into pieces Lets processing occur
Gathers the results
![Page 106: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/106.jpg)
Very large dataset
(e.g TBs)
![Page 107: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/107.jpg)
Lots of instances of ‘X’
Very large dataset
(e.g TBs)
![Page 108: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/108.jpg)
Split the log into
many small pieces
Lots of instances of ‘X’
Very large dataset
(e.g TBs)
![Page 109: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/109.jpg)
Split the log into
many small pieces
Process in an EMR cluster Lots of
instances of ‘X’
Very large dataset
(e.g TBs)
![Page 110: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/110.jpg)
Split the log into
many small pieces
Process in an EMR cluster
Aggregate the results
from all the nodes
Lots of instances of ‘X’
Very large dataset
(e.g TBs)
![Page 111: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/111.jpg)
Aggregate view of ‘X’
Lots of instances of ‘X’
Split the log into
many small pieces
Process in an EMR cluster
Aggregate the results
from all the nodes
Very large dataset
(e.g TBs)
![Page 112: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/112.jpg)
Insight in a fraction of the time
Very large dataset
(e.g TBs)
Aggregate view of ‘X’
![Page 113: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/113.jpg)
1 instance for 100 hours =
100 instances for 1 hour
![Page 114: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/114.jpg)
Small instance = $8
![Page 115: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/115.jpg)
1 instance for 1,000 hours =
1,000 instances for 1 hour
![Page 116: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/116.jpg)
Small instance = $80
![Page 117: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/117.jpg)
Input data
S3
![Page 118: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/118.jpg)
Elastic MapReduce
Code
Input data
S3
![Page 119: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/119.jpg)
Elastic MapReduce
Code Name node
Input data
S3
![Page 120: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/120.jpg)
Elastic MapReduce
Code Name node
Input data
S3
Elastic cluster
![Page 121: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/121.jpg)
Elastic MapReduce
Code Name node
Input data
S3
Elastic cluster
HDFS
![Page 122: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/122.jpg)
Elastic MapReduce
Code Name node
Input data
S3
Elastic cluster
HDFS Queries
+ BI Via JDBC, Pig, Hive
![Page 123: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/123.jpg)
Elastic MapReduce
Code Name node
Output S3 + SimpleDB
Input data
S3
Elastic cluster
HDFS Queries
+ BI Via JDBC, Pig, Hive
![Page 124: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/124.jpg)
Output S3 + SimpleDB
Input data
S3
![Page 125: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/125.jpg)
Hadoop on AWS
Configure & manage EC2 instances
Setup Namenode
Setup Datanodes
Create HDFS file system
Setup extras (Hive, Pig etc)
Prepare data
Execute job
![Page 126: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/126.jpg)
Elastic MapReduce Hadoop on AWS
Configure & manage EC2 instances
Setup Namenode
Setup Datanodes
Create HDFS file system
Setup extras (Hive, Pig etc)
Prepare data
Execute job
Specify desired cluster size
Execute job
![Page 127: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/127.jpg)
Features powered by Amazon Elastic MapReduce:
People Who Viewed this Also Viewed
Review highlights Auto complete as you type on search
Search spelling suggestions Top searches
Ads
200 Elastic MapReduce jobs per day Processing 3TB of data
![Page 128: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/128.jpg)
"With Amazon Elastic MapReduce, there
was no upfront investment in hardware,
no hardware procurement delay, and no
need to hire additional operations staff.
Because of the flexibility of the platform,
our first new online advertising campaign
experienced a 500% increase in return on
ad spend from a similar campaign a year
before.”
Mark Taylor, Program Director
Elastic MapReduce
![Page 129: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/129.jpg)
Where to go next
![Page 130: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/130.jpg)
http://aws.typepad.com
http://aws.amazon.com/whitepapers
![Page 131: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/131.jpg)
$100 of credits That’s enough to run an Amazon Elastic Compute Cloud (Amazon EC2) Standard Small Instance for approximately 600 hours Or ~600 EC2 Standard Small Instances for 1 hour
Powerof60.com
![Page 132: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/132.jpg)
Summary
![Page 133: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/133.jpg)
AWS is more than compute & storage
Databases, middleware and complex frameworks as services
Remove the operational burden from running common software services
Application services gives access scalable, sophisticated software without the overheads
![Page 134: Journey Through the AWS Cloud; Application Services](https://reader033.fdocuments.us/reader033/viewer/2022051412/54c638ad4a7959e43f8b4661/html5/thumbnails/134.jpg)
aws.amazon.com