Data Replication Options in AWS (ARC302) | AWS re:Invent 2013
(ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014
-
Upload
amazon-web-services -
Category
Technology
-
view
2.168 -
download
2
Transcript of (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014
![Page 1: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/1.jpg)
November 13, 2014 | Las Vegas, NV
![Page 2: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/2.jpg)
![Page 3: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/3.jpg)
![Page 4: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/4.jpg)
![Page 5: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/5.jpg)
“Everything fails, all the time.”
![Page 6: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/6.jpg)
500
![Page 7: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/7.jpg)
![Page 8: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/8.jpg)
![Page 9: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/9.jpg)
Black Friday balance: Submarino down, Americanas down, Walmart
down, Magazine Luiza with problems, FNAC down. #BlackFridayFail
![Page 10: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/10.jpg)
![Page 11: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/11.jpg)
Magazine Luiza is doing a clearance sale for New Year. It’s a nice sale,
but so good that their website is down
![Page 12: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/12.jpg)
–120 orders / minute
![Page 13: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/13.jpg)
XA
Going beyond the traditional high availability concept
eXtreme availability
![Page 14: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/14.jpg)
XA
![Page 15: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/15.jpg)
![Page 16: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/16.jpg)
![Page 17: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/17.jpg)
global infrastructure
![Page 18: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/18.jpg)
beginning
![Page 19: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/19.jpg)
processes
developers
architects
![Page 20: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/20.jpg)
any failure
![Page 21: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/21.jpg)
Graceful
![Page 22: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/22.jpg)
Micro services m-SOA
![Page 23: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/23.jpg)
availability needs
![Page 24: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/24.jpg)
AWS
![Page 25: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/25.jpg)
![Page 26: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/26.jpg)
Amazon EC2 Amazon EC2
Elastic Load Balancing
Availability
Zone A
Availability
Zone B
Amazon S3
Static
Website
Amazon
CloudFront
User
Amazon
Route 53
Secondary
Sta
tic
Assets
Primary + Health Checks
Auto
Scaling
Group
us-east
![Page 27: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/27.jpg)
![Page 28: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/28.jpg)
Amazon S3
Static
Website
Amazon
CloudFront
User
Amazon
Route 53
SecondaryS
tatic
Assets
Active/Active with Latency Based Routing + Health Checks
EC2 EC2
Elastic Load Balancing
Availability
Zone A
Availability
Zone B
us-east-1
EC2 EC2
Elastic Load Balancing
Availability
Zone A
Availability
Zone B
us-west-1
![Page 29: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/29.jpg)
Amazon S3
Static
Website
Amazon
CloudFront
User
Amazon
Route 53
SecondaryS
tatic
Assets
Active/Active with Latency Based Routing + Health Checks
EC2 EC2
Elastic Load Balancing
Availability
Zone A
Availability
Zone B
us-east-1
EC2 EC2
Elastic Load Balancing
Availability
Zone A
Availability
Zone B
us-west-1
![Page 30: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/30.jpg)
EC2
us-east-2
EC2
us-west-2
![Page 31: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/31.jpg)
![Page 32: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/32.jpg)
![Page 33: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/33.jpg)
![Page 34: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/34.jpg)
Amazon
CloudFront
User
Amazon
Route 53
Sta
tic
Assets
Active/Active with Latency Based Routing + Health Checks
EC2 EC2
Elastic Load Balancing
Amazon
S3
Static
Content
EC2 EC2
Elastic Load Balancing
CA
CH
EW
EB
SE
RV
ER
Availability
Zone A
Availability
Zone B
Availability
Zone A
Availability
Zone B
sa-east-1 us-east-1
CA
CH
EW
EB
SE
RV
ER
CA
CH
EW
EB
SE
RV
ER
CA
CH
EW
EB
SE
RV
ER
![Page 35: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/35.jpg)
![Page 36: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/36.jpg)
![Page 37: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/37.jpg)
Amazon RDS Multi-AZ
(with read replicas)
Amazon DynamoDB and
Amazon S3 have built-in
HA
Availability Zone A
Availability Zone B
![Page 38: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/38.jpg)
![Page 39: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/39.jpg)
Amazon RDS read replicas rock!
Availability Zone A
Availability Zone B
Amazon RDS
Read Replica
Amazon RDS
Read Replica
Availability Zone A
Availability Zone B
Amazon RDS Master
Amazon RDS
Read Replica Amazon RDS
Read Replica
Amazon RDS Standby
Availability Zone C
Amazon RDS
Read Replica
us-east-1 us-west-1
![Page 40: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/40.jpg)
Amazon RDS read replicas rock!
Availability Zone A
Availability Zone B
Amazon RDS
Read Replica
Amazon RDS
Read Replica
Availability Zone A
Availability Zone B
Amazon RDS Master
Amazon RDS
Read Replica Amazon RDS
Read Replica
Amazon RDS Standby
Availability Zone C
Amazon RDS
Read Replica
us-east-1 us-west-1
![Page 41: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/41.jpg)
Amazon RDS read replicas rock!
Availability Zone A
Availability Zone B
Amazon RDS
Read Replica
Amazon RDS
Read Replica
Availability Zone A
Availability Zone B
Amazon RDS Master
Amazon RDS
Read Replica Amazon RDS
Read Replica
Amazon RDS Standby
Availability Zone C
Amazon RDS
Read Replica
us-east-1 us-west-1
Amazon RDS Master
(promoted)
![Page 42: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/42.jpg)
Availability Zone
A
Amazon RDS
us-east-1
Amazon
EC2 as
GoldenGate
Hub for
Source DB
Availability Zone
A
Amazon RDS
Amazon
EC2 as
GoldenGate
Hub for
Source DB
us-west-1
![Page 43: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/43.jpg)
![Page 44: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/44.jpg)
GET
us-east-1 us-west-1
![Page 45: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/45.jpg)
GET
us-east-1 us-west-1
Let’s look
inside here!
![Page 46: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/46.jpg)
Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );
..
AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);
..
KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,
updateStreamsCredentials, ..)
.withMaxRecords(100)
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);
Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,
dynamoDBClient, ..);
Thread t = new Thread(worker);t.start();
![Page 47: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/47.jpg)
Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );
..
AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);
..
KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,
updateStreamsCredentials, ..)
.withMaxRecords(100)
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);
Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,
dynamoDBClient, ..);
Thread t = new Thread(worker);t.start();
![Page 48: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/48.jpg)
Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );
..
AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);
..
KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,
updateStreamsCredentials, ..)
.withMaxRecords(100)
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);
Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,
dynamoDBClient, ..);
Thread t = new Thread(worker);t.start();
![Page 49: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/49.jpg)
Consuming AWSDynamoDBstreamsAdapterClient adapterClient =new AWSDynamoDBstreamsAdapterClient(updateStreamsCredentials, .. );
..
AmazonDynamoDBClient dynamoDBClient =new AmazonDynamoDBClient(dynamoDBCredentials, ..);
..
KinesisClientLibConfiguration workerConfig =new KinesisClientLibConfiguration (.., streamId,
updateStreamsCredentials, ..)
.withMaxRecords(100)
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);
Worker worker =new Worker(recordProcessorFactory, workerConfig, adapterClient,
dynamoDBClient, ..);
Thread t = new Thread(worker);t.start();
![Page 50: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/50.jpg)
Processingpublic class StreamsRecordProcessor implements IRecordProcessor {
..
@Override
public void processRecords(List<Record> records,.. ) {
for(Record record : records) {
if (record instanceof RecordAdapter) {
Record ddbStreamRecord = ((RecordAdapter)
record).getInternalObject();
switch(ddbStreamRecord.getEventName()) {
case "INSERT" : case "MODIFY" :
DemoHelper.putItem(dynamoDBClient, tableName,
ddbStreamRecord.getDynamodb().getNewImage());
break;
...
![Page 51: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/51.jpg)
Processingpublic class StreamsRecordProcessor implements IRecordProcessor {
..
@Override
public void processRecords(List<Record> records,.. ) {
for(Record record : records) {
if (record instanceof RecordAdapter) {
Record ddbStreamRecord = ((RecordAdapter)
record).getInternalObject();
switch(ddbStreamRecord.getEventName()) {
case "INSERT" : case "MODIFY" :
DemoHelper.putItem(dynamoDBClient, tableName,
ddbStreamRecord.getDynamodb().getNewImage());
break;
...
![Page 52: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/52.jpg)
Processingpublic class StreamsRecordProcessor implements IRecordProcessor {
..
@Override
public void processRecords(List<Record> records,.. ) {
for(Record record : records) {
if (record instanceof RecordAdapter) {
Record ddbStreamRecord = ((RecordAdapter)
record).getInternalObject();
switch(ddbStreamRecord.getEventName()) {
case "INSERT" : case "MODIFY" :
DemoHelper.putItem(dynamoDBClient, tableName,
ddbStreamRecord.getDynamodb().getNewImage());
break;
...
![Page 53: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/53.jpg)
Processingpublic class StreamsRecordProcessor implements IRecordProcessor {
..
@Override
public void processRecords(List<Record> records,.. ) {
for(Record record : records) {
if (record instanceof RecordAdapter) {
Record ddbStreamRecord = ((RecordAdapter)
record).getInternalObject();
switch(ddbStreamRecord.getEventName()) {
case "INSERT" : case "MODIFY" :
DemoHelper.putItem(dynamoDBClient, tableName,
ddbStreamRecord.getDynamodb().getNewImage());
break;
...
![Page 54: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/54.jpg)
![Page 55: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/55.jpg)
ERP
Database
Replica
ERP
Database
Replica
Availability
Zone AAvailability
Zone B
sa-east-1
EC2 EC2
WORKERS
COLO
ERP
Database
AWS Direct
Connect
NoSQL NoSQL
WORKERSAmazon Simple
Queue Service
us-east-1 Internet
Same infrastructure with constant
data replication
![Page 56: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/56.jpg)
![Page 57: (ARC311) Extreme Availability for Mission-Critical Applications | AWS re:Invent 2014](https://reader035.fdocuments.us/reader035/viewer/2022062514/55a22f261a28ab7d428b46d2/html5/thumbnails/57.jpg)
Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals