Riga dev day: Lambda architecture at AWS
-
Upload
antons-kranga -
Category
Software
-
view
470 -
download
2
Transcript of Riga dev day: Lambda architecture at AWS
![Page 1: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/1.jpg)
Is Lambda Architecture really a new normal for cloud native apps?
λ+
![Page 2: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/2.jpg)
:~ whoami: Antons Kranga
Full stack developer ~ 15years
Cloud Architect
DevOps evangelist
Innovation Center of Accenture Cloud Platform
Speaker
Marathon runner
![Page 3: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/3.jpg)
Motivation
![Page 4: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/4.jpg)
What is Streaming?We often want to deploy data models based on new data that continuously arrive from the multiple sources
0101
01010101
0101
![Page 5: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/5.jpg)
Challenges
Users expect data will appear immediately after it arrived
Fault tolerant
Distributed data consistency
Scalability (how not to lose data when scale down)
![Page 6: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/6.jpg)
What is “λ”
010 10101 00
01 1101
Speed Layer Batch Layer
new data
master data
realtime view
Serving Layer
view View View…
map-red
query query
realtime view
![Page 7: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/7.jpg)
What is “λ” architectureBatch Layer: Master Data sets and Pre-compute aggregations
• Slow Data Ingestion – minutes to days intervals• Append-only data sets eventually supersedes data
captured in speed layer
Speed Layer: High throughput, near-real-time data ingestion
• Fast Data Ingestion – seconds interval• Concurrent information processing• Retrieval of most recent information
Serving Layer: Provide query capability over the Batch Layer
• Low-latency ad-hoc query• May also provide assess to speed layer views
![Page 8: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/8.jpg)
Why go Cloud Native?
![Page 9: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/9.jpg)
Cloud Provider Lock-In
![Page 10: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/10.jpg)
Avoid “Yak shaving”
![Page 11: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/11.jpg)
Rely on managed services
![Page 12: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/12.jpg)
devops automation
![Page 13: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/13.jpg)
Lower operating costs
![Page 14: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/14.jpg)
Transparent integration with other “Cloud Native” services
![Page 15: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/15.jpg)
AWS Blueprint for Lambda Architectures
https://d0.awsstatic.com/whitepapers/lambda-architecure-on-for-batch-aws.pdfPublished at July 2015
Amazon Kinesis
Amazon Kinesis–enabled app
S3 buckets
Amazon EMR
speed layer
batch layer
emr on serving and merging layer
![Page 16: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/16.jpg)
Data services form AWS
![Page 17: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/17.jpg)
Kinesis
aws region
az1 az2 az3
Lambda
S3 storage
Redshift
consumers
EC2 Instance
EMR
producers
![Page 18: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/18.jpg)
Kinesis
producers
aws region
az1 az2 az3
Lambda
S3 storage
Redshift
consumers
EC2 Instance
EMR
AmazonKinesis kinesis = ......PutRecordRequest putRecord = new PutRecordRequest();putRecord.setStreamName(streamName);putRecord.setData(ByteBuffer.wrap(bytes));putRecord.setSequenceNumberForOrdering(null);...kinesis.putRecord(putRecord);
Producer
![Page 19: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/19.jpg)
Kinesis
aws region
az1 az2 az3
Lambda
S3 storage
Redshift
consumers
EC2 Instance
EMR
AmazonKinesis kinesis = ......PutRecordRequest putRecord = new PutRecordRequest();putRecord.setStreamName(streamName);putRecord.setData(ByteBuffer.wrap(bytes));putRecord.setSequenceNumberForOrdering(null);...kinesis.putRecord(putRecord);
Producer
AmazonKinesisClient kinesisClient = ...GetShardIteratorRequest req = ...req.setStreamName("my-kinesis");req.setShardIteratorType("TRIM_HORIZON");...GetRecordsResult result = kinesisClient.getRecords(req);records = result.getRecords();for (Record record : records) {
... = record.getData();}
Consumerproducers
![Page 20: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/20.jpg)
Kinesis streamsWhat: Enables to build near-real-time data processing applications
Use cases:
• Real time analytics• Log files processing• Reporting
Durability: data streams replicated across 3AZ
![Page 21: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/21.jpg)
Kinesis streamsCost Model:
Shard Hour:• 5 read transaction per second• 2 MB data read per second• 100 write transactions per second• 1 MB data write per second
aprox 12.5USD/Mo
Extended data retention• Up to 7 days
![Page 22: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/22.jpg)
Kinesis streamsNot good when:
• Small scale throughput less than 200KB/sec
• Long term data storage (more than 24H)
![Page 23: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/23.jpg)
LambdaWhat: Lambda allows to write function without having actual server
Use cases:• Real time Stream processing• Tiny ETL• In few cases can replace EC2• Process IaaS Events
Runtimes: Java8, NodeJS, Python
Backed by: provides /tmp for ephemeral storage.
Durability: No maintenance windows, 3 retries before failure
![Page 24: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/24.jpg)
LambdaCost Model:
Requests per function:• GB/seconds• Step 100 millisec • 0.20 USD Mill-Requests; $0.00001667 per GB
![Page 25: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/25.jpg)
LambdaNot good when:
• Timeout 300 sec (cannot be changed)
• Forces developer to think stateless
• Highly dynamic web-sites.
• Competes with t2.nano ($4.75/month)
![Page 26: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/26.jpg)
S3 storage
SNS
consumers
Kinesis
Lambda
…Lambda
S3 storage
SNS
consumers
Kinesis
…
myApp.ZIP
Java8PythonNodeJS
![Page 27: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/27.jpg)
EMRWhat: Managed service of Apache Hadoop
Use cases:• MapRed data processing• Large data ETL jobs• Data movement• Log processing and analytics
Backed by: 1 or cluster of EC2 instances
Durability: on storage level provides by S3
See more:https://media.amazonwebservices.com/AWS_Amazon_EMR_Best_Practices.pdf
![Page 28: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/28.jpg)
EMRCost Model:
• Charges apply per EC2 sizes model
• S3 storage charges applies (0.03 GB/Mo)
![Page 29: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/29.jpg)
EMRNot good when:
• Small to Medium data sets
• ACID (atomicity, consistency, isolation, durability)
• Competes with RDS: Dynamo DB, Aurora DB
![Page 30: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/30.jpg)
S3 What: Highly fully managed persistent storage
• Static content web sites
• File storage (primarily for reading)
• Archives storage
Backed by: covered by AWS S3 SLA
Durability: storage: 99.999999999%; availability: 99.99%
![Page 31: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/31.jpg)
S3Cost Model: GB/Mo
• Standard Storage: $0.03 GB/Mo
• Infrequent Access Storage: $0.0125 GB/Mo
• Glacier Storage: $0.007 GB/Mo
![Page 32: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/32.jpg)
S3Not good when:
• S3 write can be slow
• Glacier can restore up to 5% of storage per months
![Page 33: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/33.jpg)
RedshiftWhat: Petabytes scale Data Warehouse as managed service
• Data warehouse (OLAP)
• BI and ETL
• Store large historical data
Backed by: AWS provides automatic data backup
Durability: on storage level provides by S3
Scaling: Start with 160GB node and then you can scale
![Page 34: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/34.jpg)
RedshiftCost Model:
• Charges apply per EC2 sizes model
• S3 storage charges applies (0.03 GB/Mo)
![Page 35: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/35.jpg)
RedshiftNot good when:
• OLTP (On-line transaction processing)
• Unstructured data
• Blob storage
![Page 36: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/36.jpg)
Kinesis
shard
shard
shard
producer
batch layer
speed layerec2
S3 Bucket Map Red
Process Stream
serving layer
View
DynamoDB
Primer Lambda(every hour)
![Page 37: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/37.jpg)
Kinesis
shard
shard
shard
producer
batch layer
speed layerec2
S3 Bucket Map Red
Process Stream
serving layer
View
DynamoDB
Primer Lambda(every hour)
computation per hour
Lambda(every hour)
h0 h1 h2 h3
batch layerSpeed layer
t
![Page 38: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/38.jpg)
Kinesis
shard
shard
shard
producer
batch layer
speed layerfec2
S3 Bucket Map Red
Process Stream
serving layer
View
DynamoDB
Primer Lambda(every hour)
Lambda(every hour)
Presentation Layer
JS appLambda
![Page 39: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/39.jpg)
Lesions learned
![Page 40: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/40.jpg)
It is better but not simple
![Page 41: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/41.jpg)
Not everything is automated
![Page 42: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/42.jpg)
Questions?
![Page 43: Riga dev day: Lambda architecture at AWS](https://reader034.fdocuments.us/reader034/viewer/2022042600/587063c41a28ab48378b47e7/html5/thumbnails/43.jpg)
Thank you!