Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean,...
Transcript of Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean,...
![Page 1: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/1.jpg)
©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect
![Page 2: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/2.jpg)
AWS storage options
• Scalable object storage • Inexpensive archive storage • Persistent block storage • Shared file system • Turnkey gateway solution
![Page 3: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/3.jpg)
We are constantly producing more data
![Page 4: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/4.jpg)
From all types of industries
![Page 5: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/5.jpg)
Object Storage
![Page 6: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/6.jpg)
Amazon S3 Simple Storage Service
![Page 7: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/7.jpg)
2012 2013 2014 2015
95% year-over-year increase in data transfer to and from S3 (Q4 2014 vs Q4 2013, not including Amazon use)
S3 usage
![Page 8: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/8.jpg)
99.999999999% durability
![Page 9: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/9.jpg)
Reduced redundancy option 99.99% saves ~20%
![Page 10: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/10.jpg)
S3 resources: buckets and objects
![Page 11: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/11.jpg)
S3 website: static content
![Page 12: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/12.jpg)
1 PB raw storage
800 TB usable storage
600 TB allocated storage
400 TB application data
S3 capacity pricing—pay only for what you use!
Amazon S3
![Page 13: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/13.jpg)
S3 continuous cost reduction • Available through 11 regions globally • Priced at per GB-month rates • 8 price reductions since launch • 51% average S3 capacity fee
reduction on 4/1/2014
• TCO: comparing on-premises to S3 – Can be challenging for some
customers – We can help!
![Page 14: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/14.jpg)
S3
Amazon EMR
![Page 15: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/15.jpg)
Amazon S3—New Features
![Page 16: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/16.jpg)
S3 event notifications Delivers notifications to Amazon SNS, Amazon SQS, or AWS Lambda when events occur in S3
S3
Events
SNS topic
SQS queue
Lambda function
Notifications
Notifications
Notifications
Foo() { … }
![Page 17: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/17.jpg)
• Preserve, retrieve, and restore every version of every object stored in your bucket
• S3 automatically adds new versions and preserves deleted objects with delete markers unless an explicit versioned DELETE operation is made
• Easily control the number of versions kept by using lifecycle expiration policies
• Easy to turn on in the AWS Management Console
S3 versioning
![Page 18: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/18.jpg)
S3 cross-region replication Automated, fast, and reliable asynchronous replication of data across AWS regions
Source (Ireland)
• Only replicates new PUTs. Once S3 is configured, all new uploads into a source bucket will be replicated
• Entire bucket or prefix based
• 1:1 replication between any 2 regions
• Versioning required
Use cases: • Compliance—store data hundreds of miles apart • Lower latency—distribute data to regional customers • Security—create remote replicas managed by separate AWS accounts
Destination (Frankfurt)
![Page 19: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/19.jpg)
S3 use cases
• Web-scale storage capacity and performance for web applications
• Single-origin store with delivery through Amazon CloudFront
• Staging and persistent store for Big Data applications
• Storage target for backup and active archive
![Page 20: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/20.jpg)
Amazon Glacier Low-cost archiving service
![Page 21: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/21.jpg)
$0.01 per GB-month
![Page 22: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/22.jpg)
$120 per TB/year
![Page 23: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/23.jpg)
99.999999999% durability
![Page 24: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/24.jpg)
3–5 hours data retrieval
![Page 25: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/25.jpg)
Storage costs vs.
retrieval costs
![Page 26: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/26.jpg)
Amazon Glacier benefits
• Reduce cost for long-term archiving • Leverage unlimited storage capacity • Replace tape museums • Improve durability
![Page 27: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/27.jpg)
Amazon S3–Amazon Glacier integration Policy-based archiving service
![Page 28: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/28.jpg)
S3 Lifecycle Policies
→• Key prefix “logs/”
• Transitions objects to Glacier 30 days after creation date
• Delete 365 days after creation date
<LifecycleConfiguration> <Rule>
<ID>archive-in-30-days</ID <Prefix>logs/</Prefix>
<Status></Status> <Transition>
<Days>30</Days>
<StorageClass>GLACIER</StorageClass> </Transition>
<Expiration>
<Days>365</Days> </Expiration>
</Rule> </LifecycleConfiguration>
![Page 29: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/29.jpg)
SoundCloud—leveraging Amazon Glacier for audio transcoding • World’s leading social sound platform • Audio files must be transcoded and stored
in multiple formats
S3 Amazon Glacier
![Page 30: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/30.jpg)
Use cases for S3 lifecycle policies
• Data tiering in the cloud
• Manage object versioning for logical data protection
• Deletion of data from Amazon Glacier by policy
![Page 31: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/31.jpg)
Block Storage
![Page 32: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/32.jpg)
Instance storage
![Page 33: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/33.jpg)
Amazon EBS Elastic Block Storage
![Page 34: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/34.jpg)
![Page 35: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/35.jpg)
![Page 36: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/36.jpg)
![Page 37: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/37.jpg)
1 GB 16 TB
Ú
![Page 38: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/38.jpg)
EBS
General Purpose (SSD)
Up to 16 TB
10,000 IOPS
Up to 160 Mbps
Provisioned IOPS (SSD)
Up to 16 TB
20,000 IOPS
Up to 320 Mbps
![Page 39: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/39.jpg)
Price Performance EBS
Magnetic General Purpose Provisioned IOPS
Use cases Infrequent data access
Boot volumes Small to med DBs
Dev and Test
I/O intensive Relational DBs
NoSQL DBs
Storage media Magnetic disk-backed SSD-backed SSD-backed
Max IOPS 40–200 IOPS 10,000 IOPS 20,000 IOPS Latency (random read) 20–40 ms 1–2 ms 1–2 ms
Availability Designed for 99.999% Designed for 99.999% Designed for 99.999%
Price $.05/GB-month $.05/million I/O $.10/GB-month $.125/GB-month
$.065/provisioned IOPS
![Page 40: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/40.jpg)
Amazon EBS snapshots
![Page 41: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/41.jpg)
![Page 42: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/42.jpg)
Amazon EBS use cases
• Persistent block storage for Amazon EC2 • Transactional workloads • File system for an instance—NTFS, ExtFS, and so on
![Page 43: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/43.jpg)
Introducing Amazon Elastic File System Cloud-based file system
![Page 44: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/44.jpg)
What is Amazon EFS?
• Fully managed file system for EC2 instances • Provides standard file system semantics • Works with standard operating system APIs • Sharable across thousands of instances • Elastically grows to petabyte scale • Delivers performance for a wide variety of workloads • Highly available and durable • NFS v4–based
![Page 45: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/45.jpg)
Amazon EFS is designed for a broad range of use cases, such as…
• Content repositories • Development environments • Home directories • Web server farms • Big Data
![Page 46: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/46.jpg)
Amazon EFS Benefits...
… to app owners and developers?
… to your business?
• Easy to move existing code, applications, and tools used today with existing NFS servers to the AWS cloud
• Simple shared file storage solution for new cloud-native applications
• Predictable pricing with no up-front investment • Increased agility • Spend less time managing file storage and more time
focusing on your business
… to IT administrators?
• Eliminates need to manage and maintain file system storage at scale
![Page 47: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/47.jpg)
AWS Storage Gateway
![Page 48: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/48.jpg)
AWS Storage Gateway Your on-ramp to AWS cloud storage: • Back up into S3 • DR on EC2 • Archive into Amazon Glacier • iSCSI or VTL interface
![Page 49: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/49.jpg)
Internet Web Services API
HTTP(S)
S3 S3
&/or Amazon Glacier
Partner Solutions
![Page 50: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/50.jpg)
Summary: the Amazon storage portfolio
S3 • Object storage: data presented as buckets of objects
EFS • File storage (analogous to NAS): data presented as a file system
Amazon Glacier • Archival storage: data presented as vaults/archives of objects
Storage Gateway
• Back up and archive data into S3 and Amazon Glacier
EBS • Block storage (analogous to SAN): data presented as disk volumes
![Page 51: Intro to AWS: Storage Services - Amazon S3and+r… · Intro to AWS: Storage Services Matt McClean, AWS Solutions Architect . AWS storage options ... • Easy to turn on in the AWS](https://reader030.fdocuments.us/reader030/viewer/2022040409/5ec7f33f714be464251821ce/html5/thumbnails/51.jpg)
LONDON