AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS Public Sector Summit...
-
Upload
amazon-web-services -
Category
Technology
-
view
450 -
download
1
Transcript of AWS Snowball: Accelerating Large-Scale Data Ingest Into the AWS Cloud | AWS Public Sector Summit...
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snowball: Accelerating Large-Scale Data Ingest into AWS
Bill Vass, VP, Object Storage and Streaming Services, AWS
June 20, 2016
Loading large datasets into AWS
Large customer:
“We want to move 250 petabytes into AWS, but it would take years to do that over the network. How can we upload our data faster into AWS?”
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Loading large datasets into AWS
Government and enterprise customer requirements • Secure• Fast• Simple• Petabyte scale• Inexpensive
Largedatasets
© 2016, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Loading large datasets into AWS
Complex customer requirements • Strong chain of custody• Cryptography key not on device • Operate where bandwidth is limited• Convert legacy file system to object storage• Retire legacy storage quickly for improved ROI
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snowball—petabyte scale data transport service
E Ink shipping label
Ruggedizedcase; can take200 G impact
All data encrypted end-to-end80 TB
10 G network
Rain and dust resistant
Tamperresistant
caseand
electronics
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snowball—data transport service for large datasets
Customerdataset
loaded intoAmazon S3
Large customerdataset
Console
Console
End-to-end custody of the data
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
10 Snowballs connected to your DMZ
=The throughput of a 100 gigabyte
network connection© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Snowball—petabyte scale data transport service
How fast is a truck full of drives?
• Less than 1 day to transfer 80 TB via a 10 G connection with Snowball, less than 1 week including shipping.
• Number of days to transfer 80 TB via the Internet at typical utilizations:
Internet Connection SpeedUtilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 19 38 63 12650% 9 19 32 6375% 6 13 21 42
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How fast is a truck full of drives?
• Less than 1 day to transfer 240 TB via 4x10 G connections with 5 Snowballs, less than 1 week including shipping.
• Number of days to transfer 240 TB via the internet at typical utilizations:
Internet Connection SpeedUtilization 1Gbps 500Mbps 300Mbps 150Mbps
25% 95 190 316 63250% 47 95 158 31675% 32 63 105 211
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How is my data transported securely?
• 256-bit encryption• Data is encrypted by the
Snowball client before it reaches the Snowball
• Keys are managed by AWS KMS and are never sent to the Snowball
• Built to handle a “nation-state” cyber attack
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How is my data transported securely?
• Strong chain of custody• Tamper resistant case• Tamper resistant electronics
(TPM)• Each Snowball is erased
according to NIST 800-88 media sanitization guidelines between every job
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What does it cost?
• $250/job plus shipping• Includes 10 days to fill the device at your site• $15/day after the tenth day on site• Standard S3 charges apply• $0.03/GB to transfer data out• $0.00/GB to transfer data in
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ben Wilson (CTO of GE Oil and Gas):“PIGs and Snowballs—a match made in heaven! AWS Snowball 25 TB of pipeline PIG data to be managed at AWS. That is our GE PIG we pulled some of the data from. It’s always fun to try new AWS features and try to break them!”
Recent launches
• 80 TB Snowball (60% capacity increase)• Export• International
- Dublin- Sydney- GovCloud- San Francisco
• Commvault integration (more storage partners coming)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Roadmap
• Availability in all regions by end of 2016• Amazon EC2, Amazon RDS, and Amazon Direct Connect
migration• SDK (S3 interface)• Clustered Snowballs for durable on-premises storage tier• Mil Spec and Marine
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!