Download - (SDD416) Amazon EBS Deep Dive | AWS re:Invent 2014

Amazon Elastic Block Store Deep Dive

• Larger and faster volumes

EBS !=

A few definitions…

• IOPS: Input/output operations per second (#)

• Throughput: Read/write rate to storage (MB/s)

• Latency: Delay between request and completion (ms)

• Capacity: Volume of data that can be stored (GB)

• Block size: Size of each I/O (k)

When performance matters, use SSD-backed volumes

• New default volume type for Amazon EBS

• Every volume can burst up to 3,000 IOPS

- Larger volumes can burst for longer periods

• 3 IOPS per GB baseline performance

- Maximum of 3,000 on a 1TB volume

• 99% performance consistency

(2) Max IO credit per bucket is 5.4M

(1) Always accumulating 3 IOPS per GB per

second

(3) You can spend up to

3000 IOPS per second

13

Volume Type Boot Time Access Time OS

GP2 3:31 4:33 Windows Server

2012

Standard 4:30 7:16 Windows Server

2012

GP2 0:36 0:45 CentOS6

Standard 0:57 1:16 CentOS6

40% Reduction in boot times by using General Purpose (SSD)

80% Cost Savings. 50% more peak I/O with

General Purpose (SSD)

• Best for I/O intensive databases that require highest

consistency

• Provision up to 4,000 IOPS per volume

• Supports IOPS to GB ratio of 30

• Designed to offer 99.9% consistency

• Best for cold workloads

• Rarely accessed data that needs always on access

• IOPS: ~100 IOPS steady-state, with best-effort bursts

• Throughput: variable by workload, best effort to 10s of MB/s

• Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10ms

General Purpose (SSD) Provisioned IOPS (SSD) Magnetic

Recommend use casesBoot volumes Small to med DBs Dev and test

I/O intensive Large DBs

Cold storage

Storage media SSD-backed SSD-backed Magnetic-backed

Volume size 1GB- 1TB 1GB- 1TB 1GB- 1TB

Max IOPS per volume3 IOPS/GB Burst up to 3000

IOPS4,000 IOPS ~100 IOPS

Read and write peak throughput 128 MB/s 128 MB/s ~50-90 MBps

Max I/O per node(16k) 48,000 48,000 48,000

Peak throughput node 800 MB/s 800 MB/s 800MB/s

Latency (random read) 1-2ms 1-2 ms 20-40 ms

API name gp2 io1 standard

Price* $.10/GB-Month$.125/GB-Month

$.065/provisioned IOPS$.05/GB-Month $.05/ 1M I/O

Always use General Purpose (SSD) for boot volumes

• L A W– L

– A

– W

1. IOPS: Read/write I/O rate (IOPS)

2. Latency: Time between I/O submission and completion (ms)

3. Throughput: Read/write transfer rate (MB/s)

– Throughput = IOPS X I/O Size

(1) Amazon EC2

(2) I/O

(4) Amazon

EBS

(3) Network

link

1. EC2 instance: Network bandwidth(Mbps)

2. EBS-optimized: EC2 instance option (On/Off)

3. Workload: Block size, read/write ratio, serialization

4. Queue depth: The number of outstanding I/Os

5. RAID: Stripe volumes to maximize performance

6. Pre-warm: Eliminates first touch penalty

1. Amazon EC2 instance• Compute optimized– C3

• Memory optimized- R3

• General Purpose – M3

Amazon

EBS

Amazon

EC2

Select Amazon EC2 instance that has the right network, RAM, and

CPU resources for your applications.

2. Amazon EBS-optimized

b

Use EBS-optimized instances for consistent EBS

performance

3. Workload

EBS SSD-backed volumes measure I/O size up to 256KB

EBS SSD-backed delivers same performance for read and write

NEW

4,000 IOPS

PIOPS volume

4,000 IOPS

128MB/s throughput

You can achieve 4,000 IOPS

when driving smaller I/O

You can achieve up to 128MB/s

when driving larger I/O

4,000 IOPS

PIOPS volume

4,000 IOPS

128MB/s throughput

4,000 X 4KB = 16MB/s

4,000 X 32KB =128MB/s

8,000 x 8KB = 64MB/s

4,000 x 8KB = 32MB/s

2,000 x 128KB=256MB/s

1,000 x 128KB =128MB/s

Block (I/O) size determines whether your application is IOPS bound or

throughput bound

4. Queue depth

An I/O

Amazon

EBSAfter it’s gone, it’s gone

Amazon

EC2

Queue depth is the pending I/O in flight

I/O Latency

0.075

35.1

0

5

10

15

20

25

30

35

40

1 4 8 12 16 20 24 28 32

La

ten

cy T

P9

0 (

ms

)

Queue Depth

Random READ latency across various QDs

Latency (TP90)

Read latency linearly increases with increase in queue depth

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random READ IOPS, Latency across various QDs

Latency (TP90) Avg Read IOPS

IOP

S

Queue depth of one has the lowest latency, but has the lowest IOPS

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth



IOP

S

Queue depth between 4-8 has the optimal IOPS and latency performance

0.075

35.1

2.09

1,865

4,152

3,851

-

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

5

10

15

20

25

30

35

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth



IOP

S

Higher queue depths have negative impact on IOPS and latency

0.08

7.71

845

4,152

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

0

1

2

3

4

5

6

7

8

9

10

1 4 8 12 16 20 24 28 32

La

ten

cy

TP

90

( m

s)

Queue Depth

16 KB random WRITE IOPS, Latency across various QDs

Latency (TP90) AvgIOPS

IOP

S

Write latency, queue depth and IOPS interaction is similar to

that of read latency

Optimal queue depth to achieve lower latency and highest IOPS is

typically between 4-8; ~1 QD per 500 IOPS

Amazon EBS-optimized offers consistent latency experience

5. RAID• Increase performance or capacity or both

• Stripe multiple volumes to achieve 10’s of TB and up-to 48,000 IOPS per Instance

• Don’t mix volume types

• Typically RAID 0 or LVM stripe

• Avoid RAID for redundancy Amazon

EBS

Amazon

EC2

Maximum performance per instance

12×400 GB PIOPS, pre-warmed, RAID 0 LVM, Stripe size 128 KB, attached to CR1 instance

Use stripe size of 128KB or 256KB

6. Pre-warming

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

Use large block size to speed up your pre-warming.

ex: sudo dd if=/dev/xvdf of=/dev/xvdf conv=notrunc bs=1M

Final tips

• Ext4 or XFS (understand journal impact!)

• Amazon EBS encryption has no visible IOPS

performance impact for many Amazon EC2

instances

• Alignment can matter, check tools use 4k

Workload/

Software

Typical block

size

Random/

Seq?

Max EBS @ 500

Mb/s

Max EBS @

1 Gb/s

Max EBS @ 10Gb/s

instances

Oracle DB Configurable:2 KB –16

KB

Default 8 KB

random ~7800 IOPS ~15,600 IOPS ~96,000 IOPS

Microsoft SQL

Server

8 KB w/ 64 KB

extents

random ~7800 IOPS ~15,600 IOPS ~80,000 IOPS

MySQL 16 KB random ~4000 IOPS ~7,800 IOPS ~48,000 IOPS

PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~96,000 IOPS

MongoDB 4 KB serialized ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS

Cassandra 4 KB random ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS

GlusterFS 128 KB sequential ~500 IOPS ~1000 IOPS ~6,000 IOPS

EBS-optimized

Oh, YEAH!!

Amazon

EC2

A “boatload” of I/O

Right sized EBS

1. EC2 instance: Network transfer rate (Mbps)

2. EBS-optimized: EC2 instance option (On/Off)

3. Workload: Block size, read/write ratio, serialization

4. Queue depth: The number of outstanding I/Os (#)

5. RAID: Stripe volumes to maximize performance

6. Pre-warm: Eliminates first touch penalty

Performance (per volume)

GP2 PIOPS GP2 PIOPS

Max Size 1 TB 1 TB 16 TB 16 TB

Max IOPS 3,000 4,000 10,000 20,000

Max Throughput 128 Mbps 128 Mbps 160 MBps 320 MBps

PREVIOUSLYW/ LARGER &

FASTER VOLUMES

Amazon EBS larger and faster volume types

General Purpose (SSD) Provisioned IOPS (SSD) Magnetic

Recommend use casesBoot volumes Small to med DBs Dev and test

I/O intensive Large DBs

Cold storage

Storage media SSD-backed SSD-backed Magnetic-backed

Volume size 1GB- 16TB 4GB- 16TB 1GB- 1TB

Max IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS

burst < 1TB to 3000 IOPS baseline baseline

Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MBps

Max IOPS per node (16k) 48,000 48,000 48,000

Peak throughput node 800 MB/s 800 MB/s 800MB/s

Latency (random read) 1-2ms 1-2 ms 20-40 ms

API name gp2 io1 standard

Price* $.10/GB-Month$.125/GB-Month

$.065/provisioned IOPS$.05/GB-Month

$.05/ 1M I/O

• 2-tiered key hierarchy using envelope

encryption

• Unique data key encrypt customer data

• AWS KMS master keys encrypt data keys

• Benefits of envelope encryption:• Limits risk of a compromised data key

• Better performance for encrypting large data

• Easier to manage a small number of master

keys than millions of data keys

Master Key(s)

Data Key 1

S3 object EBS

volume

Redshift

cluster

Data Key 2 Data Key 3 Data Key 4

Custom

application

AWS KMS

Select the right volume for your workload

Select the right instance for your workload

Take snapshots

Use encryption if you need it

Please give us your feedback on this session.

Complete session evaluations and earn re:Invent swag.

http://bit.ly/awsevals

http://bit.ly/awsevals