Flash-based (cloud) storage systems

Lecture 25Aditya Akella

• BufferHash: invented in the context of network de-dup (e.g., inter-DC log transfers)

• SILT: more “traditional” key-value store

Cheap and Large CAMs for High Performance Data-Intensive Networked Systems

Ashok Anand, Chitra Muthukrishnan, Steven Kappes, and Aditya AkellaUniversity of Wisconsin-Madison

Suman NathMicrosoft Research

New data-intensive networked systems

Large hash tables (10s to 100s of GBs)

New data-intensive networked systems

Data center Branch office

WAN

WAN optimizersObject

Object store (~4 TB)Hashtable (~32GB)

Look up

Object

Chunks(4 KB)

Key (20 B)

Chunk pointer Large hash tables (32 GB)

High speed (~10 K/sec) inserts and evictions

High speed (~10K/sec) lookups for 500 Mbps link

New data-intensive networked systems

• Other systems – De-duplication in storage systems (e.g., Datadomain)– CCN cache (Jacobson et al., CONEXT 2009)– DONA directory lookup (Koponen et al., SIGCOMM 2006)

Cost-effective large hash tablesCheap Large cAMs

Candidate options

DRAM 300K $120K+

Disk 250 $30+

Random reads/sec

Cost(128 GB)

Flash-SSD 10K* $225+

Random writes/sec

250

300K

5K*

Too slow Too

expensive

* Derived from latencies on Intel M-18 SSD in experiments

2.5 ops/sec/$

Slow writes

How to deal with slow writes of Flash SSD

+Price statistics from 2008-09

CLAM design

• New data structure “BufferHash” + Flash• Key features– Avoid random writes, and perform sequential writes

in a batch• Sequential writes are 2X faster than random writes (Intel

SSD)• Batched writes reduce the number of writes going to Flash

– Bloom filters for optimizing lookups

BufferHash performs orders of magnitude better than DRAM based traditional hash tables in ops/sec/$

Flash/SSD primer

• Random writes are expensive Avoid random page writes

• Reads and writes happen at the granularity of a flash page

I/O smaller than page should be avoided, if possible

Conventional hash table on Flash/SSD

Flash

Keys are likely to hash to random locations

Random writes

SSDs: FTL handles random writes to some extent;But garbage collection overhead is high

~200 lookups/sec and ~200 inserts/sec with WAN optimizer workload, << 10 K/s and 5 K/s

Conventional hash table on Flash/SSD

DRAM

Flash

Can’t assume locality in requests – DRAM as cache won’t work

Our approach: Buffering insertions• Control the impact of random writes• Maintain small hash table (buffer) in memory • As in-memory buffer gets full, write it to flash

– We call in-flash buffer, incarnation of buffer

Incarnation: In-flash hash table

Buffer: In-memory hash table

DRAM Flash SSD

Two-level memory hierarchyDRAM

Flash

Buffer

Incarnation table

Incarnation

1234

Net hash table is: buffer + all incarnations

Oldest incarnation

Latest incarnation

Lookups are impacted due to buffers

DRAM

Flash

Buffer

Incarnation table

Lookup key

In-flash look ups

Multiple in-flash lookups. Can we limit to only one?

4 3 2 1

Bloom filters for optimizing lookups

DRAM

Flash

Buffer

Incarnation table

Lookup keyBloom filters

In-memory look ups

False positive!

Configure carefully! 4 3 2 1

2 GB Bloom filters for 32 GB Flash for false positive rate < 0.01!

Update: naïve approachDRAM

Flash

Buffer

Incarnation table

Bloom filtersUpdate key

Update keyExpensive random writes

Discard this naïve approach

4 3 2 1

Lazy updatesDRAM

Flash

Buffer

Incarnation table

Bloom filtersUpdate key

Insert key

4 3 2 1

Lookups check latest incarnations first

Key, new value

Key, old value

Eviction for streaming apps• Eviction policies may depend on application– LRU, FIFO, Priority based eviction, etc.

• Two BufferHash primitives– Full Discard: evict all items

• Naturally implements FIFO– Partial Discard: retain few items

• Priority based eviction by retaining high priority items

• BufferHash best suited for FIFO– Incarnations arranged by age– Other useful policies at some additional cost

• Details in paper

Issues with using one buffer

• Single buffer in DRAM– All operations and

eviction policies

• High worst case insert latency– Few seconds for 1

GB buffer– New lookups stall

DRAM

Flash

Buffer

Incarnation table

Bloom filters

4 3 2 1

Partitioning buffers

• Partition buffers– Based on first few bits of

key space– Size > page

• Avoid i/o less than page– Size >= block

• Avoid random page writes

• Reduces worst case latency

• Eviction policies apply per buffer

DRAM

Flash

Incarnation table

4 3 2 1

0 XXXXX 1 XXXXX

BufferHash: Putting it all together

• Multiple buffers in memory• Multiple incarnations per buffer in flash• One in-memory bloom filter per incarnation

DRAM

FlashBuffer 1 Buffer K. .

. .

Net hash table = all buffers + all incarnations

Latency analysis

• Insertion latency – Worst case size of buffer – Average case is constant for buffer > block size

• Lookup latency– Average case Number of incarnations – Average case False positive rate of bloom filter

Parameter tuning: Total size of Buffers

.

. .

.

Total size of buffers = B1 + B2 + … + BN

Too small is not optimalToo large is not optimal eitherOptimal = 2 * SSD/entry

DRAM

Flash

Given fixed DRAM, how much allocated to buffers

B1 BN

# Incarnations = (Flash size/Total buffer size)

Lookup #Incarnations * False positive rate

False positive rate increases as the size of bloom filters decrease

Total bloom filter size = DRAM – total size of buffers

Parameter tuning: Per-buffer size

Affects worst case insertion

What should be size of a partitioned buffer (e.g. B1) ?

.

. .

.

DRAM

Flash

B1 BN

Adjusted according to application requirement (128 KB – 1 block)

SILT: A Memory-Efficient,High-Performance Key-Value Store

Hyeontaek Lim, Bin Fan, David G. AndersenMichael Kaminsky†

Carnegie Mellon University†Intel Labs

2011-10-24

26

Key-Value Store

Clients

PUT(key, value)value = GET(key)

DELETE(key)

Key-Value StoreCluster

• E-commerce (Amazon)• Web server acceleration (Memcached)• Data deduplication indexes• Photo storage (Facebook)

• SILT goal: use much less memory than previous systems while retaining high performance.

27

Three Metrics to Minimize

Memory overhead

Read amplification

Write amplification

• Ideally 0 (no memory overhead)

• Limits query throughput• Ideally 1 (no wasted flash reads)

• Limits insert throughput• Also reduces flash life expectancy• Must be small enough for flash to last a few years

= Index size per entry

= Flash reads per query

= Flash writes per entry

28

0 2 4 6 8 10 120

2

4

6

Landscape before SILTRead amplification

Memory overhead (bytes/entry)

FAWN-DS

HashCache

BufferHash FlashStore

SkimpyStash

29

?

30

SILT Sorted Index(Memory efficient)

SILT Log Index(Write friendly)

Solution Preview: (1) Three Stores with (2) New Index Data Structures

MemoryFlash

SILT Filter

Inserts only go to Log

Data are moved in background

Queries look up stores in sequence (from new to old)

LogStore: No Control over Data Layout

6.5+ bytes/entry 1Memory overhead Write amplification

Inserted entries are appended

On-flash log

Memory

Flash

31

SILT Log Index (6.5+ B/entry)

(Older) (Newer)

Naive Hashtable (48+ B/entry)

SortedStore: Space-Optimized Layout

0.4 bytes/entry High

On-flash sorted array

Memory overhead Write amplification

Memory

Flash

32

SILT Sorted Index (0.4 B/entry)

Need to perform bulk-insert to amortize cost

Combining SortedStore and LogStore

On-flash log

SILT Sorted Index

Merge

33

SILT Log Index

On-flash sorted array

<SortedStore> <LogStore>

Achieving both Low Memory Overhead and Low Write Amplification

34

SortedStore LogStore

SortedStore

LogStore

• Low memory overhead• High write amplification

• High memory overhead• Low write amplification

Now we can achieve simultaneously:Write amplification = 5.4 = 3 year flash lifeMemory overhead = 1.3 B/entry

With “HashStores”, memory overhead = 0.7 B/entry!

1.010.7 bytes/entry 5.4Memory overhead Read amplification Write amplification

35

SILT’s Design (Recap)

On-flash sorted array

SILT Sorted Index

On-flash log

SILT Log Index

On-flash hashtables

SILT Filter

Merge Conversion

<SortedStore> <LogStore><HashStore>

New Index Data Structures in SILT

Partial-key cuckoo hashing

For HashStore & LogStoreCompact (2.2 & 6.5 B/entry)

Very fast (> 1.8 M lookups/sec)

36

SILT Filter & Log Index

Entropy-coded tries

For SortedStoreHighly compressed (0.4 B/entry)

SILT Sorted Index

0 2 4 6 8 10 120

2

4

6

LandscapeRead amplification

Memory overhead (bytes/entry)

FAWN-DS

HashCache

BufferHash FlashStore

SkimpyStash

38

SILT

BufferHash: Backup

Outline

• Background and motivation

• Our CLAM design– Key operations (insert, lookup, update)– Eviction– Latency analysis and performance tuning

• Evaluation

Evaluation

• Configuration– 4 GB DRAM, 32 GB Intel SSD, Transcend SSD– 2 GB buffers, 2 GB bloom filters, 0.01 false positive

rate– FIFO eviction policy

BufferHash performance

• WAN optimizer workload– Random key lookups followed by inserts– Hit rate (40%)– Used workload from real packet traces also

• Comparison with BerkeleyDB (traditional hash table) on Intel SSDAverage latency BufferHash BerkeleyDB

Look up (ms) 0.06 4.6

Insert (ms) 0.006 4.8

Better lookups!

Better inserts!

Insert performance

0.001 0.01 0.1 1 10 100

BerkeleyDB

0.001 0.01 0.1 1 10 100

Bufferhash

0.2

0.40.6

0.81.0

CDF

Insert latency (ms) on Intel SSD

99% inserts < 0.1 ms

40% of inserts > 5 ms !

Random writes are slow! Buffering effect!

Lookup performance

0.001 0.01 0.1 1 10 100

Bufferhash

0.001 0.01 0.1 1 10 100

BerkeleyDB

0.20.40.60.81.0

CDF

99% of lookups < 0.2ms

40% of lookups > 5 ms

Garbage collection overhead due to writes!

60% lookups don’t go to Flash 0.15 ms Intel SSD latency

Lookup latency (ms) for 40% hit workload

Performance in Ops/sec/$

• 16K lookups/sec and 160K inserts/sec

• Overall cost of $400

• 42 lookups/sec/$ and 420 inserts/sec/$– Orders of magnitude better than 2.5 ops/sec/$ of

DRAM based hash tables

Other workloads

• Varying fractions of lookups• Results on Trancend SSD

Lookup fraction BufferHash BerkeleyDB0 0.007 ms 18.4 ms0.5 0.09 ms 10.3 ms1 0.12 ms 0.3 ms

• BufferHash ideally suited for write intensive workloads

Average latency per operation

Evaluation summary• BufferHash performs orders of magnitude better in

ops/sec/$ compared to traditional hashtables on DRAM (and disks)

• BufferHash is best suited for FIFO eviction policy– Other policies can be supported at additional cost, details in

paper

• WAN optimizer using Bufferhash can operate optimally at 200 Mbps, much better than 10 Mbps with BerkeleyDB– Details in paper

Related Work

• FAWN (Vasudevan et al., SOSP 2009)– Cluster of wimpy nodes with flash storage– Each wimpy node has its hash table in DRAM– We target…• Hash table much bigger than DRAM • Low latency as well as high throughput systems

• HashCache (Badam et al., NSDI 2009)– In-memory hash table for objects stored on disk

WAN optimizer using BufferHash

• With BerkeleyDB, throughput up to 10 Mbps

• With BufferHash, throughput up to 200 Mbps with Transcend SSD– 500 Mbps with Intel SSD

• At 10 Mbps, average throughput per object improves by 65% with BufferHash

SILT Backup Slides

Evaluation

1. Various combinations of indexing schemes2. Background operations (merge/conversion)3. Query latency

Experiment SetupCPU 2.80 GHz (4 cores)

Flash driveSATA 256 GB

(48 K random 1024-byte reads/sec)

Workload size 20-byte key, 1000-byte value, ≥ 50 M keysQuery pattern Uniformly distributed (worst for SILT)

51

LogStore Alone: Too Much Memory

52

Workload: 90% GET (50-100 M keys) + 10% PUT (50 M keys)

LogStore+SortedStore: Still Much Memory

53

Workload: 90% GET (50-100 M keys) + 10% PUT (50 M keys)

Full SILT: Very Memory Efficient

54

Workload: 90% GET (50-100 M keys) + 10% PUT (50 M keys)

Small Impact from Background Operations

33 K

55

40 K

Workload: 90% GET (100~ M keys) + 10% PUT

Oops! burstyTRIM by ext4 FS

Low Query Latency

56

# of I/O threads

Workload: 100% GET (100 M keys)Best tput @ 16 threads

Median = 330 μs99.9 = 1510 μs

57

Conclusion

• SILT provides both memory-efficient andhigh-performance key-value store– Multi-store approach– Entropy-coded tries– Partial-key cuckoo hashing

• Full source code is available– https://github.com/silt/silt