HBase: Extreme makeover

Vladimir RodionovHadoop/HBase architectFounder of BigBase.org

HBaseCon 2014Features & Internal Track

Agenda

About myself• Principal Platform Engineer @Carrier IQ, Sunnyvale, CA • Prior to Carrier IQ, I worked @ GE, EBay, Plumtree/BEA.• HBase user since 2009.• HBase hacker since 2013.• Areas of expertise include (but not limited to) Java,

HBase, Hadoop, Hive, large-scale OLAP/Analytics, and in-memory data processing.

• Founder of BigBase.org

BigBase

BigBase = EM(HBase)

EM(*) = ?

BigBase = EM(HBase)

EM(*) =

BigBase = EM(HBase)

EM(*) =

Seriously?

BigBase = EM(HBase)

EM(*) =

Seriously?for HBaseIt’s a Multi-Level Caching solution

Real Agenda• Why BigBase?• Brief history of BigBase.org project• BigBase MLC high level architecture (L1/L2/L3)• Level 1 - Row Cache.• Level 2/3 - Block Cache RAM/SSD.• YCSB benchmark results• Upcoming features in R1.5, 2.0, 3.0.• Q&A

• Still lacks some original BigTable’s features.• Still not able to utilize efficiently all RAM. • No good mixed storage (SSD/HDD) support. • Single Level Caching only. Simple. • HBase + Large JVM Heap (MemStore) = ?

BigBase

• Adds Row Cache and block cache compression.• Utilizes efficiently all RAM (TBs). • Supports mixed storage (SSD/HDD). • Has Multi Level Caching. Not that simple. • Will move MemStore off heap in R2.

BigBase History

Koda (2010)• Koda - Java off heap object cache, similar to

Terracotta’s BigMemory.• Delivers 4x times more transactions …• 10x times better latencies than BigMemory 4.• Compression (Snappy, LZ4, LZ4HC, Deflate).• Disk persistence and periodic cache snapshots.• Tested up to 240GB.

Karma (2011-12)• Karma - Java off heap BTree implementation to support

fast in memory queries.• Supports extra large heaps, 100s millions – billions

objects.• Stores 300M objects in less than 10G of RAM.• Block Compression.• Tested up to 240GB.• Off Heap MemStore in R2.

Yamm (2013)• Yet Another Memory Manager.– Pure 100% Java memory allocator.– Replaced jemalloc in Koda. – Now Koda is 100% Java.– Karma is the next (still on jemalloc).– Similar to memcached slab allocator.

• BigBase project started (Summer 2013).

BigBase Architecture

MLC – Multi-Level Caching

HBase 0.94

LRUBlockCache

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

One level of caching : • RAM (L2)

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

One level of caching : • RAM (L2)• Or DISK (L3)

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

BigBase 1.0

Block Cache L3SSD

Row Cache L1

Block Cache L2

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

BigBase 1.0

Row Cache L1

Block Cache L2

BlockCache L3Network

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

BigBase 1.0

Row Cache L1

Block Cache L2

BlockCache L3memcached

HBase 0.94

LRUBlockCache

HBase 0.96

Bucket cache

BigBase 1.0

Row Cache L1

Block Cache L2

BlockCache L3DynamoDB

BigBase Row Cache (L1)

Where is BigTable’s Scan Cache?

• Scan Cache caches hot rows data. • Complimentary to Block Cache.• Still missing in HBase (as of 0.98). • It’s very hard to implement in Java (off heap).• Max GC pause is ~ 0.5-2 sec per 1GB of heap• G1 GC in Java 7 does not resolve the problem.• We call it Row Cache in BigBase.

Row Cache vs. Block Cache

HFile Block HFile BlockHFile BlockHFile BlockHFile Block

BLOCK CACHE

ROW CACHE

BLOCK CACHE

ROW CACHE

BLOCK CACHE

BigBase Row Cache

• Off Heap Scan Cache for HBase.• Cache size: 100’s of GBs to TBs. • Eviction policies: LRU, LFU, FIFO,

Random. • Pure 100% - compatible Java. • Sub-millisecond latencies, zero GC.• Implemented as RegionObserver

coprocessor.

Row Cache

YAMM Codecs Kryo SerDe

BigBase Row Cache

• Read through cache. • It caches rowkey:CF. • Invalidates key on every mutation.• Can be enabled/disabled per table and

per table:CF.• New ROWCACHE attribute.• Best for small rows (< block size)

Row Cache

YAMM Codecs Kryo SerDe

Performance-Scalability

• GET (small rows < 100 bytes): 175K operations per sec per one Region Server (from cache).

• MULTI-GET (small rows < 100 bytes): > 1M records per second (network limited) per one Region Server.

• LATENCY : 99% < 1ms (for GETs) with 100K ops.• Vertical scalability: tested up to 240GB (the maximum available

in Amazon EC2).• Horizontal scalability: limited by HBase scalability. • No more memcached farms in front of HBase clusters.

BigBase Block Cache (L2, L3)

What is wrong with Bucket Cache?

Scalability LIMITED

Multi-Level Caching (MLC) NOT SUPPORTED

Persistence (‘offheap’ mode) NOT SUPPORTED

Low latency apps NOT SUPPORTED

SSD friendliness (‘file’ mode) NOT FRIENDLY

Compression NOT SUPPORTED

Scalability LIMITED

Low latency apps ?

Scalability LIMITED

Here comes BigBase

Scalability HIGH

Multi-Level Caching (MLC) SUPPORTED

Persistence (‘offheap’ mode) SUPPORTED

Low latency apps SUPPORTED

SSD friendliness (‘file’ mode) SSD-FRIENDLY

Compression SNAPPY, LZ4, LZHC, DEFLATE

Here comes BigBase

Scalability HIGH

Here comes BigBase

Scalability HIGH

Here comes BigBase

Scalability HIGH

Here comes BigBase

Scalability HIGH

Here comes BigBase

Scalability HIGH

Wait, there are more …

Scalability HIGHMulti-Level Caching (MLC) SUPPORTEDPersistence (‘offheap’ mode) SUPPORTEDLow latency apps SUPPORTEDSSD friendliness (‘file’ mode) SSD-FRIENDLYCompression SNAPPY, LZ4, LZHC, DEFLATENon disk–based L3 cache SUPPORTEDRAM Cache optimization IBCO

Wait, there are more …

Scalability HIGHMulti-Level Caching (MLC) SUPPORTEDPersistence (‘offheap’ mode) SUPPORTEDLow latency apps SUPPORTEDSSD friendliness (‘file’ mode) SSD-FRIENDLYCompression SNAPPY, LZ4, LZHC, DEFLATENon disk–based L3 cache SUPPORTEDRAM Cache optimization IBCO

BigBase 1.0 vs. HBase 0.98

BigBase HBase 0.98

Row Cache (L1) YES NO

Block Cache RAM (L2) YES (fully off heap) YES (partially off heap)

Block Cache (L3) DISK YES (SSD- friendly) YES (not SSD – friendly)

Block Cache (L3) NON DISK YES NO

Compression YES NO

RAM Cache persistence YES (both L1 and L2) NO

Low Latency optimized YES NO

MLC support YES (L1, L2, L3) NO (either L2 or L3)

Scalability HIGH MEDIUM (limited by JVM heap)

YCSB Benchmark

Test setup (AWS)

• HBase 0.94.15 – RS: 11.5GB heap (6GB LruBlockCache on heap); Master: 4GB heap.

• Clients: 5 (30 threads each), collocated with Region Servers.

• Data sets: 100M and 200M. 120GB / 240GB approximately. Only 25% fits in a cache.

• Workloads: 100% read (read100, read200, hotspot100), 100% scan (scan100, scan200) –zipfian.

• YCSB 0.1.4 (modified to generate compressible data). We generated compressible data (with factor of 2.5x) only for scan workloads to evaluate effect of compression in BigBase block cache implementation.

• Common – Whirr 0.8.2; 1 (Master + Zk) + 5 RS; m1.xlarge: 15GB RAM, 4 vCPU, 4x420 HDD

• BigBase 1.0 (0.94.15) – RS: 4GB heap (6GB off heap cache); Master: 4GB heap.

• HBase 0.96.2 – RS: 4GB heap (6GB Bucket Cache off heap); Master: 4GB heap.