The State of HBase Replication

The State of HBase ReplicationJean-Daniel CryansMay 5th, 2014

About me

• Software Engineer at Cloudera, Storage team• Apache HBase committer since 2008, PMC member

Motivation for HBase Replication• Even though HBase is:

• distributed;

• distributed;• fault-tolerant;

• distributed;• fault-tolerant;• highly available; and

• distributed;• fault-tolerant;• highly available; and• almost magic.

The Current State• It’s production-ready.

The Current State• It’s production-ready.• It’s used to replicate data between thousands of nodes across continents.

The Current State• It’s production-ready.• It’s used to replicate data between thousands of nodes across continents.• It’s used for Disaster Recovery, geo-distributed serving, and more.

Agenda• Four Years of Replication• Use Cases in Production• Roadmap

Design• Clusters are distinct• Pull VS push• Sync VS Async

Clusters are Distinct•HBase doesn’t span DCs, HDFSs

Master20 RS

Slave15 RS

Clusters are Distinct•HBase doesn’t span DCs, HDFSs• .META. operations aren’t replicated

Master20 RS

Slave15 RS

• Regions can be different

Master20 RS

Slave15 RS

• Regions can be different• Security has to be configured for each cluster

Master20 RS

Slave15 RS

Push instead of Pull

MySQLMaster

MySQLSlave

Get binlog

Apply locally

MySQL Replication uses PullCluster A Cluster B

Push instead of Pull

RS RSreplicate entries

Apply to cluster

HBase Replication uses PushCluster A Cluster B

Async instead of Sync

Cluster A Cluster B

RSHLog

MemStore

RSHLog

MemStore

Synchronous Replication

Cluster A Cluster B

RSHLog

MemStore

RSHLog

MemStore

Cluster A Cluster B

RSHLog

MemStore

RSHLog

MemStore

Ack Ack

Asynchronous Replication

Asynchronous ReplicationCluster A

RSHLog

MemStore

Asynchronous ReplicationCluster A

RSHLog

MemStore

Cluster B

RSHLog

MemStoreAck

HLogTailingThread

First Release - 0.90.0• Simple master-slave (only one)•Disabled by default• Uses ZK as a metadata store

Original Implementation

replicateLogEntries()ReplicationSource

ZooKeeperWatcher

Region Server onMaster Cluster

ReplicationSink

HTablePut

Delete

Region Server onSlave Cluster

First Lesson Learned•HDFS doesn’t support tailing files being written to. It requires:• open()• seek()// go where we stopped last time• while (not EOF || enoughData)

•read()

• close()• repeat

Second Lesson Learned• Single threaded, non-batched ZK is slow• ZK didn’t have an atomic move operation

• Doubles # ops needed, race conditions

Second Lesson Learned• Single threaded, non-batched ZK is slow• ZK didn’t have an atomic move operation

• Doubles # ops needed, race conditions

/hbase /replication /RS1 /1 /hlog1 /hlog2...

/hbase /replication /RS2 /1-RS1 /hlog1

1. create new hlog22. delete old hlog2

Second Release - 0.92.0• Cyclic replication•Multi-slave (scope LOCAL or GLOBAL)• Enable / disable peer• Special configurations

Cyclic Replication

Cluster1

Cluster2

Cluster3

Put Row X

Cyclic Replication

Cluster1

Cluster2

Cluster3

Put Row X

Cyclic Replication

Cluster1

Cluster2

Cluster3

Put Row X

Cyclic Replication

Cluster1

Cluster2

Cluster3

Put Row X

Row X is from 1Don’t replicate!

Multi-Slave

Cluster1

Cluster2

Cluster3

Put Row X

Multi-Slave

Cluster1

Cluster2

Cluster3

Put Row X

Multi-Slave

Cluster1

Cluster2

Cluster3

Put Row X

Put Row X Put Row X

Enable / Disable Peers

Cluster 1

RSHLog

Cluster 2

RSHLogTailingThread

Enable / Disable Peers> disable_peer ‘2’

Cluster 1

RSHLog

Cluster 2

RSHLogTailingThread

Is the peer enabled?

Cluster 1

RSHLog

Cluster 2

RSHLogTailingThreadHLog

Cluster 1

RSHLog

Cluster 2

Cluster 1

RSHLog

Cluster 2

HLogIs the peer enabled?

Cluster 1

RSHLog

Cluster 2

HLog Is the peer enabled?

Cluster 1

RSHLog

Cluster 2

Special Configurations• KEEP_DELETED_CELLS

• Must be used on slaves with replication when deleting data.

Special Configurations• KEEP_DELETED_CELLS

• Must be used on slaves with replication when deleting data.

•MIN_VERSION• With TTL, makes it easy to configure a slave that contains only the last few days of data.

Third Lesson Learned• It’s easy to DDOS yourself.• Replication was using the normal handlers...• ... and using them to write back!

Handler1: PutHandler2: DeleteHandler3: ReplicateHandler4: GetHandler5: Put

Replicated Put goes in the queue

Fourth Lesson Learned• Instinctively, what would something called stop_replication do?

Fourth Lesson Learned• Instinctively, what would something called stop_replication do?•Good intentions, bad outcomes, HBASE-8861

start/stop_replicationX

Third Release - 0.96.0 / 0.98.0• Replication enabled by default!• Completely refactored for readability/extensibility (Chris Trezzo)• ReplicationSyncUp tool (HBASE-9047)• Throttling (HBASE-9501)• Finer grained replication controls (HBASE-8751)

ReplicationSyncUp Tool•Works on an offline cluster• Can finish replicating the queues in ZK• Useful to finish draining a master cluster

ZooKeeper

ReplicationSyncUp

Finer Grained Replication Controls> set_peer_tableCFs '2', "table1; table2:cf1,cf2; table3:cfA,cfB"•Meaning: enable replication to peer #2 for:

• All of table1• cf1 and cf2 from table2• cfA and cfB from table3

Agenda• Four Years of Replication•Use Cases in Production• Roadmap

Flurry• Two data centers, coast to coast• Three clusters, in master-master pairs

• 1200 nodes• 800 nodes• 30 nodes

• Replication traffic: 2Gbps• Latency between DCs: 85ms

Opower• Two clusters, same data center

• Master: tens of nodes• Slave: tens of nodes

• Replication traffic: 1GB/day• Bulk load replication traffic: 180GB/day• Recent use case

Lily HBase Indexer• Collaboration between NGData & Cloudera.

• NGData are the creators of the Lily data management platform.

• Lily HBase Indexer • Service which acts as a HBase replication listener.• Custom sink writes to SolrCloud.• Integrates Cloudera Morphlines library for ETL of rows.

Agenda• Four Years of Replication• Use Cases in Production• Roadmap

Stop Relying on Permanent Znodes• Current rule is to never rely on znodes to survive cluster restarts, upgrades, etc.• State data should be kept in an HBase table.•Notification done through a new mechanism• See: https://issues.apache.org/jira/browse/HBASE-10295

Define a Replication Interface• Replication is somewhat extendable but it lacks stable interfaces.• The HBase Indexer is such an extension and it required surgery every time a committer sneezed.• See: https://issues.apache.org/jira/browse/HBASE-10504

Distributed Counters• Incrementing consists of:

1.Taking a lock;

1.Taking a lock;2.Get’ing the current value; and

1.Taking a lock;2.Get’ing the current value; and3.Put’ing the newly incremented value.

• This breaks in Master-Master because the Puts are overwriting each other.

• This breaks in Master-Master because the Puts are overwriting each other.• See https://issues.apache.org/jira/browse/HBASE-2804

More Tooling• Replication management console, one shell to rule all the clusters!• Replication bootstrapping tool.• Tool that can move queues between region servers.• Tool that can throttle replication on a live cluster.

Questions?•Or ping me async:

• @jdcryans• jdcryans@cloudera.com• jdcryans on #hbase irc.freenode.net

The State of HBase Replication

Software

Transcript of The State of HBase Replication

Secure HBase

Early Scheduling in Parallel State Machine Replication · MySQL Group Replication Galera Cluster Blockchain, ... Parallel State Machine Replication 5 R1 R4 R2 R5 R3 R0 scheduler worker

Introduction to Hbase. Agenda What is Hbase About RDBMS Overview of Hbase Why Hbase instead of RDBMS Architecture of Hbase Hbase interface.

Hyper-V Replication – Enabling VM Replica...paused state -> in this stage, Select VM ->Replication -> Resume replication. It start replicating changes if it went to paused state.

Faster HBase queries - events.static.linuxfound.org · Faster HBase queries Introducing hindex – Secondary indexes for HBase Rajeshbabu Chintaguntla rajeshbabu@apache.org ApacheCon

HBaseConEast2016: HBase and Spark, State of the Art

The State of Apache HBase - SK Planetreadme.skplanet.com/wp-content/uploads/key_hbase... · Chair of the Apache HBase PMC* ... Apache Drill & HBase! Thank You! stack@apache.org. TODO

HBase - Cleveland State Universitycis.csuohio.edu/~sschung/cis612/LectureNotes_HBaseFinal.pdf · HBase Architecture HBase is a distributed database, designed to run on a cluster of

HBase Schema Design - HBase-Con 2012

Replication › CIgnat › files › replication › Replication_C4.pdf · State-based Replication • Replicated object: a tuple (S, s 0, q, u, m) § S: state domain § Replica at

Hbase in action - Chapter 09: Deploying HBase

HBase Snapshots

Hoya : HBase on YARN (2013-08-20 HBase Hug)

Hbase Tutorial

Intro to HBase Internals & Schema Design (for HBase users)

Replication Management using the State-Machine Approach

Apache HBase 0 - files.meetup.comfiles.meetup.com/1350427/Apache HBase 0.98.v2.pdfEndpoint EXEC Grants (HBASE-6104) • HBase ACLs can grant a familiar set of privileges to users (and

HTrace: Tracing in HBase and HDFS (HBase Meetup)

Hbase hivepig

Early Scheduling in Parallel State Machine Replication