What's New in Couchbase Server 3.0: Couchbase Connect 2014

Deep Dive: What’s New in

Couchbase Server 3.0

Cihan Biyikoglu

Director of Product Management | Couchbase

Part I – Couchbase Overview

Journey so far…

Part II – Deep Dive into Couchbase Server v3.0

Developer Empowerment

Mission Critical Scale

Ultra High Availability

Secure Communication

Simplified Administration

Q&A

Agenda

Couchbase Server Overview

Overview

©2014 Couchbase, Inc. 4

General purpose

Elastic scalability Consistent high

performance

Always

available

Flexible, global

deployment

Enterprise grade

administration

Real time big

data

Data

mobility

Developer

focused

Most complete, scalable & highest performing NoSQL database

Journey so far…

Deep Dive into

Couchbase Server v3.0

DEVELOPER

EMPOWERMENT

MISSION CRITICAL

SCALE

ULTRA HIGH

AVAILABILITY

SECURE

COMMUNICATION

SIMPLIFIED

ADMINISTRATION

• Fully Integrated JSON

Programmability

• Native Async and

Reactive

Programmability

• New Integration with

Popular Frameworks

• N1QL Preview Support

in SDKs

• Optimized Massive

Databases with Tunable

Memory

• Auto-Tuning Thread

Pool for Extreme IO

Throughput

• Faster Replication with

DCP

• Improved Durability

• Lower Latency Indexing

On Views

• Low Latency, In-Memory

XDCR

• Delta Node Recovery

• Faster Rebalance

• Faster Node Restart with

Efficient Warm-up

• New Incremental

Backup option

• Encryption for Data

Access

• Encryption for

Administrative Access

• XDCR Pause and

Resume

• Improved Resource

Governance

• Graceful Failover

• Cluster-wide Diagnostics

What’s new in Couchbase Server v3.0

Developer Empowerment

Componentized libraries for simplify use cases

• Couchbase JVM Core - Reusable Foundation

• Handles core IO, topology changes. Reactive and asynchronous.

• Java Client Library

• Native JSON handling built-in

• Provides synchronous and asynchronous interface

• Preview of N1QL language integrated query

What’s New in SDK 2.0

Integrated native JSON data type programmability

• Simplified Development


• Preview of N1QL language integrated query

• Performance & Scale

• Improved connection management

• Native .Net:

• Simplified Configuration Management with Web.config & App.config


Node.js, PHP


• Core IO enhancements through updated C libcouchbase

• Better connection management

Python, Ruby

• Core IO enhancements through updated C libcouchbase

• Better connection management

Framework Integrations

Spring, gevent, Twisted, Ottoman


geventfor python

OttomanCouchbase ODM framework

Mission Critical Scale

Optimized Memory Usage with Metadata Ejection Policy Better optimization of memory for massive databases

Enable efficient management of rarely accessed data set

Cache only keys and data for the working set & eject all historic data

Tunable Memory - Optimization for Massive Databases

Tunable Memory - Optimization for Massive Databases

100sx Reduction in Metadata Memory Consumption from 2.5 to 3.0Note: The graph represents characteristics under data mutations. ~50M docs with value size ~0.5KB

v.3.0 – Large DB with Hot Working-set

3 GB Consumed for Metadata in RAM

2.5.1 or earlier – Real-time Latency

80 MB Consumed for Metadata in RAM

Efficient Auto-Tuning Engine Detect and allocate threads based on HW resources

Pool threads for best resource utilization

Improved latency across the board:

Faster Reads

Faster Rebalance

Faster Node reactivation

Faster Durability with Writes & PersistTo (more details on this on later)

…and many more

Auto Tuning Thread Pool

Faster Durability with Persistence

Improved latency on PersistTo

Shared Thread Pool Improves IO speed

Up to 3x better PersistTo latency from 2.5 to 3.0*Note: Absolute latency depend on disk IO subsystem throughput

*95th percentile PersistTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node

Auto Tuning Thread Pool - Faster Durability

0

1000

2000

3000

4000

5000

Tim

e (

ms

)

95 Percentile PersistTo=1 Latency*

2.5.1 3.0

New DCP (Database Change Protocol) protocol for replication Eliminate Disk-IO from the replication access path

High-Performance, Stream-based, In-memory

Ordered based on Sequence Numbers

Snapshot-able

Restart-able

• Powers Node to Node Replication

Replication and Rebalance

Power Delta Node Recovery – more on this later!

Faster Durability with ReplicateTo – more on this later!

• Powers View – more on this later!

• Powers XDCR – more on this later!

Database Change Protocol

Faster Durability with Replication

Improved latency on ReplicateTo

DCP Improves replication speed and durability

Better protects your data!

Up to 150x Improvement on ReplicateTo latency from 2.5 to 3.0*Note: Absolute latency depend on network bandwidth and latency

*95th percentile ReplicateTo=1 latency (ms), 1 bucket x 200M x 1KB, 250 mutations/sec/node

Database Change Protocol - Faster Replication with DCP

0

200

400

600

Tim

e (

ms

)

95 Percentile ReplicateTo=1 Latency*

2.5.1 3.0

Faster Replication with DCP

More up-to-date views Mutation show up at the indexes

faster with the new DCP protocol

No longer require persisting to disk for change detection

Improved consistency with views (stale=false behavior)

Ports to C in view engine

Database Change Protocol - Fresher Indexes

0

10000

20000

30000

40000

Tim

e (

ms

)

Indexing Under Load

2.5.1 3.0

Up to 50x Faster Indexing from 2.5 to 3.0*Note: Absolute latency depend on disk IO speed

*95th percentile Indexing latency (ms), 1 bucket x 20M x 2KB, non-DGM, 1 view, 250 mutations/sec/node, 400 queries/sec

Ultra High Availability

Improved Protections for Your Data

Drastically Lower XDCR Latency

In memory change detection with DCP

No longer require persisting to disk

Up to 4x better on XDCR latency between clusters between 3.0 & 2.5.1*Note: Absolute latency depend on WAN latency and bandwidth

Lower Latency XDCR (Cross Data Center Replication)

0

100

200

300

400

500

600

Tim

e (

ms

)

90th Percentile. XDCR Latency

2.5.1 3.0

90th percentile replication lag (ms), 5 -> 5 UniDir, 2 buckets x 500M x 1KB, 10K SETs/sec, LAN

Improved Rebalance Latency

Efficient Data Movement

Better Resume-ability after blips and failures

Incremental catch-up

Up to 2x Faster Rebalance under Load between 3.0 and 2.5.1

*Note: Absolute latency depend on network and disk IO latency and bandwidth

Faster Rebalance

0

10

20

30

Tim

e (

ms)

Rebalance Latency to Under Load*

2.5.1 3.0

*Rebalance-out (min), 4 -> 3, 1 bucket x 100M x 2KB, 10K ops/sec

Delta Node Recovery

Shorter time to add nodes back into the cluster!

Bring back a server incrementally into the cluster without having to rebuild the full node.

Incremental Node Catch-up with Delta Node Recovery

100sx Reduction in Time to Re-Add Node from 2.5 to 3.0Note: The absolute performance improvement depend on data size and mutation count that needs to be caught up

Efficient warm-up

Parallelization on cache warm-up

Further warm-up time improvements under metadata ejection

Up to 60% Faster Warm-up between 3.0 and 2.5.1

Up to 8x Faster Warm-up with Full Ejection in 3.0

*Note: Absolute latency depend on disk IO latency and bandwidth

Faster Node Re-Activation

Faster Node Re-Activation

Minimize time and resources during backups

Efficient Recovery with Incremental Backup & Restore

• Back up only the data updated

since the last backup

• Differential Backups

• Cumulative Backups

Chris Bertsch

Systems Engineer | Factset Research Systems

FactSet provides financial software for research analysts, portfolio managers and investment backers.

Factset – Chris


Couchbase at FactSet

• Total Capacity in Production so far…

1920 cores, 30TB of RAM with Bare Metal Nodes

XDCR with Bidirectional across 2 Data Centers

Exciting Couchbase 3.0 Features

• Incremental Backup & Restore

• Delta-node Recovery

• Stream-based XDCR

Factset – Chris


Secure Communications

Encrypted Admin & Data Access

Data Access Encryption

SSL based Client-server Communications

SSL based View Access

https://couchbase_server:18092/…

Admin Access Encryption – REST, CLI and HTTPS

https://couchbase_server:18091/…

Encrypted Client-Server Communication

SERVER 3SERVER 1 SERVER 2

Couchbase Server

Simplified Administration

Maintenance - XDCR Pause and Resume

Want to suspend replication? No longer need to drop replications

Ability to pause XDCR traffic anytime

Resume and incrementally catch-up from where you left off

Efficient Maintenance with XDCR

Priority Setting on Buckets

Ability to assign high and low priority to buckets

High priority buckets get higher IO priority and more throughput

Low priority buckets can still be processed without starvation

Improved Resource Governance with Bucket Priority

New option to gracefully failover a node.

Failover with a graceful handoff

May take longer to failover but prevents potential for any data loss

Node Maintenance

Collect cluster-wide diagnostics with single-click

Collect logs from all server or individual specific servers

Upload logs to shareable storage for Couchbase Support team to analyze

Cluster-wide Diagnostics

Cluster-wide Diagnostics Tool

Improved Crash Reporting

Better Serviceability

Improved Monitoring for Warm-up

Stats Enhancements

Improved Checkpointing with XDCR

Expended Options with Couchbase-cli Parallelized Warm-up

Database Change Protocol (DCP)

Increased Connection LimitsExtended Documentation

Enhanced Event Logging

Extended XDCR Resiliency

Better failover resiliency with DCP

Enhanced SSD Performance

Streamlined Build

Side by side support for DCP and TAP

Improved Error Reporting for Apps

Built-in OS Tuning for Linux Flavors

… and More

Improved Resume-ability with Intra Cluster Replication

View Engine rewrite in C

Web.Config & App.Config support with .NET in SDK 2.0

Unified New App Model with 2.0 SDK

Java SDK 2.0 built on top of RxJava

N1QL Preview Support in the 2.0 SDKs

Faster Warm-up time under Metadata Ejection

Access log for monitoring port 8091

CRAM & MD5 Support in .Net SDK

Client Side Log4Net Integration in .Net SDK

New Cluster object for Cluster Operations in SDK 2.0

Replica Read in SDK 2.0 with .Net and PHP

Added Support for Debian v7

Faster View indexing

Faster XDCR Synchronization

Faster ReplicateTo

Faster PersistTo

Faster Rebalance

Encrypted Data Access

Encrypted Admin Access

Encrypted View Access

Graceful Failover

Delta Node Recovery

Incremental Backup and Restore

Lower Latency XDCR

Auto tuning IO subsystem

Additional Configuration on .Net pooling interval

New repository for Yum for Ubuntu

Community Edition of 3.0 Release with Enterprise Edition

Ability to Name Clusters

Recap and Q&A

DEVELOPER

EMPOWERMENT

MISSION CRITICAL

SCALE

ULTRA HIGH

AVAILABILITY

SECURE

COMMUNICATION

SIMPLIFIED

ADMINISTRATION

• Fully Integrated JSON

Programmability

• Native Async and

Reactive

Programmability

• New Integration with

Popular Frameworks

• N1QL Preview Support

in SDKs

• Optimized Massive

Databases with Tunable

Memory

• Auto-Tuning Thread

Pool for Extreme IO

Throughput

• Faster Replication with

DCP

• Improved Durability

• Lower Latency Indexing

On Views

• Low Latency, In-Memory

XDCR

• Delta Node Recovery

• Faster Rebalance

• Faster Node Restart with

Efficient Warm-up

• New Incremental

Backup option

• Encryption for Data

Access

• Encryption for

Administrative Access

• XDCR Pause and

Resume

• Improved Resource

Governance

• Graceful Failover

• Cluster-wide Diagnostics

RECAP

What’s new in Couchbase Server v3.0

Today• Deep Dive: Tunable Memory in Couchbase Server 3.0 – Chiyoung SeoOctober 6 @ 4:20 pm - 5:05 pm

• Best Practices: Securing a Couchbase Server Deployment – Don PintoOctober 6 @ 4:20 pm - 5:05 pm

• Deep Dive into DCP: A Streaming Replication Protocol – Mike Wiederhold and meOctober 6 @ 5:10 pm - 5:55 pm

Tomorrow• Ultra–High Availability and Disaster Recovery with Couchbase Server – Anil KumarOctober 7 @ 10:50 am - 11:35 am

• Deep Dive: Near Real-Time Map / Reduce with Views in Couchbase Server 3.0 - SarathLakshman

October 7 @ 11:40 am - 12:25 pm

Related Talks

Feedback and Q&A

What's New in Couchbase Server 3.0: Couchbase Connect 2014

Data & Analytics

Transcript of What's New in Couchbase Server 3.0: Couchbase Connect 2014