What's New in Couchbase Server 4.0 – Couchbase Live New York 2015
Transcript of What's New in Couchbase Server 4.0 – Couchbase Live New York 2015
What’s New InCouchbase Server 4.0
Chin Hong | VP Product Management, Couchbase
©2015 Couchbase Inc. 2
Simplified Development
Connected Bigdata Experience
Improved Performance
Simplified Security Compliance
Improved HA & DR
Improved Provisioning
Simplified Familiar and Flexible Query with N1QL
Integrated BI with ODBC/JDBC
Spatial Queries for Location Aware Applications
New Frameworks and Languages - LINQ, - Spring, - Go
Surround Big-data - Spark SQL- Spark Streams- Kafka, - Sqoop, - Elastic, - SOLR
Global Secondary Indexes for Snappy Queries
Independent Scaling with Multi-dimensional Scaling
Faster Reporting and Interactive Analytics with Views Queries
Enhanced Storage Performance with ForestDB and Bloom Filters
Integrated Enterprise Identity Management with LDAP Integration
Security Forensics with Admin Auditing
Improved Data Protection with Lower latency XDCR
High Performance Global Data Distribution with XDCR Filtering
Deployment with High Performance Containers: Docker
Expanded Public and Private Cloud Support- AWS,- Google,- Azure,- Joyent,- Cisco,- Verizon
New Enterprise Platforms- SUSE - Oracle Ent.Linux
13
2
2 55
4
N1QL
©2015 Couchbase Inc. 4
What is N1QL?
4
Flexibility of JSON
Power of SQL
©2015 Couchbase Inc. 5
How Does One Evaluate a Query Language?
ExpressiveDeclarativeEcosystem
©2015 Couchbase Inc. 6
N1QL is Expressive: E-Commerce Examples
SELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10
SELECT SUBSTR(purchases.purchasedAt, 0, 7) AS month, ROUND(SUM(product.unitPrice * items.count)/1000000, 3) revenueMillionFROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY SUBSTR(purchases.purchasedAt, 0, 7) ORDER BY month
©2015 Couchbase Inc. 7
N1QL is Expressive: Image Processing Example
SELECT m.image AS gallery_image, s.image AS source_image, s.x, s.y, m.r AS gallery_r, m.g AS gallery_g, m.b AS gallery_b, s.r AS source_r, s.g AS source_g, s.b AS source_b,FROM SourcePixels sJOIN GalleryMeans m ON KEY s.rgb
Source: http://www.metanautix.com/tr01
Learn more about N1QL @ query.couchbase.com/tutorial
©2015 Couchbase Inc. 8
N1QL is Declarative: What vs. How
You specify WHATCouchbase Server figures out HOW
©2015 Couchbase Inc. 9
N1QL is Declarative: Top 10 Selling ProductsSELECT product.name, SUM(items.count) AS unitsSold FROM purchases UNNEST purchases.lineItems AS items JOIN product ON KEYS items.product GROUP BY product ORDER BY unitsSold DESC LIMIT 10
WHAT
HOW
©2015 Couchbase Inc. 10
N1QL Connects To Your Enterprise Ecosystem
Standards-based drivers
Integrations, partnerships
ODBC / JDBC
App
CB Node
ODBC / JDBC
ETL
ODBC / JDBC
BI
ODBC / JDBC
Visualization
CB Node CB Node
©2015 Couchbase Inc. 11
Sessions on N1QL Introducing N1QL: New SQL Based Query Language for JSON
1:15 pm – 1:55 pm, Grand Ballroom N1QL and SDK Support for Java, .NET, and Node.js
3:50 pm – 4:30 pm, The Library Deep Dive into N1QL with Global Secondary Indexes
4:35 pm – 5:15 pm, Grand Ballroom Agile Data Modeling at DIRECTV
2:00 pm – 2:40 pm, Stillman Evolution of Couchbase at CenterEdge: From Cache to Query
3:50 pm – 4:30 pm, Stillman Nielsen’s Interactive Data Analytics with Couchbase N1QL
4:35 pm – 5:15 pm, Stillman Introduction to BI with Couchbase Server using Tableau, Informatica, Excel
and more4:35 pm – 5:15 pm, The Library
Indexing
©2015 Couchbase Inc. 13
Why Do We Need Indexes?
Indexes are used to quickly look up objects
meeting user-specified criteria without having to search over every
object in the database.
©2015 Couchbase Inc. 14
What Are the Different Types of Indexes?
Primary IndexesSecondary Indexes
Spatial IndexesText Indexes
©2015 Couchbase Inc. 15
Index Types … Let’s Use the Library Analogy
Source: http://www.montel.com/en/markets/library/
©2015 Couchbase Inc. 16
Secondary IndexesLocal secondary indexes – Map/Reduce views in 3.0 Co-located with data Higher write performance Lower read performance: scatter-gather Scaling bottleneck with a high number of indexes or data
nodes
Global secondary indexes (GSI) – New in 4.0 Higher query performance Async writes to a large number of global indexes Independently scaled and partitioned Isolated from Key-Value operations
©2015 Couchbase Inc. 17
Box on Points Box Intersects Multi Dimensions Geo + Others
Spatial Indexes Experimental in 3.0, now GA in 4.0
– Performance and stability improvements GeoJSON output Bounding-box and range queries on multiple dimensions
©2015 Couchbase Inc. 18
Text Indexes (Developer Preview)
…
wise
…
engineer……
…
…, akay1980, …
…, akay1980, …
engineers
engineer Exact
Match
Inverted Index
Apply the sametext analysisat search timethat we usedat index time.
©2015 Couchbase Inc. 19
Sessions on Indexes Understanding Views in Couchbase Server
11:45 am – 12:25 pm, Screening Room Deep Dive into N1QL with Global Secondary Indexes
4:35 pm – 5:15 pm, Grand Ballroom Developer Preview of Full Text Search
labs.couchbase.com/cbft
Multi-Dimensional Scaling
©2015 Couchbase Inc. 21
Workload and ScalingHorizontal scaling Partitions a dataset onto one or more homogenous nodes Each node runs the same mixed workloads Re-partition dataset with additional hardware capacity
Node 8
Index Service
Couchbase Cluster
Query Service
Data Service
Node 1
©2015 Couchbase Inc. 22
Multi-Dimensional ScalingIsolated Service for minimized interference Independent “zones” for query, index, and data services
Minimize indexing and query overhead on core key-value operations.
Index Service
Couchbase Cluster
Query Service Data Service
Node 1 Node 8
©2015 Couchbase Inc. 23
Multi-Dimensional ScalingIndependent scalability for the best computational capacity per service
Couchbase Cluster
node1 Node 8 Node 9
Data Service
Index ServiceQuery Service
More storage for indexing: scale up index service nodes.
More cores for query processing:scale up query service nodes.
Node 1
©2015 Couchbase Inc. 24
Session on Multi-Dimensional Scaling Deep Dive into N1QL with Global Secondary Indexes
4:35 pm – 5:15 pm, Grand Ballroom
ForestDB: Next-Generation Storage Engine
©2015 Couchbase Inc. 26
Common Storage Structures – B+-Tree
26
Reads Writes EnginesB+-Tree Good read performance if the
fan-out is high (“short” tree) for small fixed-length keys. Read performance degrades for variable-length keys.
Update-In-Place results in random writes and bad write latency.Append-Only file improves write performance but requires periodic compaction.
BerkeleyDB, InnoDB, Couchstore (append-only), WiredTiger B+
04/26
…
…
…Key
Value (or Pointer)
longer keys
©2015 Couchbase Inc. 27
Common Storage Structures – LSM-Tree
27
Reads Writes EnginesLSM-Tree
Reads may have to traverse multiple trees – typically worse than B+ tree.
WAL(Write-Ahead Log) improves writes within-memory trees that are appended to the end of the log.
LevelDB, RocksDB, Cassandra, WiredTiger LSM
…
In-memory
Sequential log
flush/merge merge
C1 tree C2 tree
merge
Capacity increases exponentially
©2015 Couchbase Inc. 28
Fast Storage – ForestDB
Fundamentally rethinking the requirements for fast storage Compact and efficient storage for a variety of data Unified engine that performs well for various workloads Unified engine that scales from small devices to large
servers Optimized for new SSD storage technology
28
ForestDB provides a compact index structure
that significantly reduces the disk I/Oand write amplification.
©2015 Couchbase Inc. 29
ForestDBForestDB Evaluations: LevelDB, RocksDB
29
Throughput
UP TO
FASTER6X
Future Proof
FITTED
FORSSD
Efficiency
UP TO
5X
COMPACT
©2015 Couchbase Inc. 30
Trie (prefix tree) for which the nodes are B+-Trees– HB+-Trie was originally presented at ACM SIGMOD 2011 Programming
Contest, by Jung-Sang Ahn who works at Couchbase(http://db.csail.mit.edu/sigmod11contest/sigmod_2011_contest_poster_jungsang_ahn.pdf)
HB+-Trie (Hierarchical B+-Trie)
©2015 Couchbase Inc. 31
Sessions on ForestDB Next Generation Storage Engine: ForestDB
5:20 pm – 6:00 pm, Grand Ballroom
Security
©2015 Couchbase Inc. 33
Previously… In 2.2 In 2.5 In 3.0 New in 4.0
SASL AuthN with Bucket PasswordsAdmin UserSecure Build
Platform
Read-Only User
Easy Admin Password
Reset
Non-Root User
Deployments
Secure Communication for XDCR
Encrypted Client- Server
Communication
Encrypted Admin Access
Access LogData-at-Rest Encryption
• Simplified compliance with admin auditing
• External identity management for admins using LDAP
Couchbase Security Features
In a fewslides ..
33
©2015 Couchbase Inc. 34
External Identity Management Using LDAP
34
Centralized identity management Define multiple read-only admins and full-admins Centralized security policy management for
admin accounts for stronger passwords, password rotation, and auto lockouts
Individual accountability; simplified compliance Define UIDs in LDAP, and map UIDs to the read-
only/full admin role in Couchbase Comprehensive audit trails with LDAP UIDs in
audit records
©2015 Couchbase Inc. 35
Admin Auditing in Couchbase
35
Rich audit events Over 25+ different, detailed admin audit events Auditing for tools including backup
Configurable auditing Configurable file target Support for time-based log rotation and audit
filtering
Easy integration JSON format allows for easy integration with
downstream systems using flume, logstash, and syslogd
©2015 Couchbase Inc. 36
Auditing a successful login
36
{ "timestamp":"2015-02-20T08:48:49.408-08:00", "id":8192, "name":"login success", "description":"Successful login to couchbase cluster", "role":"admin", "real_userid": { "source":"ns_server", "user":"bjones” }, "sessionid":"0fd0b5305d1561ca2b10f9d795819b2e", "remote":{"ip":"172.23.107.165", "port":59383}}
WHEN
WHO
WHAT
HOW
©2015 Couchbase Inc. 37
Sessions on Security Securing Your Couchbase Server Deployment
2:45 pm –3:25 pm, Grand Ballroom Transparent Data Encryption in Couchbase using Vormetric
3:50 pm –4:30 pm, Grand Ballroom
©2015 Couchbase Inc. 38
Simplified Development
Connected Bigdata Experience
Improved Performance
Simplified Security Compliance
Improved HA & DR
Improved Provisioning
Simplified Familiar and Flexible Query with N1QL
Integrated BI with ODBC/JDBC
Spatial Queries for Location Aware Applications
New Frameworks and Languages - LINQ, - Spring, - Go
Surround Big-data - Spark SQL- Spark Streams- Kafka, - Sqoop, - Elastic, - SOLR
Global Secondary Indexes for Snappy Queries
Independent Scaling with Multi-dimensional Scaling
Faster Reporting and Interactive Analytics with Views Queries
Enhanced Storage Performance with ForestDB and Bloom Filters
Integrated Enterprise Identity Management with LDAP Integration
Security Forensics with Admin Auditing
Improved Data Protection with Lower latency XDCR
High Performance Global Data Distribution with XDCR Filtering
Deployment with High Performance Containers: Docker
Expanded Public and Private Cloud Support- AWS,- Google,- Azure,- Joyent,- Cisco,- Verizon
New Enterprise Platforms- SUSE - Oracle Ent.Linux
13
2
2 55
4
Get Started with Couchbase Server 4.0: developer.couchbase.com/server
Get Trained on Couchbase: training.couchbase.com