Cvent: Building a microservice deployment pipeline – Couchbase Connect 2016
CFCamp 2016 - Couchbase Overview
-
Upload
aaron-benton -
Category
Technology
-
view
75 -
download
3
Transcript of CFCamp 2016 - Couchbase Overview
Couchbase OverviewAaron Benton
"A good programmer is someone who looks both ways before crossing a one-way street."
- Doug LinderCredit: Couchbase
4
Agenda
What is NoSQL?
What is Couchbase?
Couchbase Architecture
CFCouchbase
Queries
Demo (if time permits)
What is NoSQL?
6
What is NoSQL?
Non-Relational
Cluster Friendly
Generally Open-Source
21st Century
Schema-Less
7
Scaling
Scale Vertically (RDBMS)Add resources to a single node
in a system
Enhance the server (more CPU, more RAM, etc)
High availability, difficult to implement
Scale Horizontally (NoSQL)Add more nodes to a system
More servers, distributing load
High Availability, easy to implement
Container Friendly
Cattle
8
Pets
Any type of data
Flexible
Application Managed
Change is easy
Known Models
Fixed Fields
Data Types
Database Managed
Change can be difficult
9
Schemas
Relational Non-Relational
10
Types of NoSQL Databases
Key-ValueRedis
Riak
Memcached
DocumentCouchbase
CouchDB
MongoDB
Column-FamilyCassandra
Base
BigTable
GraphNeo4J
Giraph
OrientDB
11
In Development…
Objects are assembled as a whole:Cart
Order
Product
Profile
Saving these Objects requires:Deconstructing
Multiple Rows
Multiple Tables
12
Impedance Mismatch
"The object-relational impedance mismatch is a set of conceptual and technical difficulties that are often encountered when a relational database management system (RDBMS) is being used by a program written in an object-oriented programming language or style, particularly when objects or class definitions are mapped in a straightforward way to database tables or relational schema." - Wikipedia
13
ORM
14
Relational Models
Database
// order (structure / dictionary / object / map /etc)
order['order_id'] = 3492843;
order['order_date'] = "2016-07-14T18:27:22.586Z";
order['products'] = [{
'product_id' = 78323,
'quantity' = 2,
'price' = 39.99,
'sub_total' = 79.98,
}];
order['user_id'] = 123;
order['billing_address_1'] = "1302 Pleasant Ridge Rd";
order['billing_address_2'] = "";
order['billing_city'] = "Greensboro";
order['billing_region_code'] = "NC"; order['billing_postal_code'] = "27409";
order['billing_country_code'] = "US";
order['shipping_address_1'] = "1302 Pleasant Ridge Rd";
order['shipping_address_2'] = "";
order['shipping_city'] = "Greensboro";
order['shipping_region_code'] = "NC";
order['shipping_postal_code'] = "27409";
order['shipping_country_code'] = "US";
order['card_number'] = "3337151609084503";
order['expiration_month'] = 11;
order['expiration_year'] = 2019;
Code
15
Non-Relational Models
// order (structure / dictionary / object / map /etc)
order['order_id'] = 3492843;
order['order_date'] = "2016-07-14T18:27:22.586Z";
order['products'] = [{
'product_id' = 78323,
'quantity' = 2,
'price' = 39.99,
'sub_total' = 79.98,
}];
order['user_id'] = 123;
order['billing_address_1'] = "1302 Pleasant Ridge Rd";
order['billing_address_2'] = "";
order['billing_city'] = "Greensboro";
order['billing_region_code'] = "NC"; order['billing_postal_code'] = "27409";
order['billing_country_code'] = "US";
order['shipping_address_1'] = "1302 Pleasant Ridge Rd";
order['shipping_address_2'] = "";
order['shipping_city'] = "Greensboro";
order['shipping_region_code'] = "NC";
order['shipping_postal_code'] = "27409";
order['shipping_country_code'] = "US";
order['card_number'] = "3337151609084503";
order['expiration_month'] = 11;
order['expiration_year'] = 2019;
Database (JSON)Code
{
"order_id": 3492843,
"order_date": "2016-07-14T18:27:22.586Z",
"products": [{
'product_id': 78323,
'quantity': 2,
'price': 39.99,
'sub_total': 79.98,
}];
"user_id": 123,
"billing_address_1": "1302 Pleasant Ridge Rd",
"billing_address_2": "",
"billing_city": "Greensboro",
"billing_region_code": "NC",
"billing_postal_code": "27409",
"billing_country_code": "US",
"shipping_address_1": "1302 Pleasant Ridge Rd",
"shipping_address_2": "",
"shipping_city": "Greensboro",
"shipping_region_code": "NC",
"shippping_postal_code": "27409",
"shippping_country_code": "US",
"card_number": "3337151609084503",
"expiration_month": 11,
"expiration_year": 2019
}
BASEACID
16
Transaction Processing
Atomicity
Consistency
Isolation
Durability
Basically Available
Soft-State
Eventual Consistency
Relational Non-Relational
17
CAP Theorem
18
Only NoSQL?
Is SQL going away?
NoNoSQL databases are just another tool
What is Couchbase?
History
21
High availability cache
Key-value store
Document database
Embedded database
Sync management
Couchbase Server Couchbase Lite CouchbaseSync Gateway
Data management for a broad range of use cases
22
Couchbase Tenants
Flexible data model
Consistent performance at scale
High availability
Easy, affordable scalability
24x365
23
Buckets
24
Couchbase Connectors
Architecture
Couchbase Node
26
Couchbase Server Node
Single-node type means easier administration and scaling
Single installation
Two major components/processes: Data manager cluster manager
Data manager:
C/C++
Layer consolidation of caching and persistence
Cluster manager:
Erlang/OTP
Administration UI’s
Out-of-band for data requests
27
Couchbase Read Operation
APPLICATION SERVER
MANAGED CACHE
DISK
DISKQUEUE
REPLICATIONQUEUE
DOC 1
GETDOC 1
DOC 1
Single-node type means easier administration and scaling
Reads out of cache are extremely fast
No other process/system to communicate with
Data connection is a TCP-binary protocol
DOC 1
28
APPLICATION SERVER
MANAGED CACHE
DISK
DISKQUEUE
REPLICATIONQUEUE
Couchbase Write Operation
DOC 1
DOC 1DOC 1
Single-node type means easier administration and scaling
Writes are async by default
Application gets acknowledgement when successfully in RAM and can trade-off waiting for replication or persistence per-write
Replication to 1, 2 or 3 other nodes
Replication is RAM-based so extremely fast
Off-node replication is primary level of High Availability
Disk written to as fast as possible –no waiting
29
Couchbase Cache Ejection
APPLICATION SERVER
MANAGED CACHE
DISK
DISKQUEUE
REPLICATIONQUEUE
DOC 1
DOC 2DOC 3DOC 4DOC 5
DOC 1
DOC 2 DOC 3 DOC 4 DOC 5
Single-node type means easier administration and scaling
Layer consolidation means read through and write through cache
Couchbase automatically removes data that has already been persisted from RAM
30
APPLICATION SERVER
MANAGED CACHE
DISK
DISKQUEUE
REPLICATIONQUEUE
DOC 1
Couchbase Cache Miss
DOC 2 DOC 3 DOC 4 DOC 5
DOC 2 DOC 3 DOC 4 DOC 5
GETDOC 1
DOC 1
DOC 1
Single-node type means easier administration and scaling
Layer consolidation means 1 single interface for App to talk to and get its data back as fast as possible
Separation of cache and disk allows for fastest access out of RAM while pulling data from disk in parallel
Cluster Overview
Scaling
Architecture
Couchbase Cluster: Node and SDK Interaction
34
Auto sharding – Bucket and vBuckets
vB
Data buckets
vB
1 ….. 1024
Virtual buckets
A bucket is a logical, unique key space
Multiple buckets can exist within a single cluster of nodes
Each bucket has active and replica data sets (1, 2 or 3 extra copies)
Each data set has 1024 Virtual Buckets (vBuckets)
Each vBucket contains 1/1024th portion of the data set
vBuckets do not have a fixed physical server location
Mapping between the vBuckets and physical servers is called the cluster map
Document IDs (keys) always get hashed to the same vbucket
Couchbase SDK’s lookup the vbucket -> server mapping
35
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
Basic Operation
SHARD5
SHARD2
SHARD9
SHARD SHARD SHARD
SHARD4
SHARD7
SHARD8
SHARD SHARD SHARD
SHARD1
SHARD3
SHARD6
SHARD SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD SHARD
SHARD6
SHARD3
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD SHARD
Application has single logical connection to cluster (client object)
Data is automatically sharded resulting in even document data distribution across cluster
Each vbucket replicated 1, 2 or 3 times (“peer-to-peer” replication)
Docs are automatically hashed by the client to a shard
Cluster map provides location of which server a shard is on
Every read/write/update/delete goes to same node for a given key
Strongly consistent data access (“read your own writes”)
A single Couchbase node can achieve 100k’s ops/sec so no need to scale reads
36
Cluster Map
37
Cluster Map
38
Cluster Map – 2 nodes added
39
Rebalance
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
ACTIVE ACTIVE
REPLICA REPLICA
Couchbase Server 4 Couchbase Server 5
SHARD5
SHARD2
SHARD SHARD
SHARD4
SHARD SHARD
SHARD1
SHARD3
SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD SHARD
SHARD6
SHARD3
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD SHARD
SHARD7
SHARD
SHARD6
SHARD
SHARD8
SHARD9
SHARD
READ/WRITE/UPDATE
Application has single logical connection to cluster (client object)
Multiple nodes added or removed at once
One-click operation
Incremental movement of active and replica vbuckets and data
Client library updated via cluster map
Fully online operation, no downtime or loss of performance
40
Fail Over Node
ACTIVE ACTIVE ACTIVE
REPLICA REPLICA REPLICA
Couchbase Server 1 Couchbase Server 2 Couchbase Server 3
ACTIVE ACTIVE
REPLICA REPLICA
Couchbase Server 4 Couchbase Server 5
SHARD5
SHARD2
SHARD SHARD
SHARD4
SHARD SHARD
SHARD1
SHARD3
SHARD SHARD
SHARD4
SHARD1
SHARD8
SHARD SHARD
SHARDSHARD6
SHARD2
SHARD SHARD SHARD
SHARD7
SHARD9
SHARD5
SHARD SHARD
SHARD
SHARD7
SHARD
SHARD6
SHARDSHARD8
SHARD9
SHARD
SHARD3
SHARD1
SHARD3
SHARD
Application has single logical connection to cluster (client object)
When node goes down, some requests will fail
Failover is either automatic or manual
Client library is automatically updated via cluster map
Replicas not recreated to preserve stability
Best practice to replace node and rebalance
41
XDCR
Cross Datacenter Replication
Replication to other clusters
Bi-Directional
Uni-directional
Filtered Replication
What about Mongo?
43
MongoDB
Replication: Master - Slave
Primaries and Secondaries
Not all writes are local
Need for 3rd Party Cache
No Mobile Solution
Complex Topology
Database + Collections + BSON
Cluster with 2 replicas / backups
Distributed Load
44
Couchbase vs MongoDB
Who's using Couchbase?
46
Who's using Couchbase…?
CFCouchbase 2.0
48
Documents are integral to the SDKs.
All SDK’s support JSON format
In addition: Serialized objects, Unquoted Strings, Binary pass-through
A Document contains:
Couchbase SDK
22
Property Description
ID The bucket-unique identifier
Content The value that is stored
Expiry An expiration time
CAS Check-and-Set identifier
CF Couchbase 2.0 Brings
Java SDK (2.2.5)
N1QL Support w/ GSI Indexes
Replica Reads
Document Locking
Prepared Statements
Design Document Management
Expanded Config
And More…
1.* 2.0
add insert
incrdecr
counter
delete remove
newQuery n1qlQueryviewQuery
set upsert
setMulti upsertMulti
setWithCAS replaceWithCAS
Method Changes
50
Other Couchbase SDKs
Official SDKs Java
.NET
Node.js
Python
For each of these there is:
Full Document support
Interoperability
Common yet idiomatic Programming Model
Others: Erlang, Perl, TCL, Clojure, Scala
Also fully REST accessible
PHP
C / C++
Go
Ruby
JDBC and ODBC
51
Connecting to Couchbase
22
public boolean function onApplicationStart(){
application['couchbase'] = new cfcouchbase.CouchbaseClient({
servers: "127.0.0.1",
bucketName: "beer-sample"
});
return true;
}
public boolean function onApplicationEnd(){
application.couchbase.shutdown( 10 );
return true;
}
52
Writing Documents
22
var document = {
"id": "512_brewing_company",
"name": "(512) Brewing Company",
"city": "Austin",
"state": "Texas",
"code": "78745",
"country": "United States",
"phone": "512.707.2337",
"website": "http://512brewing.com/",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "(512) Brewing Company is a microbrewery located in the
heart of Austin that brews for the community using as many local,
domestic and organic ingredients as possible.",
"address": [
"407 Radam, F200"
],
"geo": {
"accuracy": "ROOFTOP",
"lat": 30.2234,
"lon": -97.7697
}
};
application.couchbase.upsert
id=document.id,
value=document
);
Define a Document Save the Document
53
Retrieving Documents
22
application.couchbase.get(
id="512_brewing_company"
);
Get the Document
54
Counter Operations
22
// increment the user counter
next_id = couchbase.counter(
id="user_counter",
value=1,
defaultValue=0
);
// write the user record
couchbase.insert("user_" & next_id, {
"user_id": next_id,
"name": "John Smith",
"email": "[email protected]"
});
Increment
// decrement the tickets counter
available_tix = couchbase.counter(
id="tickets_counter",
value=-1,
defaultValue=1000
);
// if there are no more tickets
// available redirect the user
if (available_tix < 1) {
cflocation(
url="sold-out.cfm",
addtoken=false
);
}
...
Decrement
Querying
56
Map / Reduce Views
In Couchbase, Map-Reduce is specifically used to create Indexes
Map functions are applied to JSON documents and their output or "emit" data is stored in an index
57
Querying
SELECT first_name, last_name, children FROM users
looks like SQL…WHERE EVERYchild IN children SATISFIES child.age > 10
END
58
N1QL Examples
INSERT INTO ecommerce ( KEY, VALUE )VALUES ("user_1021", {
"user_id": 1021,"doc_type": "user","first_name": "John","last_name": "Smith","email": "[email protected]"
})
UPDATE ecommerceUSE KEYS "user_1021"SET email = "[email protected]",
gender = "M",token = UUID()
RETURNING token
UPSERT INTO ecommerce ( KEY, VALUE )VALUES ("user_1021", {
"user_id": 1021,"doc_type": "user","first_name": "John","last_name": "Smith","email": "[email protected]"
})
DELETEFROM ecommerceUSE KEYS "user_1021"
N1QL Operators
59
Type Support
Arithmetic + - * / % -val
Collection ANY EVERY ARRAY FIRST EXISTS IN WITHIN
Comparison = == != <> > >= < <=(NOT) BETWEEN (NOT) LIKE IS (NOT) NULL IS (NOT) MISSING IS (NOT) VALUED
Conditional CASE expression WHEN value THEN expression
Construction Array [ value, value, ... ] Object { key:value, key:value, ... }
Logical AND OR NOT
String ||
N1QL Operators
60
ARRAY_AGG(EXP)
ARRAY_AGG(DISTINCT EXP)
AVG(EXP)
AVG(DISTINCT EXP)
COUNT(*)
COUNT(EXP)
COUNT(DISTINCT EXP)
MAX(EXP)
MIN(EXP)
SUM(EXP)
SUM(DISTINCT EXP)
Aggregate Functions
Object Functions
OBJECT_LENGTH(EXP)
OBJECT_NAMES(EXP)
OBJECT_PAIRS(EXP)
OBJECT_VALUES(EXP)
Conditionals - Unknowns
IFMISSING(EXP1, EXP2, …)
IFMISSINGORNULL(EXP1, EXP2, …)
IFNULL(EXP1, EXP2, …)
MISSINGIF(EXP1, EXP2)
NULLIF(EXP1, EXP2)
Conditionals - Numbers
IFINF(EXP1, EXP2, …)
IFNAN(EXP1, EXP2, …)
IFNANORINF(EXP1, EXP2, …)
NANIF(EXP1, EXP2)
NEGINFIF(EXP1, EXP2)
POSINFIF(EXP1, EXP2)
Comparison Functions
GREATEST(EXP1, EXP2)
LEAST(EXP1, EXP2)
Meta and UUID Functions
BASE64(EXP)
BASE64_ENCODE(EXP)
BASE64_DECODE(EXP)
META(EXP)
UUID()
Number Functions
ABS(EXP)
ACOS(EXP)
ASIN(EXP)
ATAN(EXP)
ATAN2(EXP1, EXP2)
CEIL(EXP)
COS(EXP)
DEGREES(EXP)
E(EXP)
EXP(EXP)
LN(EXP)
LOG(EXP)
FLOOR(EXP)
PI(EXP)
POWER(EXP1, EXP2)
RADIANS(EXP)
RANDOM([ EXP ])
ROUND(EXP [, DIGITS])
SIGN(EXP)
SIN(EXP)
SQRT(EXP)
TAN(EXP)
TRUNC(EXP [, DIGITS])
Type Checking Functions
ISARRAY(EXP)
ISATOM(EXP)
ISBOOLEAN(EXP)
ISNUMBER(EXP)
ISOBJECT(EXP1, EXP2)
ISSTRING(EXP)
TYPE(EXP)
Type Conversion Functions TOARRAY(EXP)
TOATOM(EXP)
TOBOOLEAN(EXP)
TONUMBER(EXP)
TOOBJECT(EXP)
TOSTRING(EXP)
N1QL Operators
61
ARRAY_APPEND(EXP, VAL)
ARRAY_AVG(EXP)
ARRAY_CONCAT(EXP1, EXP2)
ARRAY_CONTAINS(EXP, VAL)
ARRAY_COUNT(EXP)
ARRAY_DISTINCT(EXP)
ARRAY_IFNULL(EXP)
ARRAY_LENGTH(EXP)
ARRAY_MAX(EXP)
ARRAY_MIN(EXP)
ARRAY_POSITION(EXP, VAL)
ARRAY_PREPEND(VAL, EXP)
ARRAY_PUT(EXP, VAL)
ARRAY_RANGE(START, END [,STEP])
ARRAY_REMOVE(EXP, VAL)
ARRAY_REPEAT(VAL, N)
ARRAY_REPLACE(EXP, VAL1, VAL2 [,N])
ARRAY_REVERSE(EXP)
ARRAY_SORT(EXP)
ARRAY_SUM(EXP)
Array Functions Date Functions
CLOCK_MILLIS()
CLOCK_STR ([FMT ])
DATE_ADD_MILLIS(EXP, N, PART)
DATE_ADD_STR(EXP, N,PART)
DATE_DIFF_MILLIS(EXP1, EXP2, PART)
DATE_DIFF_STR(EXP1, EXP2, PART)
DATE_PART_MILLIS(EXP, PART)
DATE_PART_STR(EXP, PART)
DATE_TRUNC_MILLIS(EXP, PART)
DATE_TRUNC_STR(EXP, PART)
MILLIS(EXP)
STR_TO_MILLIS(EXP)
MILLIS_TO_STR(EXP [, FMT ])
MILLIS_TO_UTC(EXP [, FMT ])
MILLIS_TO_ZONE_NAME(EX, TZ[,FMT])
NOW_MILLIS()
NOW_STR([ FMT ])
STR_TO_MILLIS(EXP)
MILLIS(EXP)
STR_TO_UTC(EXP)
STR_TO_ZONE_NAME(EXP, TZ_NAME)
Number Functions
CONTAINS(EXP, SUBSTRING)
INITCAP(EXP )
TITLE(EXP)
LENGTH(EXP)
LOWER(EXP)
LTRIM(EXP [,CHARACTERS ])
POSITION(EXP, SUBSTRING)
REPEAT(EXP, N)
REPLACE(EXP, SBSTR, REPL [, N ])
RTRIM(EXP, [,CHARACTERS ])
SPLIT(EXP [, SEP ])
SUBSTR(EXP, POS[, LEN ])
TRIM(EXP [, CHARACTERS ])
UPPER(EXP)
Pattern Matching Functions
REGEXP_CONTAINS(EXP, PATTERN)
REGEXP_LIKE(EXP, PATTERN)
REGEXP_POSITION(EXP, PATTERN)
REGEXP_REPLACE(EXP, PTRN, REPL [, N ])
JSON Functions
DECODE_JSON(EXP)
ENCODE_JSON(EXP)
ENCODED_SIZE(EXP)
POLY_LENGTH(EXP)
62
Retrieving Documents
22
couchbase.n1qlQuery(
statement="
SELECT airport_id,
airport_name, airport_type,
municipality, geo, timezone,
airport_iata, airport_icao,
FROM `flight-data`
WHERE iso_country = $1
AND iso_region = $2
AND doc_type = $3
ORDER BY airport_name ASC
LIMIT 2
",
parameters=[
"US",
"US-ME",
"airport"
]
);
CF N1QL Query
Examples
64
Questions?
box install cfcouchbase
66
67
Resources
22
ortussolutions.com/products/cfcouchbase
ortussolutions.com/blog/category/cfcouchbase
developer.couchbase.com query.pub.couchbase.com
bit.ly/n1ql-41-language-reference
blog.couchbase.com
bit.ly/cfcouchbase-travel bit.ly/cfcouchbase-samples
bit.ly/node-fakeit
thepolyglotdeveloper.com
couchbase.com/nosql-resources/presentations