NoSQL's biggest lie: SQL never went away - Martin Esmann

NoSQL's biggest lie: SQL never went away!Martin EsmannDeveloper Advocate, Couchbase

1

Let’s do something unspectacular

2

©2014 Couchbase Inc.

A query

3

SELECT * FROM `travel-sample` WHERE type = ‘airline’ AND country = ‘United Kingdom’;


A query

4

Did I say unspectacular?

5


That was JSON

6

So, what did we do?

7


We put the SQL back into NoSQL

8

But first, let’s take a step back

9


NoSQL?

10

Polyglot persistence is “…using multiple data

storage technologies, chosen based upon the

way data is being used by individual

applications. Why store binary images in

relational database, when there are better

storage systems?”

Martin Fowler and Pramod Sadalage


Types of NoSQL database

11

By Schumi4ever (Own work) [CC BY-‐SA 3.0 (http://creativecommons.org/licenses/by-‐sa/3.0)], via Wikimedia Commons


Key value

12

Email: [email protected]:email: {

“personal”: “[email protected]”,“work”: “[email protected]”

}


London

[email protected]@[email protected]@[email protected]@couchbase.com

Document

13

Developer Advocacy

[email protected]@[email protected]@[email protected]@[email protected]@couchbase.com

[email protected]:

{"city": ”Copenhagen","glasses": false,"team": "Developer Advocacy","music": ”Electronic!"

}


Document

14

London

[email protected]@[email protected]

Developer Advocacy


London and Developer Advocacy



And the others

15

Context is all

16


There's always a trade-‐off

17

• Offload from some other data store (i.e. caching)• Computation offload• Speed• Scalability• Availability• Flexibility in what you store• Query flexibility

Where Couchbase comes in

18


Couchbase Server 4.0

19

High availability cache

Key-‐value store

Document database N1QL SQL-‐like query

for JSON

N1QL: SQL for JSON

20


A user profile as JSON

21

How do we query that?

22


Querying the JSON profile

23

• Look-‐up documents: i.e. manual secondary indexing (2i)• Couchbase views: i.e. automated secondary indexing (2i)• N1QL


Manual 2i

24


Automatic 2i: views

25


N1QL

26


SELECT

27

SELECT ...


SELECT

28

SELECT 1 + 1;

{ "requestID": "3ccebac7-‐341a-‐4c31-‐a2c5-‐b46aaed54356", "signature": {

"$1": "number" }, "results": [

{ "$1": 2 }

], "status": "success", "metrics": {

"elapsedTime": "31.826219ms", "executionTime": "29.800616ms", "resultCount": 1, "resultSize": 31

}}


SELECT COUNT

29

SELECT COUNT(*) FROM `default`WHERE office = "London";

{"requestID": "6e733000-ac83-44ba-95a7-9b012e9c553d","signature": {

"$1": "number"},"results": [

{"$1": 6

}],"status": "success","metrics": {

"elapsedTime": "18.603124ms","executionTime": "18.327696ms","resultCount": 1,"resultSize": 31

}}


SELECT email FROM `default`WHERE office = "London";

SELECT

30


SELECT email FROM `default`WHERE office = "London"AND team = "Developer Advocacy";

SELECT

31


SELECT email FROM `default`WHERE office = "London"AND team != "Developer Advocacy";

SELECT

32

But this is JSON

33


SELECT conferences[0].nameAS event_name FROM `default`;

ARRAY ELEMENTS

34


SELECT DISTINCT conferences[0].nameAS event_name FROM `default`;

DISTINCT ARRAY ELEMENTS

35


SELECT DISTINCT conferences[0].nameAS event_name FROM `default`WHERE conferencesIS NOT MISSING;

REMOVE MISSING ITEMS

36


SELECT email AS person,conferences[0].name AS event FROM `default`WHERE ANY event in conferences SATISFIES event.name = "Droidcon Sweden" END;

WHO IS GOING TO DROIDCON SWEDEN?

37

What's going on underneath?

38


EXPLAIN SELECT email AS person,conferences[0].name AS event FROM `default`WHERE ANY event in conferences SATISFIES event.name = "Droidcon Sweden" END;

EXPLAIN

39

Is N1QL read-‐only?

40


Updating and deleting

41

• DELETE: provide the key to delete the document• INSERT: provide a key and some JSON to create a new document• UPSERT: as INSERT but will overwrite existing docs• UPDATE: change individual values inside existing docs


UPDATE

42


DELETE

43

A larger data-‐set: travel-‐sample

44


TRAVEL SAMPLE DATA

45


travel-‐sample

46


CREATE PRIMARY INDEX

47

CREATE PRIMARY INDEXON `travel-sample`USING GSI;


SELECT

48

SELECT * FROM `travel-sample`WHERE type = "airline";


SELECT

49


SELECT

50

SELECT * FROM `travel-sample`WHERE type = "airline"AND country = "United States";


SELECT

51

Indexes

52


CREATE INDEX

53

CREATE INDEX airline ON `travel-sample`(type)WHERE type = "airline"USING GSI;

JOINs

54


JOINs

55

• Retrieve data from two documents in a single SELECT• Join within a keyspace/bucket• Join across keyspaces/buckets


A SIMPLE JOIN

56

SELECT * FROM `travel-sample` r JOIN `travel-sample` a ON KEYS r.airlineidWHERE r.sourceairport="LHR" AND r.destinationairport = "SFO";


WHO FLIES LHR-‐>SFO?

57

SELECT DISTINCT a.name FROM `travel-sample` r JOIN `travel-sample` a ON KEYS r.airlineidWHERE r.sourceairport="LHR" AND r.destinationairport = "SFO";


UNNEST

58

• Breaks out nested JSON from the results


SOMETHING USEFUL

59

SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipmentFROM `travel-sample` r UNNEST r.schedule sJOIN `travel-sample` a ON KEYS r.airlineidWHERE r.sourceairport="LHR" AND r.destinationairport = "SFO"AND s.day=1ORDER BY s.utc;

N1QL vs View’s

60


N1QL and views

61

N1QL Views

Ad-hoc querying Predictable queries

JSON in and JSON out Number crunching

Large growth clusters Multi-dimensional/geospatialqueries

Multi-‐dimensional scaling

62


Scaling out

63

Horizontal scaling▪ Partitions a dataset onto one or more homogenous nodes▪ Each node runs the same mixed workloads▪ Re-‐partition dataset with additional hardware capacity


What is Multi-‐Dimensional Scalability? MDS is the architecture that enables independent scaling of data, query, and indexing workloads.


64



65

Isolated Service for minimized interference▪ Independent “zones” for query, index, and data services

Minimize indexing and query overhead on core key-‐value operations.



66

Independent scalability for the best computational capacity per service

Heavier indexing (index more fields): scale up index service nodes.

More RAM for query processing:scale up query service nodes.

Geospatial querying

67


Geospatial views

68

• Experimental in 3.0, now GA in 4.0• Performance and stability improvements• GeoJSON output• Bounding-‐box and range queries on

multiple dimensions

What else is in Couchbase Server 4.0?

69


Other new things in Couchbase Server 4.0

70

• ForestDB: GSIs are stored with a new storage engine• Filtered XDCR: more efficient cross-‐data centre replication• Security: LDAP, admin auditing

Developers?SDK support for N1QL!

71


.NET SDK: Connection

72

// Option 1: Create Cluster and open bucket var cluster = new Cluster(config);var bucket = cluster.OpenBucket(bucketName);

// Option 2: ClusterHelper and open bucket (Singleton, thread safe bucket instanceClusterHelper.Initialize(config);var cluster = ClusterHelper.Get();var bucket = ClusterHelper.GetBucket(bucketName);

// Close connectioncluster.CloseBucket(bucket);cluster.Dispose();


.NET SDK: Client Configuration

73

// Cluster Configurationvar config = new ClientConfiguration

{Servers = new List<Uri> { new Uri("http://10.211.55.2:8091/pools"),},UseSsl = false,BucketConfigs = new Dictionary<string, BucketConfiguration>

{{

"travel-‐sample", new BucketConfiguration{

BucketName = "travel-‐sample",UseSsl = false,Password = "",PoolConfiguration = new PoolConfiguration

{MaxSize = 10,MinSize = 5

}}

}}

};


.NET SDK: Basic Operations

74

// Create document (dynamic)var doc = new Document<dynamic>{

Id = "doc1",Content = new{

Id = "doc1",Title = "My Document",Type = "basic",Pages = 3

}};

// update or create documentvar upsertResult = await bucket.UpsertAsync<dynamic>(doc);

// Get documentvar getResult = bucket.GetDocument<dynamic>("key1");

// Delete documentvar deleteResult = await bucket.RemoveAsync<dynamic>(doc);

// Check if doc/key exsistsvar exsists = await bucket.ExistsAsync("key1");


.NET SDK: N1QL Query (raw)

75

// Create Query Request. Raw API requestvar query = QueryRequest.Create("SELECT COUNT(*) FROM `travel-‐sample` WHERE type = 'airline'");

// Execute Query. var response = await bucket.QueryAsync<dynamic>(query);

// Convert result to string for easy console print out.var result = JsonConvert.SerializeObject(response, Formatting.Indented);


.NET SDK: N1QL Query (Linq2Couchbase)

76

// Lambda syntax

var result = ClusterHelper.GetBucket("beer-‐sample").Queryable<Beer>().Where(a => a.Type == "beer").Select(a => a).Join(

ClusterHelper.GetBucket("beer-‐sample").Queryable<Brewery>().Where(airline => airline.Type == "brewery"),innerKey => innerKey.BreweryId,outerKey => N1Ql.Key( outerKey), (inner,outer) => new { Inner = inner, Outer = outer})

.Take(1)

.ToList();


.NET SDK: N1QL Query (Linq2Couchbase)

77

// Query syntax

var query = from beer in QueryFactory.Queryable<Beer>(mockBucket.Object)join breweryGroup in QueryFactory.Queryable<Brewery>(mockBucket.Object)on beer.BreweryId equals N1Ql.Key(breweryGroup) into bgfrom brewery in bg.DefaultIfEmpty()select new { beer.Name, beer.Abv, BreweryName = brewery.Name };

https://github.com/couchbaselabs/N1QL-‐Intro-‐dotNET-‐Supplement-‐Demo-‐code

Next Steps

Couchbase Developer Portal

developer.couchbase.com

80

Forums

http://forums.couchbase.com

81

Thanks for listening!

NoSQL's biggest lie: SQL never went away - Martin Esmann

Data & Analytics

Transcript of NoSQL's biggest lie: SQL never went away - Martin Esmann