Utilizing arrays: modeling, indexing, and querying
Transcript of Utilizing arrays: modeling, indexing, and querying
![Page 1: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/1.jpg)
©2016 Couchbase Inc. 1
{ "Utilizing Arrays" : ["Modeling", "Querying",
"Indexing"] }
Keshav MurthyDirector, Couchbase R&D
![Page 2: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/2.jpg)
©2016 Couchbase Inc. 2©2016 Couchbase Inc.
Agenda
• Introduction to Arrays• Data Modeling with Arrays• Query Performance With Arrays
• Array Indexing• Fun With Arrays
• Query Performance• Tag Search• String Search
![Page 3: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/3.jpg)
©2016 Couchbase Inc. 3
Introduction To Arrays
![Page 4: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/4.jpg)
©2016 Couchbase Inc. 4©2016 Couchbase Inc.
Every N1QL query returns Arrayscbq> select distinct type from `travel-sample`;{… "results": [ { "type": "route“ }, { "type": "airport” }, { "type": "hotel" }, { "type": "airline” }, { "type": "landmark” } ] , "status": "success", "metrics": { "elapsedTime": "840.518052ms", "executionTime": "840.478414ms", "resultCount": 5, "resultSize": 202 }}
Results from every query is an array.
cbq> SELECT * FROM `travel-sample`WHERE type = 'airport' and faa = 'BLR';{ "results": [], "metrics": { "elapsedTime": "9.606755ms", "executionTime": "9.548749ms", "resultCount": 0, "resultSize": 0 }}
![Page 5: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/5.jpg)
©2016 Couchbase Inc.
![Page 6: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/6.jpg)
©2016 Couchbase Inc. 6©2016 Couchbase Inc.
Introduction to Arrays
• An arrangement of quantities or symbols in rows and columns; a matrix
• An indexed set of related elements
![Page 7: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/7.jpg)
©2016 Couchbase Inc. 7©2016 Couchbase Inc.
JSON Arrays
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "hobbies" : ["lego", "piano", "badminton", "robotics"], "scores" : [3.4, 2.9, 9.2, 4.1], "legos" : [ true, 9292, "fighter 2", { "name" : "Millenium Falcon", "type" : "Starwars" } ]}
• Arrays in JSON can contain simply values, or any combination of JSON types within the same array.
• No type or structure enforcement within the array.
![Page 8: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/8.jpg)
©2016 Couchbase Inc. 8©2016 Couchbase Inc.
JSON Arrays
{ "Name": "Jane Smith", "DOB" : "1990-01-30", "phones" : [ "+1 510-523-3529", "+1 650-392-4923" ], "Billing": [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]}
Billing has two credit card entries, stored as an ARRAY
Two phone number entries
![Page 9: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/9.jpg)
©2016 Couchbase Inc. 9©2016 Couchbase Inc.
JSON Arrays : Syntax Diagram
![Page 10: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/10.jpg)
©2016 Couchbase Inc. 10
Data Modeling with Arrays
![Page 11: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/11.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Properties of Real-World Data
• Rich structure• Attributes, Sub-structure
• Relationships• To other data
• Value evolution• Data is updated
• Structure evolution• Data is reshaped
Customer
Name
DOB
Billing
Connections
Purchases
Jane Smith
Jan-30-1990
![Page 12: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/12.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Modeling Data in Relational World
Billing
ConnectionsPurchases
Contacts
Customer
• Rich structure• Normalize & JOIN Queries
• Relationships• JOINS and Constraints
• Value evolution• INSERT, UPDATE, DELETE
• Structure evolution• ALTER TABLE• Application Downtime• Application Migration• Application Versioning
![Page 13: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/13.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using JSON For Real World Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer{ "Name" : "Jane Smith", "DOB" : "1990-01-30"}
• The primary (CustomerID) becomes the DocumentKey
• Column name-Column value become KEY-VALUE pair.
{ "Name" : { "fname": "Jane", "lname": "Smith" } "DOB" : "1990-01-30"}
OR
Customer DocumentKey: CBL2015
![Page 14: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/14.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using JSON to Store Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" } ]}
CustomerID Type Cardnum Expiry
CBL2015 visa 5827… 2019-03
Table: Billing
• Rich Structure & Relationships• Billing information is stored as a sub-document• There could be more than a single credit card. So, use an array.
Customer DocumentKey: CBL2015
![Page 15: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/15.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using JSON to Store Data
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
Table: Customer
{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ]}
Customer DocumentKey: CBL2015
CustomerID Type Cardnum Expiry
CBL2015 visa 5827… 2019-03
CBL2015 master 6274… 2018-12
Table: Billing
Value evolution Simply add additional array element or
update a value.
![Page 16: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/16.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using JSON to Store Data
CustomerID ConnId NameCBL2015 XYZ987 Joe SmithCBL2015 SKR007 Sam SmithCBL2015 RGV492 Rav Smith
Table: Connections{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2542-5847-3949", "expiry" : "2018-12" } ], "Connections" : [ { "ConnId" : "XYZ987", "Name" : "Joe Smith" }, { "ConnId" : "SKR007", "Name" : "Sam Smith" }, { "ConnId" : "RGV491", "Name" : "Rav Smith" }}
Structure evolution Simply add new key-value pairs No downtime to add new KV pairs Applications can validate data Structure evolution over time.
Relations via Reference
Customer DocumentKey: CBL2015
![Page 17: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/17.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using JSON to Store Data{ "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "ConnId" : "XYZ987", "Name" : "Joe Smith" }, { "ConnId" : "SKR007", "Name" : "Sam Smith" }, { "ConnId" : "RGV491", "Name" : "Rav Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ]}
CustomerID Name DOB
CBL2015 Jane Smith 1990-01-30
CustomerID
Type Cardnum Expiry
CBL2015 visa 5827… 2019-03
CBL2015 master
6274… 2018-12
CustomerID ConnId Name
CBL2015 XYZ987 Joe Smith
CBL2015 SKR007 Sam Smith
CBL2015 RGV492 Rav Smith
CustomerID item amt
CBL2015 mac 2823.52
CBL2015 ipad2 623.52
CustomerID ConnId Name
CBL2015 XYZ987 Joe Smith
CBL2015 SKR007 Sam Smith
Contacts
CustomerBilling
ConnectionsPurchases
Customer DocumentKey: CBL2015
![Page 18: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/18.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Models for Representing Data
Data Concern Relational Model JSON Document Model (NoSQL)
Rich Structure Multiple flat tables Constant assembly / disassembly
Documents No assembly required!
Relationships Represented Queried (SQL)
Represented N1QL, MongoDB, CQL
Value Evolution Data can be updated Data can be updated
Structure Evolution Uniform and rigid Manual change (disruptive)
Flexible Dynamic change
![Page 19: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/19.jpg)
©2016 Couchbase Inc. 19
Querying Arrays
![Page 20: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/20.jpg)
©2016 Couchbase Inc. 20©2016 Couchbase Inc.
Querying Arrays
• Array Access• Expressions• Functions• Aggregates
• Statements• Array Clauses
![Page 21: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/21.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Access: Expressions, Functions and Aggregates.
21
• EXPRESSIONS• ARRAY• ANY• EVERY• IN• WITHIN• Construct [elem]• Slice
array[start:end]• Selection
array[#pos]
• FUNCTIONS• ISARRAY• TYPE• ARRAY_APPEND• ARRAY_CONCAT• ARRAY_CONTAINS• ARRAY_DISTINCT• ARRAY_IFNULL• ARRAY_FLATTEN• ARRAY_INSERT• ARRAY_INTERSECT• ARRAY_LENGTH• ARRAY_POSITION
• AGGREGATES• ARRAY_AVG• ARRAY_COUNT• ARRAY_MIN• ARRAY_MAX
• FUNCTIONS• ARRAY_PREPEND• ARRAY_PUT• ARRAY_RANGE• ARRAY_REMOVE• ARRAY_REPEAT• ARRAY_REPLACE• ARRAY_REVERSE• ARRAY_SORT• ARRAY_STAR• ARRAY_SUM
![Page 22: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/22.jpg)
©2016 Couchbase Inc. 22©2016 Couchbase Inc.
Array access{ "Name": "Jane Smith", "DOB" : "1990-01-30", "phones" : [ "+1 510-523-3529", "+1 650-392-4923" ], "Billing": [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]}
SELECT phones from t;[ { "phones": [ "+1 510-523-3529", "+1 650-392-4923" ] }]
SELECT phones[1] from t;[ { "$1": "+1 650-392-4923" }]SELECT phones[0:1] from t;[ { "$1": [ "+1 510-523-3529" ] }]
![Page 23: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/23.jpg)
©2016 Couchbase Inc. 23©2016 Couchbase Inc.
Array access: Expressions and functions{ "Name": "Jane Smith", "DOB" : "1990-01-30", "phones" : [ "+1 510-523-3529", "+1 650-392-4923" ], "Billing": [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]}
SELECT Billing[0].cardnum from t;[ { "cardnum": "5827-2842-2847-3909" }]SELECT Billing[*].cardnum from t;[ { "cardnum": [ "5827-2842-2847-3909", "6274-2542-5847-3949" ] }]SELECT ISARRAY(Name) name, ISARRAY(phones) phones from t;[ { "name": false, "phones": true }]
![Page 24: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/24.jpg)
©2016 Couchbase Inc. 24©2016 Couchbase Inc.
Array access : Functions{ "Name": "Jane Smith", "DOB" : "1990-01-30", "phones" : [ "+1 510-523-3529", "+1 650-392-4923" ], "Billing": [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]}
SELECT ARRAY_CONCAT(phones, ["+1 408-284-2921"]) from t;[ { "$1": [ "+1 510-523-3529", "+1 650-392-4923", "+1 408-284-2921" ] }]
SELECT ARRAY_COUNT(Billing) billing, ARRAY_COUNT(phones) phones from t;[ { "billing": 2, "phones": 2 }]
![Page 25: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/25.jpg)
©2016 Couchbase Inc. 25©2016 Couchbase Inc.
Array access : Functions
SELECT phones, ARRAY_REVERSE(phones) reverse from t;
{ "phones": [ "+1 510-523-3529", "+1 650-392-4923" ], "reverse": [ "+1 650-392-4923", "+1 510-523-3529" ] }]
SELECT phones, ARRAY_INSERT(phones, 0, "+1 415-439-4923") newlist from t;[ { "billing": 2, "phones": 2 }]
SELECT phones, ARRAY_INSERT(phones, 0, "+1 415-439-4923") newlist from t;[ { "newlist": [ "+1 415-439-4923", "+1 510-523-3529", "+1 650-392-4923" ], "phones": [ "+1 510-523-3529", "+1 650-392-4923" ] }]
![Page 26: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/26.jpg)
©2016 Couchbase Inc. 26©2016 Couchbase Inc.
Array access : Aggregates
SELECT ARRAY_MIN(Billing) AS minbill FROM t;[ { "minbill": { "cardnum": "5827-2842-2847-3909", "expiry": "2019-03", "type": "visa" } }]
SELECT name, ARRAY_AVG(reviews[*].ratings[*].Overall) AS avghotelrating FROM `travel-sample` WHERE type = 'hotel' ORDER BY avghotelrating descLIMIT 3;
[ { "avghotelrating": 5, "name": "Culloden House Hotel" }, { "avghotelrating": 5, "name": "The Bulls Head" }, { "avghotelrating": 5, "name": "La Pradella" }]
![Page 27: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/27.jpg)
©2016 Couchbase Inc. 27©2016 Couchbase Inc.
SELECT: ARRAY & FIRST Expression
ARRAY: The ARRAY operator lets you map and filter the elements or attributes of a collection, object, or objects. It evaluates to an array of the operand expression, that satisfies the WHEN clause, if provided.
SELECT ARRAY [name, r.ratings.`Value`] FOR r IN reviews WHEN r.ratings.`Value` = 4 END FROM `travel-sample` WHERE type = 'hotel'
SELECT FIRST [name, r.ratings.`Value`] FOR r IN reviews WHEN r.ratings.`Value` = 4 END FROM `travel-sample` WHERE type = 'hotel'
FIRST: The FIRST operator enables you to map and filter the elements or attributes of a collection, object, or objects. It evaluates to a single element based on the operand expression that satisfies the WHEN clause, if provided.
![Page 28: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/28.jpg)
©2016 Couchbase Inc. 28©2016 Couchbase Inc.
Statements
• INSERT• INSERT documents with arrays• INSERT multiple documents with arrays• INSERT result of documents from SELECT
• UPDATE• UPDATE specific elements and objects within an array
• DELETE• DELETE documents based on values within one or more arrays
• MERGE• MERGE documents to INSERT, UPDATE or DELETE documents.
• SELECT• Fetch documents given an array of keys• JOIN based on array of keys• Predicates (filters) on arrays• Array expressions, functions and aggregates• UNNEST, NEST operations
![Page 29: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/29.jpg)
©2016 Couchbase Inc. 29©2016 Couchbase Inc.
Statements:INSERT
INSERT INTO customer VALUES ("KEY01", { "cid": "ABC01", "orders": ["LG012", "LG482", "LG134"] });
INSERT INTO customer VALUES (("KEY01", { "cid": "XYC21", "orders": ["LG92", "LG859"] }),
VALUES (("KEY04", { "cid": "PQR49", "orders": ["LG47", "LG09", "LG134"] }), VALUES (("KEY09", { "cid": "KDL29", "orders": ["LG082"] });
INSERT INTO customer ( KEY uuid(), value c ) SELECT mycustomers AS c FROM newcustomers AS n WHERE n.type = "premium";
![Page 30: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/30.jpg)
©2016 Couchbase Inc. 30©2016 Couchbase Inc.
Statements: DELETE
DELETE FROM customerWHERE orders = ["LG012", "LG482", "LG134"];
DELETE FROM customerWHERE ANY o IN orders SATISFIES o = "LG012" END;
DELETE FROM customerWHERE ANY o IN orders SATISFIES o = "LG012" ENDRETURNING meta().id, *;
![Page 31: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/31.jpg)
©2016 Couchbase Inc. 31©2016 Couchbase Inc.
Statements:UPDATE
UPDATE customer USE KEYS ["KEY091"] SET orders = ["LG012", "LG482", "LG134"];
UPDATE customer USE KEYS ["KEY091"] SET orders = ARRAY_REMOVE(orders, "LG012") ;
UPDATE customer USE KEYS ["KEY091"] SET orders = ARRAY_APPEND(orders, "LG892") ;
![Page 32: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/32.jpg)
©2016 Couchbase Inc. 32©2016 Couchbase Inc.
Statements : SELECT
• SELECT• Array predicates• NEST, UNNEST• Fetch documents given an array of keys• JOIN based on array of keys
![Page 33: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/33.jpg)
©2016 Couchbase Inc. 33
SELECT statementARRAY PREDICATES
![Page 34: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/34.jpg)
©2016 Couchbase Inc. 34©2016 Couchbase Inc.
SELECT: Array predicates
• ANY• EVERY• SATISFIES• IN• WITH• WHEN
![Page 35: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/35.jpg)
©2016 Couchbase Inc. 35©2016 Couchbase Inc.
SELECT: Array predicates• Arrays and Objects: Arrays are
compared element-wise. Objects are first compared by length; objects of equal length are compared pairwise, with the pairs sorted by name.
• IN clause: Use this when you want to evaluate based on specific field.
• WITHIN clause: Use this when you don’t know which field contains the value you’re looking for. The WITHIN operator evaluates to TRUE if the right-side value contains the left-side value as a child or descendant. The NOT WITHIN operator evaluates to TRUE if the right-side value does not contain the left-side value as a child or descendant.
SELECT * FROM `travel-sample` WHERE type = 'hotel’ AND ANY r IN reviews SATISFIES r.ratings.`Value` >= 3 END;SELECT * FROM `travel-sample` WHERE type = 'hotel’ AND ANY r WITHIN reviews SATISFIES r LIKE '%Ozella%' END;
• EVERY: EVERY is a range predicate that tests a Boolean condition over the elements or attributes of a collection, object, or objects. It uses the IN and WITHIN operators to range through the collection.
SELECT * FROM `travel-sample` WHERE type = 'hotel’ AND EVERY r IN reviews SATISFIES r.ratings.Cleanliness >= 4 END;
![Page 36: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/36.jpg)
©2016 Couchbase Inc. 36©2016 Couchbase Inc.
SELECT: Array predicates• ARRAY_CONTAINS
• Returns true if the array contains value.
SELECT name, t.public_likes FROM `travel-sample` t WHERE type="hotel" AND ARRAY_CONTAINS(t.public_likes, "Vallie Ryan") = true;[ { "name": "Medway Youth Hostel", "public_likes": [ "Julius Tromp I", "Corrine Hilll", "Jaeden McKenzie", "Vallie Ryan", "Brian Kilback", "Lilian McLaughlin", "Ms. Moses Feeney", "Elnora Trantow" ] }]
![Page 37: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/37.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Expressions, Functions and Aggregates.
37
• EXPRESSIONS• ARRAY• ANY• EVERY• IN• WITHIN• Construct [elem]• Slice
array[start:end]• Selection
array[#pos]
• FUNCTIONS• ISARRAY• TYPE• ARRAY_APPEND• ARRAY_CONCAT• ARRAY_CONTAINS• ARRAY_DISTINCT• ARRAY_IFNULL• ARRAY_FLATTEN• ARRAY_INSERT• ARRAY_INTERSECT• ARRAY_LENGTH• ARRAY_POSITION
• AGGREGATES• ARRAY_AVG• ARRAY_COUNT• ARRAY_MIN• ARRAY_MAX• ARRAY_SUM
• FUNCTIONS• ARRAY_PREPEND• ARRAY_PUT• ARRAY_RANGE• ARRAY_REMOVE• ARRAY_REPEAT• ARRAY_REPLACE• ARRAY_REVERSE• ARRAY_SORT• ARRAY_STAR
![Page 38: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/38.jpg)
©2016 Couchbase Inc. 38©2016 Couchbase Inc.
SELECT: ARRAY & FIRST Expression
ARRAY: The ARRAY operator lets you map and filter the elements or attributes of a collection, object, or objects. It evaluates to an array of the operand expression, that satisfies the WHEN clause, if provided.
SELECT ARRAY [name, r.ratings.`Value`] FOR r IN reviews WHEN r.ratings.`Value` = 4 END FROM `travel-sample` WHERE type = 'hotel'
SELECT FIRST [name, r.ratings.`Value`] FOR r IN reviews WHEN r.ratings.`Value` = 4 END FROM `travel-sample` WHERE type = 'hotel'
FIRST: The FIRST operator enables you to map and filter the elements or attributes of a collection, object, or objects. It evaluates to a single element based on the operand expression that satisfies the WHEN clause, if provided.
![Page 39: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/39.jpg)
©2016 Couchbase Inc. 39
SELECT statementUNNEST and NEST
![Page 40: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/40.jpg)
©2016 Couchbase Inc. 40©2016 Couchbase Inc.
Querying Arrays: UNNEST
• UNNEST : If a document or object contains an array, UNNEST performs a join of the nested array with its parent document. Each resulting joined object becomes an input to the query. UNNEST, JOINs can be chained.
SELECT r.author, COUNT(r.author) AS authcountFROM `travel-sample` t UNNEST reviews rWHERE t.type="hotel" GROUP BY r.authorORDER BY COUNT(r.author) DESCLIMIT 5;[ { "authcount": 2, "author": "Anita Baumbach" }, { "authcount": 2, "author": "Uriah Gutmann" }, { "authcount": 2, "author": "Ashlee Champlin" }, { "authcount": 2, "author": "Cassie O'Hara" }, { "authcount": 1, "author": "Zoe Kshlerin" }]
![Page 41: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/41.jpg)
©2016 Couchbase Inc. 41©2016 Couchbase Inc.
Querying Arrays: NEST
• NEST is the inverse of UNNEST.• Nesting is conceptually the inverse
of unnesting. Nesting performs a join across two keyspaces. But instead of producing a cross-product of the left and right inputs, a single result is produced for each left input, while the corresponding right inputs are collected into an array and nested as a single array-valued field in the result object.
SELECT * FROM `travel-sample` routeNEST `travel-sample` airline ON KEYS route.airlineid WHERE route.type = ‘airline' LIMIT 1;
[ { "airline": [ { "callsign": "AIRFRANS", "country": "France", "iata": "AF", "icao": "AFR", "id": 137, "name": "Air France", "type": "airline" } ], "route": { "airline": "AF", "airlineid": "airline_137", "destinationairport": "MRS", "distance": 2881.617376098415, "equipment": "320", "id": 10000, "schedule": [ { "day": 0, "flight": "AF198", "utc": "10:13:00" }, { "day": 0, "flight": "AF547", "utc": "19:14:00" }, { "day": 0, "flight": "AF943", "utc": "01:31:00" }, { "day": 1, "flight": "AF356", "utc": "12:40:00" }, { "day": 1, "flight": "AF480", "utc": "08:58:00" }, { "day": 1, "flight": "AF250", "utc": "12:59:00" }, { "day": 1, "flight": "AF130", "utc": "04:45:00" }, { "day": 2,
![Page 42: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/42.jpg)
©2016 Couchbase Inc. 42
Query Performance with Arrays
![Page 43: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/43.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing
• Before 4.5, creating index on array attribute would index the entire array as a single scalar value.
CREATE INDEX i1 ON `travel-sample`(schedule);
"schedule": [ { "day" : 0, "flight" : "AI111", "utc" : "1:11:11"} }, { "day": 1, "flight": "AF552", "utc": "14:41:00" }, { "day": 2, "flight": "AF166",
"utc": "08:59:00" }, …]
![Page 44: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/44.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing - motivation
[ { "day" : 0, "special_flights" : [ { "flight" : "AI111", "utc" : ”1:11:11"}, { "flight" : "AI222", "utc" : ”2:22:22" } ] }, { "day": 1, "flight": "AF552", "utc": "14:41:00” }, …]
"London":[ "London", "Tokyo", "NewYork", …]
![Page 45: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/45.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Why array indexing?• When NoSQL databases asked customers to denormalize, they
put the child table info into arrays in parent tables. • E.g. Each customer doc had all phone numbers, contacts, orders in arrays.
• Not easy to query - need to specify full array value in where predicates. • Ex: list of users who purchased a product – Unknown values & large list
• Was not possible to index part of the array with objects• Bloated index size (indexes whole array value)• Example: Index just the day field in array of flights in schedule.
• Performance Limitation • ANY…IN or WITHIN array
• Ease of querying - Must specify full array value in WHERE-clause• Manageable for Known or handful of values • Difficult for Unknown or Large list of values.
![Page 46: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/46.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Who wants array indexing?
• Find my crew based on the airline.WHERE ANY p IN ods.pilot satisfies p.filen = ”XYZ1012" END ;
• Find my customer based on one of the emails on the customerWHERE ANY a IN u.telecom SATISFIES a.system = ‘email’ AND a.value = ‘[email protected]’ END ;
• Find service qualification based on arrays of arrays.WHERE ANY c_0 IN `item`.`blackoutserviceblocklist` SATISFIES
ANY c_1 IN c_0.`blackoutserviceblock`.`ppvservicelist` SATISFIES c_1.`ppvservice`.`eventcode` = "E001"
END END ;
![Page 47: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/47.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
What is Array Indexing?• Enables visibility into the array structure
schedule =
• Subset of array elements can be indexed & searched efficiently
[ { "day" : 0, "special_flights" : [ { "flight" : "AI111" , "utc" : "1:11:11"}, { "flight" : "AI932" , "utc" : "2:22:22"} ] }, { "day": 1, "flight": "AF552", "utc": "14:41:00" }, …]
![Page 48: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/48.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
How Array Indexing Helps?
• Index only required elements or attributes in the the array
• Efficient on Index storage & search time
• Benefits are lot more significant for nested arrays/objects
![Page 49: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/49.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
How Array Indexing Helps -- Example
"schedule”:[ { "day" : 0, "special_flights" : [ { "flight" : "AI111", "utc" : "1:11:11"}, { "flight" : "AI222", "utc" : "2:22:22"} ] }, { "day": 1, "flight": "AF552", "utc": "14:41:00" }, { "day": 2, "flight": "AF166",
"utc": "08:59:00" }, …]
"fli
ght"
: "A
F552
","f
ligh
t":
"AF1
66",
…
Array Index in Couchbase
![Page 50: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/50.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Create Array Index
• No syntax changes to DML statements• Supports all DML statements with a WHERE-clause• SELECT, UPDATE, DELETE, etc.
• Array index support only for GSI indexes. • Supports both standard secondary and memory optimized index.
CREATE INDEX isched ON `travel-sample`(DISTINCT ARRAY v.flight FOR v IN schedule END) WHERE type = "route";
![Page 51: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/51.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Index syntax
CREATE INDEX isched ON `travel-sample`
(ALL ARRAY p FOR p IN public_likes END)
WHERE type = "hotel" ;
"Julius Smith", [DocID] "Corrine Hill", [DocID] "Jaeden McKenzie", [DocID] "Vallie Ryan", [DocID] "Brian Kilback", [DocID] "Lilian McLaughlin", [DocID] "Ms. Moses Feeney", [DocID] "Elnora Trantow”, [DocID]
"public_likes": [ "Julius Smith", "Corrine Hill", "Jaeden McKenzie", "Vallie Ryan", "Brian Kilback", "Lilian McLaughlin", "Ms. Moses Feeney", "Elnora Trantow" ]
![Page 52: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/52.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example - Indexing individual attributes/elements
• "Find the total number of flights scheduled on 3rd day" CREATE INDEX isched ON `travel-sample`(DISTINCT ARRAY v.day FOR v IN schedule END) WHERE type = "route” ;
SELECT count(*) FROM `travel-sample` WHERE type = "route" AND ANY v IN schedule SATISFIES v.day = 3 END;
![Page 53: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/53.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example - Indexing individual attributes/elements explain SELECT count(1) FROM `travel-sample` WHERE type = "route" AND ANY v IN schedule SATISFIES v.day = 3 END; { "#operator": "DistinctScan", "scan": { "#operator": "IndexScan", "index": "isched", "index_id": "2b24c681fa54d83f", "keyspace": "travel-sample", "namespace": "default", "spans": [ { "Range": { "High": [ "3" ], "Inclusion": 3, "Low": [ "3" ]
![Page 54: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/54.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example - Index with Array Elements and Other Attributes
• "Find all scheduled flights with hops, and group by number of stops"
CREATE INDEX iflight_stops ON `travel-sample` ( stops, DISTINCT ARRAY v.flight FOR v IN schedule END ) WHERE type = "route" ;
SELECT * FROM `travel-sample`WHERE type = "route" AND ANY v IN schedule SATISFIES v.flight LIKE 'AA%' END AND stops >= 0;
![Page 55: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/55.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example - Indexing Nested Arrays
"schedule" : [ {"day" : 0, "special_flights" : [
{"flight" : "AI111", "utc" : "1:11:11"},
{"flight" : "AI222", "utc" : "2:22:22" } ]},{"day": 1,
"flight": "AF552", "utc": "14:41:00"
} … ]
![Page 56: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/56.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example - Indexing Nested Arrays
• "Find the total number of special flights scheduled"
CREATE INDEX inested ON `travel-sample` (DISTINCT ARRAY(DISTINCT ARRAY y.flight FOR y IN x.special_flights END) FOR x IN schedule END) WHERE type = "route" ;
SELECT count(*) FROM `travel-sample` WHERE type ="route" ANDANY x IN schedule SATISFIES (ANY y IN x.special_flights SATISFIES y.flight IS NOT NULL END) END ;
"schedule”:[ { "day" : 0, "special_flights" : [ { "flight" : "AI111", "utc":"1:11:11"}, { "flight" : "AI222", "utc":"2:22:22"} ] }, { "day": 1, "flight": "AF552", "utc": "14:41:00" }, { "day": 2, "flight": "AF166",
"utc": "08:59:00" }, …]
![Page 57: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/57.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Example – UNNEST
• N1QL array indexing supports both collection predicates • ANY• ANY AND EVERY
• Exploited UNNEST
CREATE INDEX idx_crew ON flight(DISTINCT ARRAY c FOR c IN public_likes END);
SELECT * FROM flight UNNEST crew_ids AS c WHERE c = "Joe Smith" ;
![Page 58: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/58.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Restrictions in 4.5
Variable names and index keys, such as v & v.dayused in CREATE INDEX and subsequent SELECT statements must be same.
CREATE INDEX isched ON `travel-sample`(DISTINCT ARRAY v.day FOR v IN schedule END) WHERE type = "route" ;
SELECT count(*) FROM `travel-sample` WHERE type = "route" AND ANY v IN schedule SATISFIES v.day = 3 END;
![Page 59: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/59.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Restrictions in 4.5
• Supported operators: DISTINCT ARRAY ALL ARRAY ARRAY ANY ANY AND EVERY IN, WITHIN UNNEST
• NOT supported operators: EVERY
![Page 60: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/60.jpg)
©2016 Couchbase Inc. 60
Fun with Arrays
![Page 61: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/61.jpg)
©2016 Couchbase Inc. 61©2016 Couchbase Inc.
SELECT: Fetch Documents
SELECT * FROM customer USE KEYS ["KEY01"] ;
SELECT * FROM customer USE KEYS [ "CUST:09", "CUST:29", "CUST:234", "CUST:852", "CUST:258"] ;
SELECT status, COUNT(status)FROM customer c USE KEYS [ "CUST:09", "CUST:29", "CUST:234", "CUST:852", "CUST:258" ] WHERE c.region = 'US’GROUP BY status;
SELECT product, COUNT(product)FROM customer c USE KEYS [ "CUST:09", "CUST:29", "CUST:234", "CUST:852", "CUST:258" ] INNER JOIN locations ON KEYS c.lid WHERE c.region = 'US’GROUP BY product;
![Page 62: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/62.jpg)
©2016 Couchbase Inc. 62©2016 Couchbase Inc.
SELECT: JOIN
SELECT COUNT(1)FROM `beer-sample` beer INNER JOIN `beer-sample` brewery ON KEYS beer.brewery_idWHERE state = ‘CA’
• JOIN operation combines documents from two key spaces
• JOIN criteria is based on ON KEYS clause
• The outer table uses the index scan, if possible
• The fetch of the inner table (brewery) document-by-document
• Couchbase 4.6 improves this by fetching in batches.
![Page 63: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/63.jpg)
©2016 Couchbase Inc. 63©2016 Couchbase Inc.
SELECT: JOIN
SELECT COUNT(1)FROM ( SELECT RAW META().id FROM `beer-sample` beer WHERE state = ‘CA’) as blistINNER JOIN `beer-sample` brewery ON KEYS blist;
SELECT COUNT(1)FROM ( SELECT ARRAY_AGG(META().id) karray FROM `beer-sample` beer WHERE state = ‘CA’) as bINNER JOIN `beer-sample` brewery ON KEYS b.karray;
• Why not get all of the required document IDs from the index scan then do a big bulk get on the outer table?
• Two ways to do it.a) Use the array aggregate (ARRAY_AGG()) to create the listb) Use RAW to create the the array and then use that to JOIN.
![Page 64: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/64.jpg)
©2016 Couchbase Inc. 64©2016 Couchbase Inc.
Data.gov : New York Names{ "meta": { "view": { "id": "25th-nujf", "name": "Most Popular Baby Names by Sex and Mother's Ethnic Group, New York City", "category": "Health", "createdAt": 1382724894, "description": "The most popular baby names by sex and mother's ethnicity in New York City.", "displayType": "table", … "columns": [{ "id": -1, "name": "sid", "dataTypeName": "meta_data", "fieldName": ":sid", "position": 0, "renderTypeName": "meta_data", "format": {} }, { "id": -1, "name": "id", "dataTypeName": "meta_data", "fieldName": ":id", "position": 0, "renderTypeName": "meta_data", "format": {} } ... ] "data": [ [1, "EB6FAA1B-EE35-4D55-B07B-8E663565CCDF", 1, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GERALDINE", "13", "75"], [2, "2DBBA431-D26F-40A1-9375-AF7C16FF2987", 2, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GIA", "21", "67"], [3, "54318692-0577-4B21-80C8-9CAEFCEDA8BA", 3, 1386853125, "399231", 1386853125, "399231", "{\n}", "2011", "FEMALE", "HISPANIC", "GIANNA", "49", "42"] ... ] }
![Page 65: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/65.jpg)
©2016 Couchbase Inc. 65©2016 Couchbase Inc.
Data.gov : New York NamesINSERT INTO nynames (KEY UUID(), VALUE kname)SELECT {":sid":d[0],
":id":d[1], ":position":d[2],":created_at":d[3], ":created_meta":d[4],":updated_at":d[5],":updated_meta":d[6],":meta":d[7],"brth_yr":d[8],"brth_yr":d[9],"ethcty":d[10],"nm":d[11], "cnt":d[12],"rnk":d[13]} kname
FROM (SELECT d FROM datagov UNNEST data d) as u1;
![Page 66: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/66.jpg)
©2016 Couchbase Inc. 66©2016 Couchbase Inc.
Data.gov : New York NamesINSERT INTO nynames
( KEY UUID(), value o )
SELECT o FROM (
SELECT meta.`view`.columns[*].fieldName f, data FROM datagov) d UNNEST data d1
LET o = OBJECT p:d1[ARRAY_POSITION(d.f, p)] FOR p IN d.f END ;
![Page 67: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/67.jpg)
©2016 Couchbase Inc. 67©2016 Couchbase Inc.
SPLIT & CONQUOR:
SELECT name FROM `travel-sam5ple` WHERE type = 'hotel' LIMIT 5;
[ { "name": "Medway Youth Hostel" }, { "name": "The Balmoral Guesthouse" }, { "name": "The Robins" }, { "name": "Le Clos Fleuri" }, { "name": "Glasgow Grand Central" }]
• Problem: Search for a word within a string
![Page 68: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/68.jpg)
©2016 Couchbase Inc. 68©2016 Couchbase Inc.
SPLIT & CONQUER:
select name from `travel-sample` where type = 'hotel' and lower(name) LIKE '%grand%';
[ { "name": "Glasgow Grand Central" }, { "name": "Horton Grand Hotel" }, { "name": "Manchester Grand Hyatt" }, { "name": "Grande Colonial Hotel" }, { "name": "Grand Hotel Serre Chevalier" }, { "name": "The Sheraton Grand Hotel" }]
• Use the LIKE predicate• Runs in about 81 milliseconds to
search 917 documents
![Page 69: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/69.jpg)
©2016 Couchbase Inc. 69©2016 Couchbase Inc.
SPLIT & CONQUER:
CREATE INDEX idxtravelname ON `travel-sample` (DISTINCT ARRAY wrd FOR wrd IN SPLIT(LOWER(name)) END) where type = 'hotel';
SELECT name FROM `travel-sample` WHERE ANY wrd IN SPLIT(LOWER(name)) satisfies wrd = 'grand' END AND type = 'hotel';[ { "name": "The Sheraton Grand Hotel" }, { "name": "Horton Grand Hotel" }, { "name": "Grand Hotel Serre Chevalier" }, { "name": "Glasgow Grand Central" }, { "name": "Manchester Grand Hyatt" }]~
• Convert into LOWER case• Split the name into words.• SPLIT() returns a ARRAY of these
words.• Create the INDEX on this array.• Query using the Array predicate.• Query runs in 10 ms.• Benefits grow with number of
docs.
![Page 70: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/70.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Bucket: article
{ { "tags": "JSON,N1QL,COUCHBASE,BIGDATA,NAME,data.gov,SQL", "title": "What's in a New York Name? Unlock data.gov Using N1QL " }, { "tags": "TWITTER,NOSQL,SQL,QUERIES,ANALYSIS,HASHTAGS,JSON,COUCHBASE,ANALYTICS,INDEX", "title": "SQL on Twitter: Analysis Made Easy Using N1QL" }, { "tags": "CONCURRENCY,MONGODB,COUCHBASE,INDEX,READ,WRITE,PERFORMANCE,SNAPSHOT,CONSISTENCY", "title": "Concurrency Behavior: MongoDB vs. Couchbase" }, { "tags": "COUCHBASE,N1QL,JOIN,PERFORMANCE,INDEX,DATA MODEL,FLEXIBLE,SCHEMA", "title": "JOIN Faster With Couchbase Index JOINs" }, { "tags": "NOSQL,NOSQL,BENCHMARK,SQL,JSON,COUCHBASE,MONGODB,YCSB,PERFORMANCE,QUERY,INDEX", "title": "How Couchbase Won YCSB" }}
![Page 71: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/71.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Questions:Find all the articles with N1QL in their titleFind all the articles with COUCHBASE in their tags
{ { "tags": "JSON,N1QL,COUCHBASE,BIGDATA,NAME,data.gov,SQL", "title": "What's in a New York Name? Unlock data.gov Using N1QL " }, { "tags": "TWITTER,NOSQL,SQL,QUERIES,ANALYSIS,HASHTAGS,JSON,COUCHBASE,ANALYTICS,INDEX", "title": "SQL on Twitter: Analysis Made Easy Using N1QL" }, { "tags": "CONCURRENCY,MONGODB,COUCHBASE,INDEX,READ,WRITE,PERFORMANCE,SNAPSHOT,CONSISTENCY", "title": "Concurrency Behavior: MongoDB vs. Couchbase" }, { "tags": "COUCHBASE,N1QL,JOIN,PERFORMANCE,INDEX,DATA MODEL,FLEXIBLE,SCHEMA", "title": "JOIN Faster With Couchbase Index JOINs" }, { "tags": "NOSQL,NOSQL,BENCHMARK,SQL,JSON,COUCHBASE,MONGODB,YCSB,PERFORMANCE,QUERY,INDEX", "title": "How Couchbase Won YCSB" }}
![Page 72: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/72.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Basic Framework
"tags": "JSON,N1QL,COUCHBASE,BIGDATA,NAME,data.gov,SQL"
["JSON", "N1QL", "COUCHBASE", "BIGDATA", "NAME", "data.gov", "SQL" ]
SPLIT() into an array Array
Index
Distinct array wrd for wrd in split(tags,”,”) end
Index this array N1QL Query Service
SELECT *FROM articlesWHERE ANY wrd IN SPLIT(tags, ",") satisfies wrd = "COUCHBASE” END
![Page 73: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/73.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Basic Framework
"title": "What's in a New York Name? Unlock data.gov Using N1QL "
["What's", "in", "a", "New", "York", "Name?", "Unlock", "data.gov", "Using", "N1QL" ]
SPLIT() into an array Array
Index
??? N1QL Query Service
???
![Page 74: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/74.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
New Function: TOKENS() in Couchbase 4.6 – OUT in DP now. TOKENS(expression [, parameter])
expression : JSON expression parameter : options {"names":true} Include the key names in the JSON “key”:value pair. {"case":"lower"} Return the values in upper/lower case
{"specials":true} Recognize special characters like @, - to form tokens.
"tags": "JSON,N1QL,COUCHBASE,BIGDATA,NAME,data.gov,SQL", "tagsarray": [ "data", "gov", "bigdata", "n1ql", "couchbase", "sql", "json", "name" ],
select title, tags, tokens(tags, {"case":"lower"}) tagsarray, tokens(title) titlearray from articles limit 1;
"title": "What's in a New York Name? Unlock data.gov Using N1QL ", "titlearray": [ "s", "Unlock", "data", "N1QL", "gov", "in", "Using", "New", "What", "York", "a", "Name" ]
![Page 75: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/75.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using TOKENS() – Index on title, lower case create index ititlesearch on articles(distinct array wrd for wrd in tokens(title, {"case":"lower"}) end);explain select title from articles where any wrd in tokens(title, {"case":"lower"}) satisfies wrd = 'n1ql' end; { "#operator": "DistinctScan", "scan": { "#operator": "IndexScan", "index": "ititlesearch", "index_id": "7a162af1199565b5", "keyspace": "articles", "namespace": "default", "spans": [ { "Range": { "High": [ "\"n1ql\"" ], "Inclusion": 3, "Low": [ "\"n1ql\"" ] } } ], "using": "gsi" } },
![Page 76: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/76.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Using TOKENS() Index on the WHOLE document create index ititlesearch2 on articles (distinct array wrd for wrd in tokens(articles, { "case":"lower" , "names":true }) end);
explain select title from articles where any wrd in tokens(articles, {"case":"lower", "names":true }) satisfies wrd = ’title' end; "#operator": "DistinctScan", "scan": { "#operator": "IndexScan", "index": "ititlesearch2", "index_id": "c60792ca9f957cfd", "keyspace": "articles", "namespace": "default", "spans": [ { "Range": { "High": [ "\"title\"" ], "Inclusion": 3, "Low": [ "\“title\"" ] } } ], "using": "gsi" } },
![Page 78: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/78.jpg)
©2016 Couchbase Inc. 78
Thank You!
![Page 79: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/79.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Goal of N1QL
Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.
![Page 80: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/80.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing – How array is expanded in GSI
Sl. Create Index Expression Key versions generated by Projector
Index Entries in GSI storage
1. age [K1] [K1]docid
2. age, name, children [K1, K2, [c1, c2, c3]] [K1, K2, [c1, c2, c3]]docid
3. ALL ARRAY c FOR c IN cities END [[K11, K12, K13]] [ K11]docid[ K12]docid[ K13]docid
4. ALL ARRAY c FOR c IN cities END, age [[K11, K12, K13], K2] [ K11, K2]docid[ K12, K2]docid[ K13, K2]docid
4.1 age, ALL ARRAY c FOR c IN cities END, name [K1, [ K21, K22, K23,], K3] [K1, K21, K3]docid[K1, K22, K3]docid[K1, K23, K3]docid
![Page 81: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/81.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing – How array is expanded in GSI
Sl. Create Index Expression Key versions generated by Projector
Index Entries in GSI storage
5. ALL ARRAY c FOR c IN cities END, children
[[K11, K12, K13], [c1, c2, c3]] [ K11, [c1, c2, c3]]docid[ K12, [c1, c2, c3]]docid[ K13, [c1, c2, c3]]docid
6. ALL ARRAY (ALL ARRAY y FOR y IN c END) FOR c IN cities END
[ [K1, K2, K3, K4, K5]]
[K1]docid[K2]docid[K3]docid[K4]docid[K5]docid
![Page 82: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/82.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing Performance in ForestDB (3.6K sets)
Metrics KPI Measured comments
Array Q2(stale=ok) 13000 15140 Single doc match & fetch
Array Q2(stale=false) 700 9420 Same with consistency
Array Q3(stale=ok) 1100 1435 100 doc match and fetch.
Array Q3(stale=false) 428 1084 Same with consistency
![Page 83: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/83.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing Performance in MOI with 30K sets
Metrics KPI Measured comments
Array Q2(stale=ok) 13000 15251 Single doc match & fetch
Array Q2(stale=false) 700 7545 Same with consistency
Array Q3(stale=ok) 1100 1371 100 doc match and fetch.
Array Q3(stale=false) 428 1580 Same with consistency
![Page 84: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/84.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
UNITED – POC on 4.0
Response times in milliseconds1 Thread 10 Thread 50 Thread
Q1 13 35.1 197.84Q2 28 66.8 285.32Q3 - 7d 160 606 2960.2Q3 - 28d 1725 8240.3 41439.86
1 Thread 5 Threads Q1 1500 31000Q2 Timed out.Q3 23000 90000
MongoDB Query
Couchbase Query
Response times in milliseconds
![Page 85: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/85.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
UNITED -- POC• Query 2 – Get the selected flight using the document key. For
each crew member (pilot and flight attendant) found in the flight details.
• Fetch the previous flight assigned to the crew member• Fetch the next flight assigned to the crew member
select ods.GMT_EST_DEP_DTM,ods.PRFL_ACT_GMT_DEP_DTM,ods.PRFL_SCHED_GMT_DEP_DTM,ods.GMT_EST_ARR_DTM,ods.PRFL_ACT_GMT_ARR_DTM,ods.PRFL_SCHED_GMT_ARR_DTM,ods.FLT_LCL_ORIG_DT,ods.PRFL_FLT_NBR,ods.PRFL_TAIL_NBR,PILOT.PRPS_RSV_IND from ods unnest ods.PILOT where ods.TYPE='CREW_ON_FLIGHT' and ((ods.PRFL_ACT_GMT_DEP_DTM is not missing and ods.PRFL_ACT_GMT_DEP_DTM > "2015-07-15T02:45:00Z") OR (ods.PRFL_ACT_GMT_DEP_DTM is missing and ods.GMT_EST_DEP_DTM is not null and ods.GMT_EST_DEP_DTM > "2015-07-15T02:45:00Z")) and any p in ods.PILOT satisfies p.FILEN = "U110679" end order by ods.GMT_EST_DEP_DTM limit 1
![Page 86: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/86.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
UNITED – POC Queries on 4.5• 422,137 documents.• Query2: BEFORE array indexing • Primary index scan• 38.91 seconds.
create index idx_odspilot on ods(DISTINCT ARRAY p.FILEN in p in PILOT END);
• Query2: AFTER array indexing • Array index scan [DistinctScan]• 8.51 millisecond
•Improvement of 4572 TIMES
![Page 87: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/87.jpg)
©2016 Couchbase Inc.©2016 Couchbase Inc.
Array Indexing – Size and numbers• There is no limit on number of elements in the array.
• Total size of array index key should not exceed setting max_array_seckey_size (Default = 10K)
CREATE INDEX i1 on default(ALL flights, airlineid) . Lets say a given document is:
{ "flights": ["AF552", "AF166", "AF268", "AF422"], "airlineid": "airline_137"}
The indexable array keys for the document are: [ ["AF552", "airline_137"], ["AF166", "airline_137"], ["AF268", "airline_137"], ["AF422", "airline_137"] ]
Sum of lengths above items should be < max_array_seckey_size. Setting can be increased but not decreased.
![Page 88: Utilizing arrays: modeling, indexing, and querying](https://reader035.fdocuments.us/reader035/viewer/2022081604/5874f5d51a28ab917a8b7e31/html5/thumbnails/88.jpg)
©2016 Couchbase Inc. 88©2016 Couchbase Inc.
Statements : MERGE
BIG MERGE statement – Use travel-sampleexplain merge into b1 using b2 on key "11" when matched then update set b1.o3=1;
merge into b1 using (select id from b2 where x < 10) as b3 on key b3.id when matched then update set b1.o4=1;
merge into `travel-sample` using default on key "2" when matched then update set `travel-sample`.name="aaa";MERGE into WAREHOUSE using `beer-sample` ON KEY to_string("yakima_brewing_and_malting_grant_s_ales-deep_powder_winter_ale²) when matched then delete;