MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms...
-
Upload
mongodb -
Category
Data & Analytics
-
view
135 -
download
2
Transcript of MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms...
![Page 1: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/1.jpg)
ENABLING IOT ATPROXIMUS
![Page 3: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/3.jpg)
IN THIS SESSIONHow to implement mongoDB in a IoT M2M use case.Data modelling is key when storing time series data.How to insert data into mongoDB using Rest interface.
![Page 4: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/4.jpg)
INITIAL QUESTIONSHow to store data while keeping storage amount minimal?What storage engine to choose?How to store my data using a third party system?How can I easily deploy my mongoDB solution?
![Page 5: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/5.jpg)
ESSENTIAL PROBLEMSHow do we model our data?De�ning storage engineSetting up your mongo InstancesHow do we insert our data in mongoDB?How do we query our data?
![Page 6: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/6.jpg)
SETTING THE SCENEMythings Project at ProximusWhat is IoT?LoRa Technology
![Page 7: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/7.jpg)
2 Front-end 3 Back-end
MYTHINGS DEVELOPMENT TEAM
Small team:
Mongo experience: 1,5 year
![Page 8: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/8.jpg)
MYTHINGS AT PROXIMUS
![Page 9: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/9.jpg)
MYTHINGS AT PROXIMUS
![Page 10: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/10.jpg)
INTERNET OF THINGSVery broad, not contained to a single technology.
Not what but HOW !
Internet is the keyword
Collecting, analyzing and above all communicating data.
![Page 11: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/11.jpg)
LORA TECHNOLOGY
![Page 12: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/12.jpg)
LORA TECHNOLOGY VS ...
![Page 13: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/13.jpg)
LORA ARCHITECTURE
![Page 14: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/14.jpg)
SENSORS
Any type is possible as long as it is connected with a LoRa chip CO2, PIR, Humidity, Luminance, Temperature, Pressure, …
![Page 15: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/15.jpg)
SENSORS
multiple containers per sensor possible
![Page 16: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/16.jpg)
LORA ALLIANCE
https://www.lora-alliance.org
![Page 17: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/17.jpg)
ESSENTIAL PROBLEMSHow do we model our data?De�ning storage engineSetting up your mongo InstancesHow do we insert our data in mongoDB?How do we query our data?
![Page 18: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/18.jpg)
DATA MODELINGSENSOR HISTORY
company infosensor infocontainer infonoti�cation valuenoti�cation timestamp
![Page 19: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/19.jpg)
SENSORHISTORY COLLECTION{ "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "data": { "type": "bool", "ts": 1444139821000, "value": "1" } }
![Page 20: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/20.jpg)
Daily document size/sensor =
SENSORHISTORY COLLECTION SIZING
Document size = 496 BytesSensor will send 48 time a dayOn average 3 containers/sensor
496 B * 48 documents * 3 = 71,4 KB
71,4 KB
1 M sensors => 68.09 GB/day
![Page 21: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/21.jpg)
Document with 3 additions:
BUCKETING TECHNIQUE
Data array (data values)Count metadata (Array size)Date �eld (day timestamp)
Use in combination with Upsert mechanism!
![Page 22: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/22.jpg)
SENSORHISTORY COLLECTION (BUCKETING){ "_id": ObjectID("5613d33401e101c27f8e668d"), "company": { "name": "Proximus", "companyId": 2 }, "container": { "container": "0x0050.0x0006.0.m2m", "type": "temperature", "containerId": 597, "name": "temperature" }, "date": 1444086000000, "sensor": { "sensorId": 773, "mac": "F03D291000001301" }, "count": 48, "data": [ { "type": "bool", "ts": 1444139821000, "value": "1" }, { "type": "bool", "ts": 1444152754000, "value": "1" },... ]}
![Page 23: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/23.jpg)
daily document size/sensor =
SENSORHISTORY COLLECTION SIZING(BUCKETING)
Document size = 3.98 KBSensor will send 48 time a dayOn average 3 containers/sensor
3.98 KB * 1 document * 3 = 11,9 KB
11,9 KB
1 M sensors => 11.34 GB/day
![Page 24: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/24.jpg)
Use if possible (depending on use-case)
vs
DATA MODELINGBUCKETING
11GB/day 68GB/day
![Page 25: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/25.jpg)
Concurrent updates on the bucket document may lead to queue
operations Do not overdue it!
DATA MODELINGBUCKETING
Advantages:Collection size will be smallerIndexes size will be considerable reducedReads will be faster
Remarks:
! Maximum document size 16MB !
![Page 26: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/26.jpg)
DEFINING STORAGE ENGINEAPPLICATION REQUIREMENTS:
Write intensiveKeep storage volume to a minimumRead and write simultaniously to collection
![Page 27: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/27.jpg)
DEFINING STORAGE ENGINEWIREDTIGER 3.2
Document level lock instead of collection levelData is stored in a compressed formatIndex size is smaller
Matched the application requirements
WiredTiger was recommended by MongoDB
![Page 28: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/28.jpg)
SETTING UP MONGODB INSTANCESCLOUD MANAGER
![Page 29: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/29.jpg)
Once automation agent is installed, smooth sailing!
SETTING UP YOUR MONGODBINSTANCES
CLOUD MANAGER
Easy setupEasy managementMetrics
![Page 30: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/30.jpg)
INSERTING DATAMYTHINGS ARCHITECTURE
![Page 31: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/31.jpg)
INSERTING DATAREST INTERFACE
Sleepy Mongoose (Python)https://github.com/mongodb-labs/sleepy.mongoose Rest Heart (Java)latest feature support http://restheart.org/ Advised to change to Rest Heart by MongoDB
![Page 32: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/32.jpg)
Latest feature support
Uses PATCH method for updates (Upsert) => Not supported current release
Not mature enough: no bucketing for upsert!
INSERTING DATAREST HEART
![Page 33: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/33.jpg)
Back to square one!
INSERTING DATASLEEPY MONGOOSE
![Page 34: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/34.jpg)
Interchange Rest heart <> Sleepy Mongoose
Easy transition (1-2 days development)!
INSERTING DATASLEEPY MONGOOSE
![Page 35: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/35.jpg)
Simple insert:
Advanced upsert:
INSERTING DATA
$ curl --data 'docs=[{"x":2},{"x":3}]' 'http://localhost:27080/foo/bar/_insert'
$ curl --data 'criteria= { "sensor":{"sensorId":1533,"name":"Binary mesurement","mac":"020000FFFF00B151"}, "company":{"companyId":134,"name":"Opinum"}, "container":{"containerId":1223,"name":"Door counter"}, "date":1456354800000 }, "count":{"$lt":100} &newobj={ "$push":{"data":{"ts":1456403435000,"value":"456","type":"int"}}, "$inc":{"count":1}, "$setOnInsert":{"location":{"latitude":"-1","longitude":"-1"}}} &upsert=true' 'http://localhost:27080/foo/bar/_insert'
![Page 36: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/36.jpg)
QUERING DATAQuery frameworkAggregation framework
![Page 37: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/37.jpg)
Retrieve sensorhistory for a week for all companies:
AGGREGATION FRAMEWORK
"op" : "command", "ns" : "sensoringlabs.sensorHistory", "command" : { "aggregate" : "sensorHistory", "pipeline" : [{ "$match" : { "company.companyId" : { $in" : [,54,61,76,77,79,80,81,82,83,86,87,90,122,123 ,124,125,126,127,128,129,130,131,132,133,134,135,136, 137,138,146,147,148,149,150,151,152,153,154] } } }, { "$match" : { "date" : {"$gte" : NumberLong("1457049600000")}, "$and" : [{"date" : {"$lte" : NumberLong("1457568000000")}}] } }, {"$skip" : NumberLong(0)}, {"$limit" : NumberLong(100)} ], "cursor" : {} } "responseLength" : 278888, "protocol" : "op_command", "millis" : 366
![Page 38: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/38.jpg)
Retrieve sensorhistory for a week for all companies:
QUERY FRAMEWORK
"op" : "query", "ns" : "sensoringlabs.sensorHistory", "query" : { "find" : "sensorHistory", "filter" : { "company.companyId" : { "$in" : [2,54,61,76,77,79,80,81,82,83,86,87,90,122,123,124, 125,126,127,128,129,130,131,132,133,134,135,136,137, 138,146,147,148,149,150,151,152,153,154] }, "date" : { "$gte" : NumberLong("1457049600000"), "$lte" : NumberLong("1457568000000") } }, "limit" : 100, "singleBatch" : false } "nreturned" : 100, "responseLength" : 278888, "protocol" : "op_command", "millis" : 1
![Page 39: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/39.jpg)
Same search:
Query framework :
Aggregation framework :
QUERING DATA
1ms
360 ms
![Page 40: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/40.jpg)
-> fast for retrieving data
-> data aggregations that take more time
Read concern: read from secondary
QUERING DATAQuery framework
Aggregation framework
![Page 41: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/41.jpg)
Query example:
QUERING DATASLEEPY MONGOOSE
$ curl -X GET 'http://localhost:27080/foo/bar/_find ?criteria={"_id":{"$oid":"4f8c6f05db61e2a72600001d"}} &sort={"date”:-1} &limit=1'
![Page 42: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/42.jpg)
INDEX STRATEGYCOMPOUND INDEXES
Multitenant system: company idTime based data: date
![Page 43: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/43.jpg)
Data retention policy:
-> Deterministic way to drop documents once expired
-> Fixed size collections
CC Not suitable because time range of documents will depend onthroughput!
(high throughput will decrease the time range stored).
CAPPED COLLECTION VS TTL INDEXESGETTING RID OF THE EXPIRED DATA
3 months
TTL index
Capped collection
![Page 44: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/44.jpg)
PUTTING IT ALLTOGETHER
![Page 45: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/45.jpg)
Bucketing
DATA MODELING
![Page 46: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/46.jpg)
Wired tiger 3.2
STORAGE ENGINE
![Page 47: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/47.jpg)
Sleepy Mongoose RestHeart (future) Java mongo Driver
DATA INSERTION
![Page 48: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/48.jpg)
query framework aggregation framework
DATA EXTRACTION
![Page 49: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/49.jpg)
compound indexes TTL index
INDEXES
![Page 50: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/50.jpg)
SHARDINGWHEN
There is more data than one machine can hold on its drivesThe application is write heavy and you could experience too muchlatencyThe working set outgrows the memory can be allocated to a singlemachine
![Page 51: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/51.jpg)
CONCLUSIONWHAT WE LEARNED
Take enough time to model new collections (bucketing)=> Can decrease the storage volume when done right
The use of cloud manager drastically decreases deployment andmaintenance time=> More time to think about your application
There are endless ways of getting your data into mongoDB=> Mongo supplies and support many di�erent solutions
![Page 52: MongoDB Europe 2016 - Enabling the Internet of Things at Proximus - Belgium's Largest Telecoms Company](https://reader034.fdocuments.us/reader034/viewer/2022052705/587064ca1a28ab48378b4b8d/html5/thumbnails/52.jpg)
THANKS FOR LISTENING!Questions?