Sensing the world with data of things
-
Upload
wso2-inc -
Category
Technology
-
view
640 -
download
1
Transcript of Sensing the world with data of things
![Page 1: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/1.jpg)
Sensing the world with Data of Things By:Sriskandarajah Suhothayan (Suho)
Technical Lead at WSO2@[email protected]
STRUCTURE DATA 2016MARCH 9 - 10 • SAN FRANCISCO
![Page 2: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/2.jpg)
Any customer can have a car painted any colour that he wants so long as it is black
~ Henry Ford ~
![Page 3: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/3.jpg)
Me Me Me !!!
Your customers want to have a personalized experience. We are in the time of ME!
![Page 4: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/4.jpg)
![Page 5: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/5.jpg)
![Page 6: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/6.jpg)
What to do ?
![Page 7: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/7.jpg)
Is IoT New ?
• source: http://community.arm.com/groups/internet-of-things/blog/2014/06
![Page 8: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/8.jpg)
Internet of Things
http://na1.www.gartner.com/imagesrv/newsroom/images/HC_ET_2014.jpg;wadf79d1c8397a49a2
source : http://na1.www.gartner.com/imagesrv/newsroom/images/HC_ET_2014.jpg;wadf79d1c8397a49a2
![Page 9: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/9.jpg)
IoT Ecosystem
![Page 10: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/10.jpg)
WSO2 IoT Server M3 : https://goo.gl/nhbxnG
http://wso2.com/iot
![Page 11: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/11.jpg)
Concepts of IoT Analytics
● Type of Data● Distributed Nature● Event-Drivenness ● Possible Type of Analytics● Scalability ● Edge Analytics● Uncertainty
![Page 12: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/12.jpg)
Data Types of Things
● Time based data○ Continuous monitoring & reporting ○ Time series processing (e.g. Energy
consumption over time)○ Specialised DBs - OpenTSDB
● Location based data○ Things are allover the place & they move○ Tracked via GPS / iBeacons○ Geospatial processing (e.g Traffic planning,
better route suggestion for vehicles) ○ Geospatial optimised processing engines -
GeoTrellis
![Page 13: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/13.jpg)
IoT is Distributed
● Constant changes ○ When components added and removed○ Data flows are modified or repurposed
● Data collection need to support ○ Weak 3G networks to Ad-hoc peer-to-peer networks. ○ Message Queuing Telemetry Transport (MQTT) ○ Common Open Source Publishing Platform (CoApp)○ ZigBee or Bluetooth low energy (BLE)
● Dynamic scaling ○ Hybrid cloud
![Page 14: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/14.jpg)
IoT Analytics are Event-Driven
● Sensors report data as Event Streams ● Analysis on flowing (or perishable) data
● Realtime Analytics○ Detect temporal and logical patterns○ Identify KPIs and Thresholds○ Send out alerts immediately ○ E.g. Alert when temperature sensor hit a limit, notify in
car dashboard of low tire pressure ○ Systems : Apache Storm, Google Cloud DataFlow &
WSO2 CEP
![Page 15: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/15.jpg)
History Repeats
● Present vs usual behavior ● Understand the history
● Batch Analytics○ Perform periodic summarisation/analytics ○ E.g. Average temperature in a room last month, total
power usage of the factory last year ○ Systems : Apache Hadoop, Apache Spark + Storage
![Page 16: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/16.jpg)
● Ad-Hoc Queries
● Interactive Analytics○ Provides searchability ○ E.g. Identify fraud rings from simple fraud alerts ○ Systems : Apache Drill, indexed storage systems such
as Couchbase, Apache Lucene
Deep Investigations
![Page 17: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/17.jpg)
Thinking Ahead
● When you don’t Know the equations ● Focusing conditions & preventing issues
● Predictive Analytics○ Incremental Learning ○ E.g. Proactive maintenance, fraud detection and health
warnings○ Systems : Apache Mahout, Apache Spark MLlib,
Microsoft Azure Machine Learning, WSO2 ML, Skytree
![Page 18: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/18.jpg)
Technology we’ve chosen
Realtime Batch
Interactive Predictive
![Page 19: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/19.jpg)
WSO2 Data Analytics Server
![Page 20: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/20.jpg)
Plenty of Data
Scalable Data Processing
source : http://www.websitemagazine.com/content/blogs/posts/archive/2014/09/25/customer-service-in-2039.aspx
![Page 21: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/21.jpg)
Scalable Realtime Deployment
More info : https://docs.wso2.com/display/CEP410/Creating+a+Storm+Based+Distributed+Execution+Plan
![Page 22: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/22.jpg)
Scalable Deployment
Interactive
Batch Realtime & Predictive
![Page 23: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/23.jpg)
● Publishing all events is not good!○ Hardware may not be scalable ○ Network getting flooded
● What we usually need ○ Aggregation over time ○ Trends that exceed thresholds○ Event matching a rare condition
● Results in○ Local optimisation○ Quick detection of issues○ Instant notification
Is Every Event Significant?
![Page 24: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/24.jpg)
Edge Analytics
Analytics on the Edge with WSO2 Siddhi
Push
![Page 25: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/25.jpg)
Outliers ...
● E.g. Anomaly detection, Fraud Analytics
● Alerts for known and unknown frauds and Deep Search Analyticshttps://goo.gl/TWV5C1
![Page 26: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/26.jpg)
Outliers
● We used: Linear Regression, Markov Models & Credit Scoring
![Page 27: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/27.jpg)
Uncertainty in Data of Things
Data can be ● Duplicated● Arrives out of order● Not arrive at all● Wrong readings
![Page 28: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/28.jpg)
Events Duplicates & Out of Order …
● Due redundant sensors & network latency ● Difficult for temporal data processing
○ Time Windows ○ Temporal ordering
● Such as Fraud detection
define stream Purchase (price double, cardNo long,place string);
from every (a1 = Purchase[price < 10] ) -> a2 = Purchase[ price >10000 and a1.cardNo == a2.cardNo ]
within 1 dayselect a1.cardNo as cardNo, a2.price as price, a2.place as placeinsert into PotentialFraud ;
![Page 29: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/29.jpg)
Events Arriving Out of Order
E.g. Realtime Soccer Analytics (DEBS 2013) https://goo.gl/c2gPrQ
● Identify ball kicks, ball possession, shot on goal & offside
● Solutions : K-Slack Based Algorithmshttps://www2.informatik.uni-erlangen.de/publication/download/IPDPS2013.pdf
![Page 30: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/30.jpg)
Missing Data
● Due to network outages
● E.g. Smart Meters (DEBS 2014)○ Smart home electricity data: 2000 sensors,
40 houses, 4 Billion events in four months○ Processed 400K events/sec
● Solutions: ○ Approximate using complimenting
sensor reading ■ Electricity Monitoring
● Frequent Load readings● Occasional Work readings
○ Fault-tolerant data streams : Google Millwheel
![Page 31: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/31.jpg)
Wrong Sensor Readings
● From GPS ● E.g.TFL Traffic Analysis
○ Using Transport for London open data feeds.
○ http://goo.gl/04tX6k, http://goo.gl/9xNiCm
○ Scales to 500,000 Events/Sec and more
● From iBcons at shops, ships and airport
● Solution: Kalman Filter
![Page 32: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/32.jpg)
Visualisation
● Per-device & Summarization View● Ability to group by categories
● Solutions: Composable Dashboard with sampling & indexing
![Page 33: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/33.jpg)
Communicate to Mobile & 3rd Party Apps
● Expose analytics Results as API○ Mobile Apps,
Third Party
● Provides ○ Security, Billing, ○ Throttling, Quotas
& SLA
● Solution ○ Write data to database ○ Expose them via secured APIs (E.g. WSO2 API Manager)
![Page 34: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/34.jpg)
Reference Architecture for IoT Analytics
![Page 35: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/35.jpg)
IoT Analytics
●○
●○
●○
http://wso2.com/analytics
http://wso2.com/iot
![Page 36: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/36.jpg)
Thank You
Any Questions ?
![Page 37: Sensing the world with data of things](https://reader034.fdocuments.us/reader034/viewer/2022042907/587f04271a28abc26f8b482d/html5/thumbnails/37.jpg)
Contact us !