Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 1
!Damian Black, CEO SQLstream Strata + Hadoop World, 2014!October 15-17 2014, New York
!SmartCity StreamApp: An Internet of Things Service for Real-‐;me Traffic Management
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 2
!Product: SQLstream Blaze Stream Processor Focus: Powering smart services for the IoT/IoE Located: San Francisco
!Real-time Action in the Internet of Everything
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 3
“We predict that the Internet of Everything will be a $19 trillion dollar market over the next several years.”
John Chambers, CEO, Cisco
“When you look at the Internet of Things, it is clear 2014 will be a tipping point in the evolution of the Internet.”
Marissa Mayer, CEO and President, Yahoo
“We are equipping our products with sensors that constantly measure performance so our customers see major productivity gains and minimize unplanned downtime.”
Jeff Immelt, Chairman and CEO, GE
A world transformed by an explosion of sensors
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 4
A World of Big Data § Big Data – datasets (3V) beyond the ability of traditional data management systems § Driven by Hadoop: massively scalable, multi-server systems on low cost hardware § Designed for semi-structured and unstructured data – ideal for sensors § Driving expectation for real-time responses and action
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 4
Internet Wireless Sensors Smartphones
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 5
0!
45!
90!
135!
180!
225!
2014 2015 2016 2017 2018 2019 2020
Analytics will be dominated by IoT Data
IoE (net IoT) 9.5% CAGR [IDC]
IoT (real-time) 44% CAGR [Gartner] $
Billio
ns
Characteris;cs of Internet of Things sensor data > Very high data volumes > Systems must react to each and every record > Mostly ‘business as usual’ – no need to warehouse every record
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 6
Stream Processing!
Relational Databases!
Hadoop !
In-memory Databases!
Big Data Technology: From Human Time to Real-time
6
High Low
Speed of Response
Throughput Capacity
Low
High
Store First
Process First
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 7
Stream Processing versus Stored Data Queries
7
Databases store then query. Stream Processors query then store.
Query
Query
Databases The query traverses the table. With every record, the table is updated and the process must start again.!
Stream Processing The incoming streams of data) move through a continuous, windowed SQL query. Unlimited capacity and continuous, real-time results.!
Storage(Optional — RDBMS, Hadoop, NoSQL …)
Performance Metrics
§ 1 million EPS/core § Multi-server scale-out § Latency < 10 milliseconds
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 8
A real-time data hub for the Internet of Things
IoT Smart Srvcs
Smart Srvcs
Smart Srvcs
Smart Srvcs
Smart Srvcs
Mobile Apps Apps Apps Apps
Server Server Server Server
Internet
Switch Proxy Router
Billing Fraud SLA
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 9
SmartCity StreamApp"Roads & Maritime Services Case Study
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 10
We live in a Modern World § 800 million vehicles on the world’s roads today, estimated to increase to 4 billion by 2050 § Environmental concerns and air pollution driving innovation, but just the beginning § The expectation of on-demand, real-time and accurate information has not been realized § The Quality of Life for travelers and commuters is deteriorating
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 10
Conges4on Environment Poor Informa4on Quality of Life?
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 11
The Internet of Things Smart People to Smart Cities with Smart Services
Traffic Analy;cs Mul;-‐Modal Smart City Real-‐;me Informa;on
§ Step 1: Real-time traffic flow and congestion for transportation agencies § Step 2: Extend for multi-modal Travel Time with public transportation § Step 3: Smart City Smart Services – dyanamic traffic management § Step 4: Traveler and Commuter Smartphone App access
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 11
Traffic Analy4cs Mul4-‐Modal Smart City Real-‐4me Informa4on
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 12
From Big Data to real-time actions § Stream processing means high throughput with low latency actions at Big Data scale § A real-time data hub delivers multiple applications on a single stream processing platform § The platform for the real-time SmartCity StreamApp § Delivering a better traveler experience
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 12
Stream Processing Real-‐4me Data Hub Smart City PlaIorm A BeKer Experience
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 13
Case Study: Big Data Innovation for ITS
TT5 : Real-‐4me SmartCity StreamApp Traffic App Hackathon (Powered by SQLstream, Sept 20-‐21 2014)
• GPS fleet data – low solu;on cost, faster ;me to value
• Real-‐;me traffic flow & conges;on predic;on
• All classified roads in NSW networks
• Opera;onal dashboards based on Google Maps
• Public API (web service feed and JDBC app connect)
• Encourage innova;on and app development
• Focus on conges;on hotspots
• Funding for the winning team
• Building on SQLstream TT5 API
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 14
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 15
Calculate Road Segment (10m) Mean Travel Time
Vehicle 1 Vehicle 2 Vehicle 3 Vehicle 4
Vehicle N
Aggregate Mean Travel Time per Road Element
Vehicle GPS record Vehicle Travel Time (green light) Vehicle Travel Time (red light)
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 16
Travel Time Calculation using Vehicle GPS Data Streams
Calculate Vehicle Travel Time along specified route bounded by REi and REj: • RE: Road Element (10 meter segments)
• l: Length of RE (typically 10 meters)
• D: Distance traveled by vehicle between GPS reports
• T: Time taken by vehicle to travel between GPS reports
• n: Number of GPS reports
TTREi−REj =1nRE
Tv,rlREsDv,rr(reports)
∑#
$%%
&
'((
v(vehicles)∑
#
$%%
&
'((
REs∑
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 17
Update average speed incrementally across any subset of network over rolling time windows
Code Example #2: Continuous Average Speed Calculation
CREATE OR REPLACE VIEW "SpeedZoneStats"DESCRIPTION ‘Rolling averages for multiple windows partitioned by zone' AS SELECT STREAM "zone", -- zone id "segmentid", -- parent road segment "speedlimit", -- speed limit for zone AVG("Speed")OVER last1Min AS "avgSpeed1", -- 1-min running average AVG("Speed")OVER last5Min AS "avgSpeed5", -- 5-min running average AVG("Speed")OVER last10Min AS "avgSpeed10" -- 10-mn running average FROM "RoadPositionInfo" WINDOW last1Min AS (PARTITION BY "zone" RANGE INTERVAL '1' MINUTE PRECEDING), last5Min AS (PARTITION BY "zone" RANGE INTERVAL '5' MINUTE PRECEDING), last10Min AS (PARTITION BY "zone" RANGE INTERVAL '10' MINUTE PRECEDING);
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 18
Improve System Accuracy"Join streaming analytics with historical trends from the data warehouse
0:00
1:00
2:00
3:00
4:00
5:00
6:00
7:00
8:00
9:00
10:00
11:00
12:00
13:00
14:00
15:00
16:00
17:00
18:00
19:00
20:00
21:00
22:00
23:00
0:00
Average (Trend) Real-‐;me > Trend data comparisons help
to differentiate exceptional
event from normal behavior
> Predicting extent and
duration of an event can still
be difficult
> Assessing the traveler
sentiment from Social Media
analytics offers additional
insight
Average Travel Time over a 24 hour period
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 19
SmartCity StreamApp"Architecture
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 20
SQLstream Blaze Distributed SQL stream processing engine with
intelligent guide discovery and real-‐;me dashboards.
s-Visualizer real-time dashboards for Enterprise Power Users
s-Server Distributed SQL Stream
Processor
s-Dashboard HTML5 real-time
dashboards for Developers
Storm Adapter s-Studio Developer & Admin
StreamLab Intelligent guided data stream discovery, analytics and visualization without coding
Enterprise-Class Real-time Data HubStream Processing for Operational Intelligence and the Internet of Things
Industry StreamApps
Pre-‐built libraries for rapid development of industry
stream processing solu;ons.
Telecoms Smart City Emergency Services Oil & Gas
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 21
Control Systems
SQL Optimizer Parallel Scheduler Real-time Indexing
RT Memory Manager Dynamic Java Analytics (UDX)
Streaming Data Protocol
Interac4ve Stream Discovery and Visualiza4on
Stream Processing Engine
(HTML5)!Discovery
API Connect
SmartCity StreamApp – Core Platform Architecture
Devices & Apps
Hadoop Ingestion (MB/s)
Events/Sec/Core)
Native Tables
Repor4ng Tools
JDBC Web Agent
Web Sockets REST
(HTML5)!Dashboards
(Flash)!Dashboards
JDBC
Remote Systems
Enterprise Systems
Agents
Adapters
Devices & Apps
JDBC
Machine Data
Enterprise Systems
Agents
Adapters
Hadoop & NoSQL Enterprise BI
Data Warehouse
SQL Database
Predictive Analytics
Hadoop / HDFS
HBase
Storm & Kafka
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 22
SmartCity StreamApp – Software Architecture
Data Stream Acquisi4on Toolbox
GIS & Geospatial Library
Data Quality & Density Analysis
Data Parsing & Transformation
Data Enrichment
StreamApp Analy4cs Toolbox
Congestion Prediction Library
Congestion Detection Algorithms
Real-time Traffic Flow
Object Tracking
Integra4on & Delivery Toolbox
Alerting Module
Control System Metadata
KML & Ruby/Rails Library
Email & Workforce Activation
Customer Extensions Customer Extensions Customer Extensions
Travel Time App
Travel Time Subsystem
Customer Extensions
Conges4on App
Congestion Subsystem
Customer Extensions
Network KPIs App
Network KPIs Subsystem
Customer Extensions
Custom App
Custom App Subsystem
Distributed SQL & UDX(Java) Execution Engine
Query Management Distributed Query Optimization Query Scheduler
Platform - Distributed Server Cluster
JDBC Driver Adapter Plug-ins UDX (Java) Libraries Agent Framework
StreamApp End-user Applications
StreamApp Toolkits & External Libraries
StreamApp Execution Platform
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 23
Real-‐4me Data Hub for SmartCity StreamApp
SmartCity StreamApp – Solution Architecture Roadside Furniture
Traffic !Lights
Speed Signs
Agents
Vehicle GPS Bus & Train Roads Agents
Weather Feeds Agents
Social Media
Twitter Feeds Agents
Historical Trends Data!
Warehouse Predictive Analytics
Adapters
Logs
XML, JSON
XML, JSON
JDBC, API
Write API Live Road Updates Traffic !Lights
Speed Signs
HTTP RT Travel Time Apps
HTTP RT Conges4on Maps
Smart phone Device
Website
KML/Ruby RT Traffic Opera4ons
Real-time Dashboards
KML/Ruby Planning
KPI Reports
Real-time updates to traffic light phasing, speed signs and dynamic lane configuration
Commuters on the move
Commuters travel planning
Network Operations
Network Planning
SmartCity StreamApp
Streaming Integration Layer Distributed Stream Processing
Execution Environment
Operational Servers
Internet!Servers
Historical Trends Data!
Warehouse Predictive Analytics
Adapters JDBC, API
Copyright © 2014 – Proprietary and Confidential Information of SQLstream Inc. 24
> Minimum Time to Value Combination of subscription-based GPS vehicle feeds with pre-built StreamApp means operational system for
first requirements available in a few weeks.
> Strong Return on Investment Estimated $20million saving on initial project, with > $100 million over project’s lifetime.
> Traffic congestion reduction Real-time operational displays and greater accuracy for long term forecasting means a significant reduction in
congestion at key hotspots.
> Improved commuter experience Accurate and reliable Travel Time estimates with real-time updates means commuters are better prepared and better able to plan re-routes.
SmartCity StreamApp – Operational Results
Top Related