Architecting the Internet of Things Darren Hubert M256.
-
Upload
barnard-skinner -
Category
Documents
-
view
220 -
download
0
Transcript of Architecting the Internet of Things Darren Hubert M256.
Architecting the Internet of ThingsDarren Hubert M256
Collecting information from lots of devices is cool - but it is just telematics.
Merging perspectives between devices, systems, and humans to build a better understanding of the world around us.
But tying together insight with action —there lies the promise of IoT.
Objectives
Understand what IoT means for IT architects!Learn the technologies, and how to begin applying them to deliver a modern IoT project!Be better able to talk to customers about IoT!
Session OverviewWhat is IoT, and why is it important?
Technologies Architects need to know
Quick real world example(s)
Common patterns and practices
IoT from an Architect’s perspective
Expectations
Level 200 ARCHITECTURE SessionThere will be no codeThere will be no ”flashing light” demos
IoT: The “Internet of Things”…
“ ”
The network of physical objects that contain embedded technology to communicate and interact with their internal states or the external environment.
Source: Gartner
25Billion (2020)
Source: Gartner
what is it?
Unique objects connected to InternetDevices, not peopleBi-directional communicationLarge, complex data flowsNew types of insight
why is it important?
Worldwide market for IoT solutions to reach $7.2 trillion in 2020 (IDC)Economic value-add is forecast to be $1.9 trillion across sectors in 2020 (Gartner)
Leading Industry examples : utilities, insurance, agriculture, factory, automobiles, transport, consumer, etc
IoT 2010
Cell phone
VoIP phone
HVAC
Computer
Vending
Printer
Security
Media player
Oven
Automobile
Smart scale
Refrigerator
Television
Microwave
Coffee maker
Alarm clock
HOME HOMEWORKPLACE
Sleep tracking
Home security Home automation Leak detection
Smart appliances
Indoor navigation
Health monitoring
Smart lighting
Pet tracking
Information capture
Trip tracking and car health
Control
Child and eldermonitoring
Sports and fitness
Air conditioning and temperature control Environmental
sensors
Behavior modification
Garden, lawn and plant care
Food and nutrition tracking
Beacons and proximity
New devices and sensors
Object tracking
Identity Smart vending machines
Medication adherence
Bike ride stats and protection
Entertainment systems
Office equipment
IoT 2015
HOME HOMEWORKPLACE
The Gartner Hype Cycle
Technology Trigger
Peak of Inflated
Expectations
Trough of Disillusionmen
t
Slope of Enlightenme
nt
Plateau of Productivity
The Gartner Hype Cycle
An Architect’s Perspective
POV: IoT is at an Inflection Point
Connectivity is
proliferating
Hardware Is getting
cheap
Cloud cost, scale,
flexibility
M2M solutions are mainstream
Software is more
advanced
General Technical Requirements
Volumes of Data
Vague Security
RequirementsEnd to End IntegrationMany Devices Large Scale
Information Exchange PatternsBasics of IoT Communication
Telemetry
Information flowing from a device to other systems for conveying status of device and environment
Inquiries
Requests from devices looking to gather required information or to initiate activities
Commands
Commands from other systems to a device or a group of devices to perform specific activities
Notifications
Information flowing from other systems to a device (group) for conveying status changes
IoT SolutionsHow to Solve those Basic Patterns
EndpointsCommand
RoutingMessage passing
Message Security
Publish-Subscribe
IoT Challenges
Data VolumeScale ConnectivityAddressing Device Size
First Principles
Data VolumeScale ConnectivityAddressing Device Size
Cloud Storage AMQP/MQTT
CloudPub/Sub Queues
Reference Architecture
Presentation Device, Event and Data ProcessingData Transport
Devices andData
Sources
Solution PortalProvisioning API
Device Identity & Registry
Analytics
Data Visualization
& Presentation
Device Management
Gateway
Storage
Low power
devices
ingestion
IP capable devices
Existing IoT
devices Event Processing
What Do Architects Need to Know?
Producers Data Transport Storage Analysis Presentation & action
Event Hubs (Service Bus)
SQL DatabaseMachine Learning
Azure Websites
Heterogeneous client agents
Table/Blob Storage
HD Insight/Storm
Mobile Services
External Data Sources
DocumentDBStream Analytics
Notification Hubs
External Data Sources
Cloud Services Power BI
External Services
{ }
Microsoft Azure IoT Services
Depth?
Event HubsTelemetry Ingestion ServicesScalable publish-subscribe telemetry ingestorsProcesses massive amounts of data (Generally Available WW)• 1M Publishers• 1 GB/S Ingress• 1T Messages/Month (1k per message)TTP/AMQP Protocol Support Pluggable adapters for other cloud services
DocumentDBFully managed NoSQL document db service
{ }
NoSQL JSON and JavaScript DB ServiceJSON DB for rapid DevelopmentSchema free – for storage and queryAutomatic indexing of every document propertyCRUD access, query, and JavaScript processing Integrates with HDInsight, Azure Search, etc.
{ }
Azure HDInsight100% Apache Hadoop-based service in the cloudScales from terabytes to petabytes on demandProcesses unstructured or semi-structured data from devices and sensorsDeployable in Windows or LinuxConnects with on-premises Hadoop clusters Apache Storm for real-time events Apache Spark for in memory data analysis
Azure Stream AnalyticsProcess real-time data in Azure. “Data in Motion”Real Time AnalyticsIntake millions of events per second (up to 1 GB/s)Correlate between different streams, or with static data or modelsEasy processing on continuous streams of dataEnables the detection of anomalies. Ability to trigger an alert when a specific error or condition appears.
Azure Machine LearningCloud based predictive analytics engine
Designed for “applied” machine learning, Streamlined experience for data scientists, across multiple skill levelsDrag-and-drop, and data-flow graphs to set up experimentsBuild and test predictive models, predict future trends or behaviorPublish models as a fully managed web service (API)
Raw materials Acquire Raw Materials
Transform raw materials into “finished goods”
Deliver
Data Sources
Ingest Transform and Analyze Publish
Azure Notification HubsScalable mobile push notification engineTarget millions of devices/messages with single API call. Target audiences with dynamic tags. Tailor notifications by audience, language, and locationUse with any back end, in the cloud or on-premisesDynamically define and reach audience segments
Microsoft PowerBILive, single pane of glass dashboard solution for visualizations and KPIs Cloud based business analytics service:• Track data in real-time with support for streaming data• Drill through to underlying reports to explore and discover new
insight• Pin new visualizations and KPIs to monitor performance
IoT Cloud Patterns
Field Gateway
Device Connectivity & Management
IoT Device & Cloud PatternsD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs
Field Gateway
Protocol Adaptation
Event Hubs• High scale telemetry ingestion
service• HTTP/AMQP protocol support• Each Event Hub supports
• 1 million publishers• 1GB/s ingress
• Generally available worldwide• 1 Trillion messages/month
Field Gateway
Device Connectivity & Management
IoT Device & Cloud PatternsD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs
Field Gateway
Protocol Adaptation
Additional IoT Requirements• Command & control• Device identity• Device registry• Device management
Field Gateway
Device Connectivity & Management
IoT Device & Cloud PatternsD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs&IOT Hub
Field Gateway
Protocol Adaptation
Protocol Adaptation
IoT Hub• Capability coming with Azure IoT
Suite• Bi directional D2C and C2D• Up to 10M devices• Telemetry ingestion• Command & control• Device registry & identity• Device Management• More…
Hot Path Business Logic
Service Fabric & Actor FrameworkField Gateway
Device Connectivity & Management
IoT Device & Cloud PatternsD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs&IOT Hub
Field Gateway
Protocol Adaptation
Protocol Adaptation
Analytics & Operationalized Insights
Batch “Cold Path” Analytics
Azure HDInsight, DocumentDB, Azure ML
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight (Storm)
{ }
{ }
Hot Path Business Logic
Service Fabric & Actor FrameworkField Gateway
Device Connectivity & Management
IoT Device & Cloud PatternsD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs&IOT Hub
Field Gateway
Protocol Adaptation
Protocol Adaptation
Analytics & Operationalized Insights
Presentation & Business Connectivity
Presentation &
Business Connectivity
App Service, Websites
Dynamics, BizTalk Services, Notification Hubs
Batch “Cold Path” Analytics
Azure HDInsight, DocumentDB, Azure ML
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight (Storm)
{ }
{ }
Hot Path Business Logic
Service Fabric & Actor Framework
Device Connectivity & Management
Pattern: Predictive MaintenanceD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs&IOT Hub
Protocol Adaptation
Analytics & Operationalized Insights
Presentation & Business Connectivity
Presentation &
Business Connectivity
App Service, Websites
Dynamics, BizTalk Services, Notification Hubs
Batch “Cold Path” Analytics
Azure HDInsight, DocumentDB, Azure ML
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight (Storm)
{ }
{ }
Hot Path Business Logic
Service Fabric & Actor FrameworkField Gateway
Device Connectivity & Management
Pattern: Service Delivery ManagementD
evi
ces
RTO
S, L
inux,
Win
dow
s, A
ndro
id, i
OS
Cloud Gateway
Event Hubs&IOT Hub
Field Gateway
Protocol Adaptation
Analytics & Operationalized Insights
Presentation & Business Connectivity
Presentation &
Business Connectivity
App Service, Websites
Dynamics, BizTalk Services, Notification Hubs
Batch “Cold Path” Analytics
Azure HDInsight, DocumentDB, Azure ML
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight (Storm)
{ }
{ }
Event Hub Stores
Streaming Data
Stream Analytics processes events as
they arrive in the EventHub
AML Model Web Service
BES endpoint
Power BI / D3
Dashboard
Data for
Real-time Processing
Aggregations
External Data
Azure Services
Azure SQLContains Historical
Data
Real time data stats
Azure Data Factory
Pipeline invokes AML Web Service
Real Ti
me
Batc
h
Example Architecture
Real Time Telemetry
Data
Azure Data Factory
Pipeline Moves Data
Batch updates of predictions
Data for
Real-time Processing
Real Time Energy
Consumption Data (Public
Source)
Event Hub Stores
Streaming Data
Stream Analytics processes events as
they arrive in the EventHub
AML Model Web Service
BES endpoint
Power BI / D3
Dashboard
Data Stream
Job
Hourly Prediction
Updates
External Data
Azure Services
Copy to Azure SQL
for batch predictions
Scrape Data
5 mins
Azure WebJob Runs jobs to scrape
data from public source
Azure SQLContains Historical
Energy Consumption Data
Real time data stats
Azure Data Factory
Pipeline invokes AML Web Service
Real Ti
me
Batc
h
Example Architecture
Old ways of Thinking can be dangerous Understand the business model Beware of new patterns: eventual
consistency, etc. Don’t focus on the device Avoid analysis paralysis. Get it done!
Risks of IoT
Examples
88 Acres. Darrel Smith, Dir. Energy and Sustainability, Microsoft RE&F
Architecture is at the centre of IoT IoT is Advanced ”Modern” Architecture IoT Projects are Complex - Teamwork is
necessary These projects are mission critical and
difficult We can’t learn everything - but we need
breadth Don’t be afraid - get started and learn
Summary
Complete your session evaluation now and win!
© 2015 Microsoft Corporation. All rights reserved.Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or
other countries.MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
Real Time Telemetry
Data
Event Hub Stores
Streaming Data
Stream Analytics
processes events as they arrive in
the EventHub
AML Model (Published
Web Service)
Power BI (Dashboard)
Data for
Real-time Processing
Streaming Data
Near Real-time
Updates
IoT Clients Azure Services
IoT Clients Azure Services
ML Predictions consumed
through the RRS web service
interface
Example Architecture 1
Example Architecture 4External
DataAzure
Services
External Data
Azure Services
On Premise
SQL Server
Data Strea
m
Devices and
Systems
Azure Data Factory Pipeline
schedules On- Premise Data
Transfer
Scored Result
s
Copy Activity
Azure Storage Blob
Input
Azure Data Factory
Pipeline invokes Copy and Compute
Services
Compute
Activity
Copy Activity
Azure Storage Blob Output (ML
Scores)
AML Model Predictions consumed
through the web service interface
Consu
me
s V
ia
AD
F
Azure Data Factory Pipeline
transfers data to On-Premise data
storageCopy
Activity
Consumes
Power BI Dashboard
Data at rest