Post on 21-Jan-2018
The Convergence of Robotics, the Web, and the Internet of Things
Michael McCool, Principal Engineer, Intel
IoT Dev Fest, June 4, 2017
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
AbstractWhat do the web, the IoT, and robotics have in common? For one, they can all benefit from the network effect, where the whole is greater than the sum of the parts. For the Web, the network allows distribution of information between agents anywhere in the world. For the IoT, the focus is on the remote monitoring and coordination of physical sensors and actuators, and the analysis of data to provide actionable insight. Robots can be defined as autonomous cyberphysical systems that can sense and manipulate their environment. Although not essential to the definition, robots are often connected to the internet as well, and can use it to enhance or enable their function in various ways: supporting, for example, telepresence on the web, remote sensing integrated with the IoT, or coordination between multiple robots and IoT devices, or just to offload computation. In the past these three ecosystems had their own software frameworks and standards but there are indications they are converging: IoT systems often integrate with cloud services that have web APIs. Web API standards are being used to define the data models for IoT devices. Machine learning models can be trained with data from the Web and used for real-time perception in robots. IoT gateways can be used to command robots by voice. Robots can serve their own graphical user interfaces over the Web. Robots are adopting service-oriented distributed software architectures and programming models originally developed for web servers. IoT devices and radio networks can be used a navigation beacons and even raw sensory data for robots. In this talk, I will talk about a number of these convergences and what the implications are.
2
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Outline and Agenda
• Ecosystem Overview• Web, IoT, and Robotics
• Requirements and technologies
• Commonalities and differences
• Trends and Technologies• Embedded HPC and AI
• Remote compute offload
• Real-time (sub)systems
• Semantic data modelling
• Emerging Standards• OSRF*/ROS2* and OMG*/DDS*
• OCF* and W3C* Web of Things
• OpenFog*3
Web
IoT Robotics
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
The World-Wide Web
• “Distributed information space with resources identified by URLs”
• Hyperlinked documents
• Delivered by Internet
• Displayed in browsers
• Many, many extensions…
• Search engines
• Web service RESTful APIs
• Applications as content
• Semantic markup
4https://en.wikipedia.org/wiki/World_Wide_Web
This Wikipedia and Wikimedia Commons image is from the user Chris 73 and is freely available at //commons.wikimedia.org/wiki/File:WorldWideWebAroundWikipedia.png under the creative commons cc-by-sa 3.0 license.
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Technologies Engendered by the Web
• Web Services
• RESTful: Set of properties that allows an online service to scale gracefully
• Stateless interface, small set of verbs, structured data
• Metadata: RAML, Swagger/OpenAPI
• Cloud Services: X-aaS, SD-X, virtualization…
• “Software applications as content”
• Containers and Microservices
• Docker, Snaps, etc.
• Compose services out of loose collections of other services
• Flexible structured data formats (and now efficient binary encodings…)
• JSON CBOR; metadata: JSON-Schema
• XML EXI; metadata: XSI
5
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Semantic WebTo enable Web 3.0:
Allow the meaning (semantics) of data to be understood
Technologies to facilitate data interchange:
RDF: Resource Description Framework
OWL: Web Ontology Language
SPARQL: Protocol and RDF Query Language
Syntaxes: XML, N3, JSON-LD, etc.
Ontologies (Vocabularies): Haystack, Schema.org, etc.
6/37
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
RDF: Resource Description Framework
7/37
General approach for describing data as sets of “triples”
Subject-Predicate-Object
RDF triples are stored in a distributed fashion
Federated database
Subsumes relational databases
Standard terms and relations are defined in ontologies (“vocabularies”), including RDF and OWL
Socrates
ManIs-a
Mortal
IsIs
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Internet of ThingsNetwork of devices, each capable of communication and associated with a set of sensors and/or actuators.
• Enable numerous applications
• Multiple verticals: home, industrial, city, building, vehicle, etc.
• Typically rely on backend systems for data analytics and services (eg web services)
• Edge + “infrastructure” nodes…8
https://commons.wikimedia.org/wiki/File:Nest_Learning_Thermostat_(cropped).JPG
Optimization Notice
Internet of Things (IoT) Systems
Gateway
Gateway
Analysis
(Server)
Human
Interface
Wi-Fi*
LTE
Wired
Endpoint
Endpoint
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Robots
• Cyberphysical systems with the capability to autonomously interact with their environments
• Ideally capable of autonomous
• Perception
• Planning
• Mobility
• Manipulation
• Human-Robot Communication
• Note: typical “Industrial Robots” may only be capable of a subset of these
10
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Typical Robot Software Stack
ServosWheel
Motors
RGB
Camera
Depth
CameraMotion Touch Mic Array
Discrete SensorsCamera Drivers Motor Drivers
Se
nso
r Fusio
n
Point Cloud
2DMap
Body Tracking
Face Tracking
Object Recognition
RGBD+IMU / SLAM
Speech RecognitionBehaviorControl
Reflex
Goals
Interact
Speech
OutputDialog
Control
User/ World
Knowledge
Gesture /
Action
World
View
Robot
State
Command
Inputs
Behavior Arbitrator
Path
Planning
Movement
External
Device
Control
Voice Command
External Controller
Gesture InputEnvironment
StimulusBehavior
BehaviorBehaviors
Applications
Acoustics
Op
tion
al In
pu
ts
Cloudoffload &control
IRUltra-
Sonic
Sense Plan Act
Sensor Hardware and Drivers Actuators
• Sense a potentially dynamic environment
• Determine current location
• Plan path and then navigate from one location to another
• Avoid obstacles
• Interact with humans as needed (avoid, follow, respond to commands, etc.)
Mobile Robotics
12
• Speech recognition
• Face recognition
• Body pose understanding
• Object recognition
• Dialogue management
• Context awareness
• Emotional state recognition
• Social relationship understanding
Social Robotics
13
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice14
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Robotics and the IoT
Robots are…
Cyberphysical systems with the capability to autonomously interact with their environments
Not just passive sensors
If connected to the internet, robots may also participate in the IoT ecosystem
Share experiences via the network
Use sensor data from other IoT devices
Offload work to external systems
Access large amounts of data for training data
Coordinate motion across multiple robots/devices
Next-generation connected robots are explicitly leveraging IoT technologies and middleware
But they also need to be autonomous15
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Comparative Requirements
16
Web IoT Robots
Scalability
Connectivity
ScalabilityLow costLow powerFunctional safetyConnectivity
Functional safety
High performanceReal time
ReliabilityAvailabilityFlexibilitySecurity
ReliabilityAvailabilityFlexibilitySecurity
ReliabilityAvailabilityFlexibilitySecurity
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Emerging Trends and Technologies
Artificial Intelligence
• Machine Learning
• Perception
• Semantic Processing
Embedded HPC (especially for Robotics)
• Computer Vision and Speech
• Localization and Mapping
Distributed computing
• Remote (cloud, fog) compute offload
• Service-oriented computing
Real-time (sub)systems
• Control systems
• Functional safety
17
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Case Studies
ROS*: Robot Operating System
• Microservices architecture, Web integration, ROS2, OMG’s DDS, embedded perception
OCF*: IoT standardization effort
• Microservices architecture using CoAP/CBOR, IoTivityimplementation of Smart Home example system
W3C* Web of Things: IoT standardization effort
• Standardized metadata for microservices architecture
OpenFog*: Edge computing standardization effort
• Infrastructure support for microservices architecture
18
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
ROS*: Robot Operating System
• Supports distributed system based on a graph of communicating nodes
• Both pub-sub and request-response patterns
• Large community with many existing nodes for sensors, planning, navigation, etc.
• Various visualization tools and simulators are also provided
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Autonomous Navigation ROS* Node Graph
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Rviz: Robot Visualization
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Robot Web Tools* and ROSWeb*
http://robotwebtools.org/ and https://github.com/EESC-LabRoM/rosweb22
ROSWeb Robot Manager
• Manage/control ROS robot via web console
• Uses rosbridge, which bridges ROS to a JSON-over-Websocketprotocol
• Javascript library can then access ROS functionality either from Node.js or a browser
• Can also use WebGL, TLS, D3S, Google maps, etc.
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
ROS2*
Designed to resolve a number of issues with current ROS architecture:
• Real-time support
• Quality of service control in communications
• Distributed node management
• Lifecycle management
• Determinism
Built on top of OMG’s DDS standards, a middleware framework for industrial IoT systems, that provides real-time pub-sub
https://github.com/ros2/ros2/wiki and http://www.omg.org/spec/DDS/
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
SAWR Project
Simple Autonomous Wheeled Robot
• Demonstration system for autonomous
robotics
• Can be built using either
• Intel® RealSense™ Robotic Development Kit,
which combines Aaeon’s UP* Board and an
Intel® RealSense R200 camera
• Intel® Joule™ compute module and an Intel®
RealSense ZR300 camera
• OSH design: laser-cut frame/BOM
https://github.com/01org/sawr
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Robotis* Turtlebot 3
25
Modular Robotics Research
and Training Platform
• Intel® Joule™ compute
module
• Intel® RealSense R200
camera
• Laser triangulation 360
degree rangefinder
• Real-time Arduino-based
subsystem
http://turtlebot3.robotis.com/en/latest/hardware.html
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization NoticeIntel Confidential
Intel® RealSense™ SDK: Perceptual Capabilities
Objects Database
Library of pre-
trained
or customer
specific objects
Object Tracking
Track objects of
interest, even when
robot and/or object
are moving
Object Localization
Determine
where objects
are in the scene
Object Segmentation Pixels that are part of
the object for
manipulation or
removal
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Person Interaction
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
OCF* & IoTivity*
Specification
Open Source Coordination
Business (Marketing, Strategy, Membership)
Open Source ProjectReference Implementation of OCF
(and Non-Spec Related Code)
Sponsored (funded) by OCF
Separate Governance
Coordination
April 2017
Open Connectivity Foundation Public Information
29
Certification
Data Modelling
Specs & Open Source ready simultaneously
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
OCF – Conceptual Framework
30 April 2017
Core Framework
ProfilesConsumer Enterprise Industrial Auto Education Health
Security, Identity & Permissions
DiscoveryData
Transmission
Data
ManagementDevice
Management
Transports(Smart)
RemoteAccess Cloud
Resource Model
Interaction / Data Model
Open Connectivit
y Foundation
Public Information
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
OCF – Resource Model
31 April 2017
Resource URI
rt: Resource Type
if: Resource Interface
prop: Policy
n: Resource Name
…
…
CommonProperties
ResourceSpecific
Identifies the type of resource
List of interfaces associated with the resource
Policy associated with resource: discoverable, observable, secure,
etc
Friendly name
§ The resource model, coupled with the common data model, provides the base interoperability of OCF
§ Any physical or software artifact on a device that needs to be manipulated or made visible across the network can be described via the resource model
§ A resource has a URI and a collection of Properties
links: Other Resource URI Resources can include links to
other resources
Open Connectivit
y Foundation
Public Information
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Resource Model - examples
32 April 2017
14
Resource URI :/a/light1
rt: oic.ex.light
if: oic.if.rw
prop: discoverable,
observable
n: myHallWayLight
State: 0 (OFF)
Dim Level: 0
Resource URI :/a/fan1
rt: oic.ex.fan
if: oic.if.rw
prop: discoverable
n: myKitchenFan
State: 1 (ON)
Speed: 10
Open Connectivit
y Foundation
Public Information
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
CREATE Create a new Resource on the Server
RETRIEVE Get the current State or representation of a Resource from a Server
UPDATE Request a partial or complete update to the information stored in a Resource
DELETE Remove a Resource from the Server
NOTIFY Request asynchronous notification of state changes in a Resource
Special-cased to support OBSERVE for publish/subscribe support
Common Interaction Model – Transport Agnostic
April 201733
Open Connectivity Foundation
Public Inform
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Let’s look deeper at a Resource …
34
Server - DevA
/dimming
/oic/d
/oic/res
Main Living Room Light
/switch
/oic/d
/oic/p
oic.r.switch.binary
oic.wk.d
oic.wk.p
false100
/switch true
/dimming oic.r.light.brightness
/oic/p LED-Light-Co, 40W Equivalent Bulb
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
There can be more Properties …
35
/oic/pmnml
mnmn
pi
http://ledlight.com/gg3-40
Led-Light-Co
GoGlow III - 40W
mnmo GG3-40
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Resource Type - Data Modeling
36
JSON Schema*
• Describes the Properties and
Representation
ReSTful API Modelling Language
(RAML)*
• Describes the Request and Response
*JSON Schema and RAML are normative and also can be used for code generation
"oic.r.switch.binary": {
"type": "object",
"properties": {
"value": {
"type": "boolean",
"description": "Status of the switch"
}
}
}
get:
responses :
200:
body:
application/json:
schema: |
{
}
Defined by:
(see OIC_Resource_Type_Specification)
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
OCF/IoTivity Smart Home Demo
https://github.com/01org/SmartHome-Demo37
Prototype Smart Home
• Example architecture for Smart Home
• Based on OSS IoTivityimplementation of OCF
• Combines cloud services, gateways, and sensor nodes
• Technologies: Docker containers, Node.js, Zephyr RTOS, HTTPS, CoAP, WebGL, RAML, JSON/CBOR
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
W3C Web of Things
https://www.w3.org/WoT/ and https://github.com/w3c/wot38
Thing
API A
Applications
Thing
API B
Applications
Interaction Model Interaction Model
Protocol Mappings Protocol Mappings
CoAPOneM2M
CoAP
OCFCoAP
BACnetBridging Gateway
Thing
Description
HTTP
W3C Working Group
• Standardize descriptions of data and interaction models for IoT devices (“Things”)
• Support semantic interoperability via linked data, JSON-LD, RDF, IoT ontologies
• Bridge multiple other standards
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
https://www.openfogconsortium.org/39
Description
• Reference architecture for distributed “fog” computation
• Fog: cloud computing, but supported throughout the network, both in data centers and at the edge
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Summary and Conclusions
• Web, IoT, and Robotics ecosystems to interoperate, but not merge
• Several key differences mean completely shared technology base is not feasible
• However, many core components can and should be shared
• Many key “IoT” services will depend on these systems interoperating
• New opportunities and challenges
• Moving cloud services to the edge:
• Embedded microservice architectures, fog computing, containerization
• Using web technologies and infrastructure in IoT and Robotics:
• RESTful APIs, cloud services, semantic interoperability
• Software development challenges:
• Data-centric, distributed computing, microservices, scaling, security
40
Copyright © 2017, Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.Optimization Notice
Legal Disclaimer & Optimization Notice
INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.
Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.
Copyright © 2014, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
Optimization Notice
Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision #20110804
41