Introduction to OMG DDS (1 hour, 45 slides)

Your systems. Working as one.

Introduc)on to OMG DDS

OMG Technical Mee9ng, Washington D.C., March 2013 Gerardo Pardo-‐Castellote, Ph.D. [[email protected]] CTO, Real-‐Time Innova9ons, Inc. [www.r9.com] Co-‐chair of the OMG Data-‐Distribu9on SIG

Systems that interact with the Real World

•  Must adapt to changing environment •  Cannot stop processing the informa9on •  Live within world-‐imposed 9ming

Beyond tradi9onal interpreta9on of real-‐9me

2 © 2012 RTI • ALL RIGHTS RESERVED

3

Challenge: More Data, More Speed, More Sources

TRENDS: •  Growing Informa9on Volume •  Lowering Decision Latency •  Increasing System Availability •  Accelera9ng technology inser9on and deployment

Next-‐genera)on systems needs: •  Scalability •  Integra9on & Evolu9on •  Robustness & Availability •  Performance •  Security


“Real World” Systems are integrated using a Data Model

•  Grounded on the “physics” of the problem domain –  Tied to the nature of the sensors and real objects in the system (vehicles,

device types, …)

•  Provides governance across disparate teams & organiza9ons –  The “N^2” integra9on problem is reduced to a “N” problem

•  Increased decoupling from use-‐cases and components –  Avoids over constraining applica9ons

•  Open, Evolvable, Plaborm-‐Independent –  The use-‐cases, algorithms might change between missions or versions of

the system

Realizing this data-‐model requires a middleware infrastructure

App App App


DDS: Standards-‐based Integra9on Infrastructure for Cri9cal Applica9ons

Streaming Data

Sensors Events

Real-‐Time Applica)ons

Enterprise Applica)ons

Actuators


RPC over DDS

2013

Family of Specifica9ons

6

DDS Implementa9on

Network / TCP / UDP / IP

App

DDS Implementa9on

App

DDS Implementa9on DDS

Interoperablity

2006

UML Profile for DDS

2008

DDS for Lw CCM

2009

DDS X-‐Types

2010

DDS Security

2013 2010

DDS-‐STD-‐C++ DDS-‐JAVA5

App


DDS Spec

2004

7

Broad Adop9on

•  Vendor independent – API for portability – Wire protocol for interoperability

•  Mul9ple implementa9ons –  10 of API –  8 support RTPS

•  Heterogeneous –  C, C++, Java, .NET (C#, C++/CLI), ADA, Python, Scala, …

–  Linux, Windows, VxWorks, Integrity, other embedded & real-‐9me

•  Loosely coupled

DDS Real-‐Time Publish-‐Subscribe Wire Protocol (RTPS)

Middleware

DDS API

Cross-‐vendor portability!

Cross-‐vendor interoperability!

© 2012 RTI • ALL RIGHTS RESERVED

8

Interoperability between the applica)ons implemented by six different vendors (March 2012)

OCI ETRI PrismTech IBM RTI TwinOaks

DDS mandated by key DoD Programs •  UK Generic Vehicle Architecture

–  Mandates DDS for vehicle comm. –  Mandates DDS-‐RTPS for interop.

•  DISR – Mandates DDS for Pub-‐Sub API – Mandates DDS-‐RTPS for Interop

•  Army, OSD – UCS, Unmanned Vehicle Control –  FACE (Future Airborne Capability

Environment) •  US Navy Open Architecture

– Mandates DDS for Pub-‐Sub •  SPAWAR NESI

– Mandates DDS for Pub-‐Sub SOA


DDS Deployed Applica9on Examples

© 2012 RTI • ALL RIGHTS RESERVED 10

Everyday Example: Calendaring

Alterna)ve Process #1 (message-‐centric): 1.  Email: “Mee9ng Monday at 10:00.”

2.  Email: “Here’s dial-‐in info for mee9ng…”

3.  Email: “Mee9ng moved to Tuesday”

4.  You: “Where do I have to be? When?”

5.  You: (siFing through email messages…)


Example: Calendaring

Alterna)ve Process #2: 1.  Calendar: (add meeIng Monday at 10:00) 2.  Calendar: (add dial-‐in info) 3.  Calendar: (move meeIng to Tuesday)

4.  You: “Where do I have to be? When?” 5.  You: (check calendar. Contains

consolidated-‐state)

The difference is state! The infrastructure consolidates changes and maintains it


Copyright © 2010 RTI -‐ All rights Reserved. . 13

Data-‐Centric Middleware allows applica9ons to be integrated to the Informa9on Model

APP APP APP APP

DDS Global Data Space

Data Model

Standard Mapping(*)

Standard API

No custom mappings / code necessary Direct support for data-‐centric ac9ons: create, dispose, read/take

Copyright © 2010 RTI -‐ All rights Reserved. . 14

Integra9ng components to generic middleware technology

Comp Comp Comp Comp

Middleware Ar)facts

Data Model

Custom Mapping

Custom Integra9on

Akin to implemen9ng an OO design on a Procedural Language: Requires mapping inheritance, encapsula9on, excep9ons, …

Tradi9onal data-‐centric technologies not suited to scalable near real-‐9me systems

Other data-‐centric technologies: –  Databases: SQL –  Web: HTTP (mostly)

•  …assume the world changes slowly •  …use network resources inefficiently •  …are highly centralized

State

Server App

App

App App

App

App

Slow A few updates/sec

Not scalable 100 apps => 100x load

Unreliable Failure here kills many apps 15 © 2012 RTI • ALL RIGHTS RESERVED

DDS is decentralized. Can be deployed without servers/brokers

DDS: •  …allows you to observe frequent changes •  …uses network resources efficiently •  …is peer-‐to-‐peer and decentralized

App App App App App App

Fast 100,000’s updates/sec

Scalable Load indep. # apps

Reliable No single pt. failure

Managed with QoS

DDS Data-‐Centric Bus


The data-‐centric communicaIons model


DDS Adressing: Data-‐Objects in the Global Data Space

•  Domain: world you’re talking about •  Topic: group of similar objects

–  Similar structure (“type”) what –  Similar way they change when

over 9me (“QoS”) how •  Instance: individual object

•  DataWriter: source of observa9ons about a set of data-‐objects (Topic)

•  DataReader: observer of a set of data-‐objects (Topic)

Domain (e.g. Yellowstone Park)

Topic (e.g. bears in the park)

Topic Snow Depth Sensors

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

ID GPS value

Instance (e.g. Yogi the bear)

Booboo Instance

Data-‐Centric Qos-‐Aware Pub-‐Sub Model

Persistence Service

Recording Service

Virtual, decentralized global data space

CRUD opera9ons

Source (Key) Speed Power Phase

WPT1 37.4 122.0 -12.20

WPT2 10.7 74.0 -12.23

WPTN 50.2 150.07 -11.98


Demo: Publish-‐Subscribe ShapesDemo


Data Reader “Alarm”

Domain Par9cipant

Data Writer “Alarm”

Domain Par9cipant

Data-‐Centric Communica9ons Model

•  Par)cipants scope the global data space (domain) •  Topics define the data-‐objects (collec9ons of subjects) •  DataWriters publish data on Topics •  DataReaders subscribe to data on Topics •  QoS Policies are used configure the system •  Listeners are used to no9fy the applica9on of events

Listener Offered QoS Listener

Got new data

Requested QoS

New subscriber!

example


Quality of Service (QoS)

•  Aside from the actual data (WHAT) to be delivered, users ozen need to specify HOW to send it … … reliably (or “send and forget”) … how much data (all data , last 5 samples, every 2 secs) … how long before data is regarded as ‘stale’ and is discarded … how many publishers of the same data is allowed … how to ‘failover’ if an exis9ng publisher stops sending data … how to detect “dead” applica9ons … …

•  These op9ons are controlled by formally-‐defined Quality of Service (QoS) policies

Real-‐Time Quality of Service Control

•  Quality of Service (QoS) determined per-‐en6ty

•  QoS Contract: Request -‐ Offered

•  Publishing and subscribing applica9ons can be no9fied when QoS contract is violated

–  e.g. Messages lost or deadlines missed

•  High availability via automa9c failover

Subscriber

Subscriber

Subscriber

Collision avoidance application

Radar Track Publisher

Reliable; 1 Hz;

get history

Best effort; 0.2 HZ; get history

Airline operations center

Best effort; 0.01 Hz; no history

Database, real-time log

•  Publish reliably •  10 Hz •  Keep 10 minute history (6000 samples)

Backup Publisher

Real-‐Time Quality of Service (QoS)

QoS Policy DURABILITY

HISTORY

LIFESPAN

WRITER DATA LIFECYCLE

READER DATA LIFECYCLE

ENTITY FACTORY

RESOURCE LIMITS

RELIABILITY

TIME BASED FILTER

DEADLINE

CONTENT FILTERS

Cache

User Q

oS

Delivery

Presenta)on

Availability

Resources

Transport

QoS Policy USER DATA

TOPIC DATA

GROUP DATA

PARTITION

PRESENTATION

DESTINATION ORDER

OWNERSHIP

OWNERSHIP STRENGTH

LIVELINESS

LATENCY BUDGET

TRANSPORT PRIORITY

Qos in Ac9on

•  Detec9ng presence

•  History cache •  Dele9ng objects

•  Ownership •  Liveliness •  Filtering •  Durability

© 2009 Real-‐Time Innova9ons, Inc. 25

ShapesDemo

OperaIonal robustness and performance


Architecture for the next-‐genera9on systems

•  Exis9ng technologies are reaching robustness/performance/scalability limits

•  DDS provides a fundamentally new DataBus architecture and approach –  Powerful data-‐centric model

–  Ultra-‐scalable and robust –  Fully decentralized, peer-‐to-‐peer, “no bo}lenecks” architecture –  Superior Wire Protocol –  Standards-‐based, mul9-‐plaborm

Single-‐lane traffic No priori9za9on

Brokers as choke-‐points Connext DDS Approach


0

50

100

150

200

250

300

350

400

Average Laten

cy (M

icrosecond

s)

Throughput (Messages per Seconds)

1 (1 per CPU and NIC)

20 (1 per CPU and NIC)

40 (1 per CPU, 2 per NIC)

Performance

•  Reliable mul9cast •  Fully meshed, reliable

Number of Subscribers

Orders of magnitude faster than IT solu9ons

Fastest DDS solu)on


Performance results are vendor-‐specific These results are for RTI Connext DDS.

Scalability

1 ~1000 subscribers, < 15% throughput decrease

600,000

500,000

400,000

300,000

200,000

100,000

0

Messages pe

r Second

Pe

r Subscriber (2

00 Bytes)

0 200 400 600 800 1,000

Number of Subscribers

•  Scalable Performance!!•  Millions of data

elements!•  .5m updates/sec

(batched)!•  10s µs latency!•  1000s consumers

per update!•  Orders of magnitude more scalable than IT solu9ons


Scalability results are vendor-‐specific These results are for RTI Connext DDS.

Comparison with other technologies

Message Length (samples)

Adapted from Vanderbilt presenta9on at July 2006 OMG Workshop on RT Systems 30 © 2012 RTI • ALL RIGHTS RESERVED

Comparison results are vendor-‐specific These results compare with RTI Connext DDS.

Common use-‐cases / paRerns


Common Use Cases

•  Isola9ng subsystems •  Detec9ng presence of applica9ons •  Discovering publishers and subscribers •  Keeping a “last-‐value” cache •  Monitoring the health of applica9ons •  Monitoring the health of data •  Reliable data delivery •  Building a highly-‐available system •  Managing network load and behavior

3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 32

Isola9ng Subsystems

•  Concept of Domains and Domain Par9cipants

N1 App 1 Pub/Sub (A,B/C,D)

N2 App 2 Subscribe (C)

N4 App 4 Pub/Sub (D/C,E,F)

N4 App 5 Publish (C)

N3 App 3 Pub/Sub (E,F/A,C)

N5 App 6 Subscribe (B,C)

Domain

Single ‘Domain’ System

•  Container for applica9ons that want to communicate

•  Applica9ons can join or leave a domain in any order

•  New Applica9ons are “Auto-‐Discovered”

•  An applica9on that has joined a domain is also called a “Domain Par9cipant”

Isola9ng Subsystems

•  Use mul9ple domains tor scalability, modularity and isola9on

Node 1 -‐ App 1 Pub/Sub

Node 2 -‐ App 1 Subscribe


Node 4 -‐ App 2 Publish


Node 5 -‐ App 1 Subscribe

Domain A



Domain B

Domain C Added Func.

Mul)ple Domain System

Detec9ng presence of applica9on

•  DDS has a buil9n discovery service •  It provides the means for an applica9on to discover the presence of other par9cipants on the Domain – The Topic “DCPSPar9cipants” can be read as a regular Topic to see when DomainPar9cipants join and leave the network

•  Applica9ons can also include meta-‐data that is sent along by DDS discovery

Discover Publishers and Subscribers

•  DDS provides a mechanism to detect en99es, their capabili9es and requirements

•  An applica9on can discover all the other en99es in the system that match the requirements – The Topics “DCPSPublica9ons”, “DCPSSubscrip9ons”, “DCPSTopics”, and “DCPSPar9cipants” can be read to observe the other en99es in the domain


Publishing data that outlives it source

•  The concept of durability in the global dataspace –  Vola9le

•  No durability –  Transient local

•  Durability maintained by the publishing applica9on –  Transient

•  Durability maintained by a 3rd party applica9on (persistence service)

–  Persistent •  Durability of data is maintained by a 3rd party and saved to disk

•  Durability maintained azer restart


Keeping a “last value” cache

•  A last-‐value cache is already built-‐in into every Writer in the system

•  A late joiner will automa9cally ini9alize to the last value

•  Last value cache can be configure with history depth greater than 1

•  The Persistence Service can be used to provide a last value cache for durable data


Monitoring the health of applica9ons

•  DDS has mechanisms to monitor presence, health and ac9vity of all en99es

•  Supports a concept of liveliness –  Automa9c

•  Managed by the infrastructure – Manually asserted

•  Managed by the applica9on

•  What does it mean if the applica9on no longer publishes data? –  No news, good news? –  Applica9on has crashed? –  Applica9on is disconnected?


Monitor the health of data-‐objects

•  Concept of deadline – DDS can monitor the ac9vity of each individual data-‐instance in the system

–  If an instance is not updated according to the requirements of the receiving applica9on, the applica9on is no9fied

– Trigger for built-‐in failover mechanism

•  Concept of lifespan – DDS understands if a data-‐object has outlived its purpose and is considered ‘stale’ data


Ensuring a reliable data delivery

•  DDS supports a transport independent efficient reliability protocol –  In-‐order delivery – Reliable delivery – Detec9on and no9fica9on of data loss – Very configurable

• Warning thresholds

•  Heartbeats, gaps, acks, nacks, blocking or non-‐blocking behaviour


Building a high-‐available system

•  High Availability has mul9ple facets, many of which can be handled by DDS – Detec9on of presence

•  DISCOVERY – Detec9on of health and ac9vity

•  LIVELINESS, DEADLINE, LIFESPAN –  Survive applica9on and system failures

•  DURABILITY – Ability to handle redundant data source and failover

•  OWNERSHIP –  Reliable data transfer

•  RELIABILITY


Manage network load & behaviour

•  DDS provides mechanism to limit the data-‐rates – Filter by 9me – Filter by content – Par99ons – Use of mul9cast – Configured reliability level – TransportPriority QoS policy – LatencyBudget QoS policy


Conclusions •  DDS is a mature interna9onal Standard from OMG

–  Plaborm Neutral: Opera9ng systems and Programming Languages

–  Deployed worldwide in Military systems and other Demanding real-‐9me applica9ons

•  DDS Is mandated by DoD for Publish-‐Subscribe and data-‐distribu9on applica9ons

•  DDS is an ideal integra9on plaborm for Intelligent Systems

–  Highly Tunable via Quality of Service (QoS) –  Rich services (persistence, filtering, high-‐availability)


Find out more… www.r9.com

community.r9.com

demo.r9.com

www.youtube.com/real9meinnova9ons

blogs.r9.com

www.twi}er.com/RealTimeInnov

www.facebook.com/RTIsozware

www.slideshare.net/GerardoPardo

dds.omg.org

www.omg.org


Thank You!


dds.omg.org

www.omg.org

Introduction to OMG DDS (1 hour, 45 slides)

Documents

Transcript of Introduction to OMG DDS (1 hour, 45 slides)