Slide 1 6/30/2005

Open Architecture Publish-Subscribe Benchmarking

Bruce McCormick, EG&GLeslie Madden, NSWCDD

Slide 2 6/30/2005

ObjectivesObjectives! To gain familiarity with existing publish-subscribe

middleware! Understand ability of products to support bounded

latencies and sustained throughput required by combat system applications

! Understand architectural differences in products that may affect ability of products to scale to large numbers of processors & processes

! Provide a baseline of comparison as products migrate toward DDS compliance

Slide 3 6/30/2005

Data Store

SPLI

CE

queries

Application

Publish/Subscribe API

queries

Application

Publish/Subscribe API

SPLICE Daemon

Network

Node

Pub/SubM/W

Pub/SubM/W

pub

pub

sub

sub

pub

pub

sub

sub

waitsetwaitset waitsetwaitset

wait fordata

THALES SPLICE 2THALES SPLICE 2

Slide 4 6/30/2005

ND

DS

Application

Publish/Subscribe API

Publish/Subscribe M/W

Application

Publish/Subscribe API

Publish/Subscribe M/W

Network

Node

subs

crip

tion

subs

crip

tion

publ

icat

ion

publ

icat

ion

subs

crip

tion

subs

crip

tion

publ

icat

ion

publ

icat

ion

poll for datapoll for datacallbackcallback

RTI NDDS 3.0MRTI NDDS 3.0M

Slide 5 6/30/2005

Architectural Features Affecting Results

Architectural Features Affecting Results

Implements Reliable/FIFO – all issues delivered. Improperly sized queues can lead to flow control behavior

Implements reliable but not FIFO – most recent message issue (based on timestamp) overwrites previous issue

Receives data into message queues

Stores data into in-memory data store for future access

NDDSSPLICE

Slide 6 6/30/2005

Mett ToolMett Tool

Slide 7 6/30/2005

Mett DesignMett Design

Display/ControlHost 1

METT EngineHost 2

METT EngineHost 3

Test Stub Test Stub

groupthread

groupthread

rsh rsh

multicastrun test

command

Data Messages

tcp configure/launch

stub

Slide 8 6/30/2005

Test Stub Window

Test Configuration Window

Mett StubsMett Stubs

Slide 9 6/30/2005

Test EnvironmentTest Environment

! All nodes in the same subnet! Use multicast! Privileged User

– Mett sets high (Max – 5) RT thread and process priorities

– Not bound to any CPU (SMP)

Slide 10 6/30/2005

Host OS Description NetworkInterface

Alphe1 Solaris8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Alphe2 Solaris8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Alphe3 Solaris8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Alphe4 Solaris 8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Serpens Solaris 8SUN Ultra-Enterprise 8 processor UltraSparc – sun4u Fast Ethernet

Alphe4 Solaris 8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Alphe5 Solaris 8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Alphe6 Solaris 8 SUN Ultra2 w/ Dual 200 MHz UltraSparc - sun4u128 MB RAM Fast Ethernet

Host ConfigurationHost Configuration

Slide 11 6/30/2005

Single topic with one publisher and 5 subscribers

Single topic with 5 publishers and one subscriber

Host 1

Host 2 Host 3

Host 4 Host 6

Host 6

Host 1

Host 2 Host 3

Host 4 Host 6

Host 6

Single topic with one publisher and onesubscriber

Host 1 Host 2

Configuration Setup

Test SetupsTest Setups

Slide 12 6/30/2005

MethodologyMethodology

Subscriber Publisher

One-way latencies

Time recv - Time sent

• NTP for clock sync• NDDS 3.0m configuration

– Reliable queue size 1

• Splice 2• Vendor tool code generation

– C libraries

Time Synchronization

0

0.02

0.04

0.06

0.08

0.1

Mill

isec

onds

Max Offset

Slide 13 6/30/2005

Latencies of 200 Byte Message Delivery at 10 Hz

Small Message Low RateSmall Message Low Rate

Slide 14 6/30/2005

Average Latencies of Different Message Sizes at Different Rates

Reliable Delivery - Average Latencies

Reliable Delivery - Average Latencies

Slide 15 6/30/2005

Maximum Latencies of Different Message Sizes at Different Rates

Reliable Delivery - Maximum Latencies

Reliable Delivery - Maximum Latencies

Slide 16 6/30/2005

Maximum Latencies for SPLICE Reliable, NDDS Reliable/FIFO, and UDP at 2000 Hz

UDP, NDDS, & SPLICE – Maximum Latencies

UDP, NDDS, & SPLICE – Maximum Latencies

Slide 17 6/30/2005

Average Latencies of Different Message Sizes at Different Rates with Collocated Hosts

Simultaneous NDDS & SPLICE –Average Latencies

Simultaneous NDDS & SPLICE –Average Latencies

Slide 18 6/30/2005

Maximum Latencies of Different Message Sizes at Different Rates with Collocated Hosts

Simultaneous NDDS & SPLICE -Maximum Latencies

Simultaneous NDDS & SPLICE -Maximum Latencies

Slide 19 6/30/2005

Average Latencies of Five Publishers Sending Data To One Subscriber

5 Publishers to 1 Subscriber -Average Latencies

5 Publishers to 1 Subscriber -Average Latencies

Slide 20 6/30/2005

Maximum Latencies of Five Publishers Sending Data to One Subscriber.

5 Publishers to 1 Subscriber –Maximum Latencies

5 Publishers to 1 Subscriber –Maximum Latencies

Slide 21 6/30/2005

Maximum Latencies of One Publisher Sending Data to Five Subscribers – NDDS Best Effort

NDDS 1 Publisher to 5 Subscribers Best Effort – Maximum Latencies

NDDS 1 Publisher to 5 Subscribers Best Effort – Maximum Latencies

Slide 22 6/30/2005

Maximum Latencies of One Publisher Sending Data to Five Subscribers – NDDS Reliable/FIFO

NDDS 1 Publisher to 5 Subscribers Reliable - Maximum Latencies

NDDS 1 Publisher to 5 Subscribers Reliable - Maximum Latencies

Slide 23 6/30/2005

Maximum Latencies of One Publisher Sending Data to Five Subscribers – SPLICE Best Effort

Splice 1 Publisher to 5 Subscribers Best Effort - Maximum Latencies

Splice 1 Publisher to 5 Subscribers Best Effort - Maximum Latencies

Slide 24 6/30/2005

Maximum Latencies of Publisher Sending Data to Subscribers – SPLICE Reliable

Splice 1 Publisher to 5 Subscribers Reliable - Maximum Latencies

Splice 1 Publisher to 5 Subscribers Reliable - Maximum Latencies

Slide 25 6/30/2005

Middleware Solaris CPU Usage For SubscribersTake and read

Subscriber CPU UsageSubscriber CPU Usage

Slide 26 6/30/2005

Middleware Solaris CPU Usage For PublishersTool overhead included

Publisher CPU UsagePublisher CPU Usage

Slide 27 6/30/2005

Middleware Message Overhead

0

5000

10000

15000

20000

25000

30000

sent recv sent recv

Publish Subscribe

19,000 Content Messages

Tota

l Mes

sage

s

Splice

NDDS

Middleware Message Overhead

IPC Message OverheadIPC Message Overhead

Slide 28 6/30/2005

SummarySummary! Overall, both products perform well within the

range of performance requirements of typical US Surface Navy Domain combat systems.

! Current indications are that both will provide good performance and scalability as a publish-subscribe middleware for combat system applications.

! Based on the results to date, it appears likely that performance of the products may not be a major differentiator, but other features not examined in this evaluation, such as life-cycle cost and support may influence the selection of one product over the other.

Slide 29 6/30/2005

What’s Next?What’s Next?! Testing of DDS-compliant products as they

become available! Evaluation of marshalling overhead associated

with different, domain applicable IDL representations

! Evaluation of cross-platform performance issues, including marshalling

! Performance with large message sizes! Performance on real-time operating systems such

as LynxOS, RT Linux