11

Grids for Real-time and Streaming Applications

GCC2005 Beijing China

December 1 2005

Geoffrey Fox

Computer Science, Informatics, Physics

Pervasive Technology Laboratories

Indiana University Bloomington IN 47401

http://grids.ucs.indiana.edu/ptliupages/presentations/gcf@indiana.edu http://www.infomall.org

22

Four Data Streaming Application Areas Data Assimilation applied to link the data deluge

(satellites, sensors, seismometers) in real time to small and large scale parallel simulations• Use in Earthquake Science

Department of Defense (and Homeland Security) have built the Global Information Grid with a target architecture NCOW (Network Centric Operations and warfare)• They submit no jobs; rather stream data to brokers from

which they are filtered and distributed• Includes their rather dated distributed simulation HLA

Audio-Video Conferencing implemented with services and Grid messaging

Hand-held Grid linking PDA/cell-phones to Grids

3

Database

SS

SS

SS

SS

SS

SS

SS

SS

SS

SS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS

FS

FS

FS

FS

FS

FS FS

FS

FS

PortalFS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

OS

MD

MD

MD

MD

MD

MD

MD

MD

MD

MetaDataFilter Service

Sensor Service

OtherService

AnotherGrid

Raw Data Data Information Knowledge Wisdom

Decisions

SS

SS

AnotherService

AnotherService

SSAnother

Grid SS

AnotherGrid

SS

SS

SS

SS

SS

SS

SS

SS

FS

SOAP Messages

44

Key Concepts Grid of Grids (System of Systems) allows “library”

approach to composing Grids Service Oriented architectures (Web or Grid services)

are attractive for many/most distributed systems There are many applications that are NOT best

considered as jobs and files (classic Grid) but rather as streams and filters (services)

Services exchanging messages becomesServices exchanging streams (sets of messages)

Publish-Subscribe messaging gives better QoS and management than point to point messaging with negligible performance loss

Always use standards including those for GIS

5

Typical use of Grid Messaging in NASA

Datamining Grid

Sensor Grid

Grid Eventing GIS Grid

66

GIS and Sensor Grids OGC has defined a suite of data structures and services

to support Geographical Information Systems and Sensors

GML Geography Markup language defines specification of geo-referenced data

SensorML and O&M (Observation and Measurements) define meta-data and data structure for sensors

Services like Web Map Service, Web Feature Service, Sensor Collection Service define services interfaces to access GIS and sensor information

Grid workflow links services that are designed to support streaming input and output messages

We are building Grid (Web) service implementations of these specifications for NASA’s SERVOGrid

77

Grid Eventing via NaradaBrokering The Scripps Orbit and Permanent Array Center

(SOPAC) GPS station network data published in RYO format is converted to ASCII and GML

8

Grid Messaging with Grid of Grids Architecture

HPSearchManages

NaradaBrokering

Sensor Grid

WS-ContextStores dynamic data

WFS (GIS data)

Post beforeProcessing

Post afterProcessing

Notify

SubscribeDatabaseArchives

Web Feature Service

GIS Grid

GeographicalInformation System

DoD Principles

Filter orDatamining Grid

99

Earthquake Faults from the WMS Client

1010

WMS uses WFS that uses data sources

Railroads

RiversBridges

Interstate Highways

90

WFS Server

SQL Query

Railroads

[a-b]

SQ

L Q

uery

Riv

er [a

-d]

Bri

dge

[1-5

]

SQL QueryHigway [12-18]

`

ClientWMS

GetFeature

FeatureCollection

Get

Feat

ure

Feat

ureC

olle

ctio

n

<gml:featureMember> <fault> <name> Northridge2 </name> <segment> Northridge2

</segment> <author> Wald D. J.</author> <gml:lineStringProperty> <gml:LineString

srsName="null"> <gml:coordinates>

-118.72,34.243 -118.591,34.176 </gml:coordinates>

</gml:LineString> </gml:lineStringProperty> </fault> </gml:featureMember>

11

Google Map Client

Google Central

Google Map Client

UDDI

WFS2

Databases withSERVOGrid Faults

WFS1

SOAP

Sensor Grid

HTTP

Helper Services

Archived Real Time

DoD and Homeland Security can in a crisis combine custom geo-referenced data with that available from hundreds of thousands of computers from Microsoft, Yahoo and Google Just build simple services using Interoperability standards!

12

Real Time GPS and Google Maps

Subscribe to live GPS station. Position data from SOPAC is combined with Google map clients.

Select and zoom to GPS station location, click icons for more information.

13

Google maps can be integrated with Web Feature Service Archives to filter and browse seismic records.

Integrating Archived Web

Feature Services and Google Maps

14

Google Maps as Service

accessed from our WMS

Client

15

Google plus GIS Grid Integrated with Los Alamos Critical Infrastructure Simulations

for DHS

Natural Gas Layer

Energy Power Layer

1616

NaradaBrokering 2003-2006 Messaging infrastructure for collaboration, peer-to-peer and Grids

Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop)

Order-preserving message transport with QoS and security profiles Support for different underlying transport such as TCP, UDP,

Multicast, RTP SOAP message support and WS-Eventing, WS-RM and WS-Reliability.

• WS-Notification when specification agreed Active replay support: Pause and Replay live streams. Stream Linkage: can link permanently multiple streams – using in

annotation of real-time video streams Replicated storage support for fault tolerance and resiliency to storage

failures. Management: HPSearch Scripting Interface to streams and brokers

(uses WS-Management) Broker Topics and Message Discovery: Locate appropriate Integration with Axis2 Web Service Container (?) High Performance Transport supporting SOAP Infoset

1717

0

1

2

3

4

5

6

7

8

9

100 1000

Tra

nsit

Del

ay

(Mill

isec

onds

)

Message Payload Size (Bytes)

Mean transit delay for message samples in NaradaBrokering: Different communication hops

hop-2

hop-5 hop-7

hop-3

Pentium-3, 1GHz, 256 MB RAM100 Mbps LAN

JRE 1.3 Linux

18

Average Video Delays for one broker – Performance scales proportional to number of brokers

Latency ms

# Receivers

One sessionMultiple sessions

30 frames/sec

19

Collaboration GridServices

UDDINaradaBroker

HPSearch

WS-Context

Gateway

WS-Security

NaradaBroker

NaradaBroker

Gateway

Gateway

Gateway

XGSP MediaService

Video Mixer

Transcoder

Audio Mixer

Replay

Record

Annotate

Thumbnail

WhiteBoard

SharedDisplay

SharedWS

20GlobalMMCS SWT Client

Chat

TV

WebcamVideo Mixer

GIS

2121

e - Annotation Player

Archived stream player Annotation / WB

player

Archieved stream list

Real time stream list

e - Annotation Whiteboard

Real time stream player Archived Real Time Real Time

Stream List Stream List Player

e-Annotation Archived Stream Annotated e-Annotation

Player Player Stream Player Whiteboard

2222

Location of software for Grid Projects in Community Grids Laboratory

htpp://www.naradabrokering.org provides Web service (and JMS) compliant distributed publish-subscribe messaging (software overlay network)

htpp://www.globlmmcs.org is a service oriented (Grid) collaboration environment (audio-video conferencing)

http://www.crisisgrid.org is an OGC (open geospatial consortium) Geographical Information System (GIS) compliant GIS and Sensor Grid (with POLIS center)

http://www.opengrids.org has WS-Context, Extended UDDI etc.

The work is still in progress but NaradaBrokering is quite mature

All software is open source and freely available