Http:// FIWARE WP6 GEs Overview Sergio García Gomez Telefónica Digital, WP6 Leader & Architect...
-
Upload
nigel-knight -
Category
Documents
-
view
213 -
download
0
Transcript of Http:// FIWARE WP6 GEs Overview Sergio García Gomez Telefónica Digital, WP6 Leader & Architect...
http://www.fi-ware.euhttp://www.fi-ppp.eu
FIWARE WP6 GEs OverviewSergio García GomezTelefónica Digital, WP6 Leader & [email protected]
WP6 overview
CDVA
WP6 provides a set of GEs for gathering, processing, interchanging and exploiting data at large scale.
Publish Subscribe (Context) Broker
Publish Subscribe (Context) Broker
It enables publication of context information by entities (Context Producers), so that published context information becomes available to other entities (Context Consumers), which are interested in processing the published context information.
Applications or even other GEs in the FI-WARE platform may play the role of Context Producers, Context Consumers or both.• Producers and Consumers are totally decoupled.
Two ways communications: push and pull towards both the Context Producer and the Context Consumer.
Data persistence of context events.
Based on NGSI-9 and NGSI-10 specifications.
Two different implementations available in FI-WARE, by Telecom Italia and Telefonica.
Big Data
Big Data Data analysts and applications that want to extract some
knowledge from big sets of data through Map&Reduce algorithms.
Two working modes• Batch processing mode (latency is not important) based on Hadoop (de
facto standard) and MongoDB
• Stream processing mode (latency is important). Stream injectors + data buckets feeding the processing module.
Plugable intelligence modules both for batch & stream, without taking care of distribution/scalability.
Following the EC recommendation, the existing implementation is being superseded by the one used in Telefonica for commercial services (COSMOS).• Planned to be available by the end of 2nd major release.
Big Data Architecture
Complex Event Processing CEP
Complex Event Processing
Users: applications playing the role of event producers (source point of events) and/or event consumers (sink point of events). • Total decoupling of both roles is realized and complex situations
rather than simple events are detected.
It analyses event data in real-time, generates immediate insight and enables instant response to changing conditions.
Complex Event Processing
While standard reactive applications are based on reactions to single events, the CEP GE reacts to situations rather than to single events.•A situation is a condition that is based on a series of events that
have occurred within a dynamic time window called processing context.
•Situations include composite events (e.g., sequence), counting operators on events (e.g., aggregation) and absence operators.
Architecture / How it works
Event Processing Networks (EPNs)
Architecture / How it works (II)
- Start up
- Run time
Constantly polls data source for changes (1,2), transforms an entry within the data source into CEP readable event object (3), sends the information via server connector to the server (4), without being aware of the underlying communication infrastructure which the connector uses to establish the connection & send the data to the server.
Upon startup it is supplied with server connector which handles all communication of the adapter with CEP runtime.
Compressed Domain Video Analysis
CDVA
CDVA
Users: all applications that want to extract meaningful information from video content and that need to automatically find characteristics in video streams on given tasks.
CDVA is a set of tools for analysing video streams in the compressed domain. Its purpose is to avoid costly video content decoding prior to the actual analysis. •Thereby, the tool set processes video streams by analysing
compressed or just partially decoded syntax elements.
The GE can work for previously stored video data as well as for video data streams (e.g., received from a camera in real time).
Architecture / How it works
CDVA APIs & standards
ONVIF specifications and OASIS Webservices Notification standard define XML structures and elements used within the notification module of CDVA.
Codoan tools use RTP (RFC3550) and RTSP (2326) as transport protocols.
Common video codecs are integrated in the tool stack. E.g. H.242/AVC, MPEG-4.
Descriptions of RTP/RTSP payload can be used, for instance RFC3984 for H.242/AVC.
ISO Base Media File Format as standardized in ISO/IEC 14496-12
Location Server
Location Server
Users: Any application that aims to retrieve mobile device positions and Location area events.• 3rd party location clients: uses OMA MLP or OMA RESTful
Network API
•Mobile end-users: OMA SUPL (Secure User Plane)
Addresses issues related to Location of mobile devices in difficult environments such as urban canyons and light indoor environments where the GPS receiver in the mobile device is not able to acquire GPS signals.
It improves GPS coverage whilst waiting for a GPS position, which helps to enhance the user experience of end-users (mobile), and the performance of applications requesting the position of mobile devices.
Architecture / How it works
Elements:1. SLP = SUPL Location Platform, containing: SPC (SUPL Positioning Centre), QoP manager and Access Control&Privacy Management.2. SET = SUPL Enabled Terminal. Talks to SLP with SMS and SUPL over TCP/IP.3. GPS&SBAS Receivers. Talk to SLP over TCP/IP.4. Third Parties. Talk to SLP with MLP (or NetAPI Terminal Location, with limited functionallity ) over TCP/IP.
Query Broker
Query Broker
Unified access to distributed & heterogeneous data (especially, MM) repositories.
Abstraction from heterogeneous retrieval paradigms in the underlying DB & search engines.•Capable to handle queries formulated in any of a defined set of
query languages/APIs (SQL, XQuery, SPARQL)
• Incoming queries converted into an internal abstract format MPQF (extends XQuery functionalities).
•MPEG Query Format (MPQF) supports most functions in traditional query languages and also incorporates several types of multimedia specific queries (temporal, spatial…).
Query Broker ArchitectureTwo main query processing strategies are supported:• Local processing: when registered and participating retrieval systems
are able to process the whole query locally.• Distributed processing: registered and participating retrieval
systems that allow distributed processing on the basis of a global data set.
Metadata Pre-processing
Metadata Pre-processing
Usage/Users: applications that need to convert metadata formats or need to generate objects (as instantiation of classes) that carry metadata information.
In real life various components implementing different metadata formats need to inter-work and then requirements to transform metadata appear.
Typically products from different vendors are plugged together. In this case, the “Metadata Pre-processing” GE acts as a mediator between the various products.
Metadata Pre-processing features
Timed meta-data is received and transformed. Not restrictive to specific metadata schemes.
The input stream is encapsulated in the Real-time Transport Protocol (RTP).
Input and output data format is XML. Alternatively, the input data can be transformed into serialized Java classes using XSLT.
Encapsulation of transport and metadata transformation can be implemented as-a-service, usable from other web applications or components.
Semantic Annotation
Semantic Annotation
Users/Usage: applications willing to enrich content by the means of: •Augmented content (news, books, etc): additional
information + links to LOD
•Filtering and search: LOD resources used as categories/tags.
• “Html snippets”, describing content for dbpedia entries.
It performs "Named Entity Recognition” & semantically links them with Linked Open Data (LOD) objects: persons, places and organizations in a text.• Each entity is passed to a semantic broker, which tries to
identify the correct correspondence over the most used linked open data repositories (dbpedia, geonames).
Semantic Application Support
Semantic Application Support
Users/Usage: Semantic web apps needing:•an infrastructure for semantic web apps that support large
scale applications including: metadata storage in RDF, publication of RDF triples, querying by SPARQL and inference.
•a framework for supporting methodologies and engineering processes related with metadata management and ontology development.
• It provides a framework for ontology engineers and developers of semantically-enabled applications offering RDF/OWL management, storage and retrieval capabilities.
An infrastructure for metadata publication, retrieval and subscription (scalable, distributed and secure).
KIARA communication middleware
KIARA – FI-WARE Advanced Middleware
Flexible, efficient, scalable and secure communication between distributed applications & GEs, based on Data Distribution System• Simple for developers, through simplified declarative IDL and APIs for the
usage of the application native data types.
• Dynamic and transparent negotiation & selection of the communication mechanisms, protocols and data representations (from SOAP/REST to binary protocols).
• High level specification of QoS and security (policies) requirements.
Some additional features:• Run time optimization with embedded compiler/Interpreter
• Different communication patterns (RPC, N-to-N, Publish-Subscribe, ...)
• Synchronous, asynchronous operations
• Extended transport protocols and mechanisms (Shared Memory, UDP, SDN Plugin,...)
• High performance dispatching agent (RPC)
• C & C++ support in first release
32
KIARA – Components & Advanced features
Application describes its data types and structures
Middleware generates support code at runtime (from IDL & declaration)
• Data mapping between app and IDL (renaming, type transformation, ...)
• Support of different marshaling formats (CDR, XML, JSON)
• Optimized by embedded compiler
Meta-data annotations (QoS, Security, Transport)
Negotiation
• Best Transport mechanism / protocol (SDN Support)
Definition of Security policies
33
Application
IDLParser
IDL Interface
Definitions
• KIARA IDLwith SecurityAnnotations(Based on OMG IDL)
• WADL
API / Data Access
Marshalling / Serialization
Mon
itorin
g
QoS
Neg
otia
tion
TransportMechanis
ms
Interaction / Service Protocols
RESTDDS
Service Transport- / Wire- Protocols
RTPS HTTP-
Network Transport Protocols
UD
P
TC
P
TLS
, D
TLS
Sh
are
d
Mem
ory
Backp
la/
Fab
ric
XML JSONCDR
Secu
rity
SDN Plugi
n
Data Transfer
Embedded Runtime Compiler/Interprete
r
Data Mapping
Declarative Data/Function
description
Security / QoS
Parameter
Security / QoSPolicy
34
Thanks !!
http://www.fi-ppp.eu