Remote C ollaborative Real-Time Multimedia Experience over ... · Task Name Start Finish...

ROMEO WP6 /68

Remote Collaborative Real-Time Multimedia Experience over

the Future Internet

ROMEO

Grant Agreement Number: 287896

D6.3

Second Report on System Components Development Plan

ROMEO WP6 /68

Document description

Name of document Second report on system components development plan

Abstract This document defines ROMEO system’s components’ development plan according to the updates between first report and today. It takes architecture and first report as reference and builds development plan accordingly.

Document identifier D6.3

Document class Deliverable

Version 1.0

Author(s) H. Gökmen, B. Demirtaş, E. Ertaş, S. Marangoz, O. Dinçer, V.Ağ, U. Halatoğlu, H. Keskin (ARC) A. Akman, E. Çimen, S. Çiftçi, S. Ö. Pelvan (TTA) J. Lauterjung, H. Ibl (R&S) N. Just, P. tho Pesch, M. Weitnauer (IRT) H. Silva, H. Marques, J. Rodriguez, J. Ribeiro (IT) I. Politis, K. Birkos (UP) I. Elfitri, C. Kim, E. Ekmekcioglu (US) X. Shi, B. Evans (MULSYS) B. Iribarne (TID) N. Tizon (VITEC) D. Doyen (TEC) V. Petrov (MMS)

QAT team J. Lauterjung (R&S), K. Georgiev (MMS),A. Akman (TTA) and All Partners

Date of creation 03-Oct-2012

Date of last modification 28-Nov-2012

Status Final

Destination European Commission

WP number WP6

Dissemination Level Public

Deliverable Nature Report

ROMEO WP6 /68

TABLE OF CONTENTS

LIST OF TABLES ....................................................................................................................... 5

LIST OF FIGURES...................................................................................................................... 6

1.1 Purpose of the Document ........................................................................................... 8

1.2 Scope of the Work ....................................................................................................... 8

1.3 Objectives and Achievements ..................................................................................... 8

1.4 Structure of the Document .......................................................................................... 8

2 SYSTEM DEVELOPMENT PLAN ...................................................................................... 9

2.1 General System Development Plan ............................................................................ 9 2.1.1 Integration Steps ................................................................................................. 9 2.1.2 Integration Phase Relations with Milestones and Deliverables ........................ 10

2.2 General Integration Step Completion Procedure ...................................................... 10 2.2.1 Component Release and Test Procedure ......................................................... 11 2.2.2 Bug / Change Request Release Mechanism .................................................... 11

3 COMPONENT DEVELOPMENT PLANS ......................................................................... 12

3.1 User Terminal / Peer Development ........................................................................... 12 3.1.1 DVB Reception .................................................................................................. 12 3.1.2 P2P .................................................................................................................... 15 3.1.3 Video Decoding ................................................................................................. 19 3.1.4 Video Rendering ................................................................................................ 21 3.1.5 Audio Decoding ................................................................................................. 22 3.1.6 Audio Rendering ................................................................................................ 24 3.1.7 Synchronisation ................................................................................................. 25 3.1.8 Audio-Visual Communication Overlay ............................................................... 29 3.1.9 Audio-Visual Adaptation and Network Monitor.................................................. 31 3.1.10 User Interface and Control ................................................................................ 35 3.1.11 Authentication, Registration and Security ......................................................... 37 3.1.12 User Generated Content ................................................................................... 38

3.2 Server Development.................................................................................................. 40 3.2.1 Content Generation ........................................................................................... 40 3.2.2 Transport Stream Adaptation for Additional Content Signalling........................ 42 3.2.3 Video Encoding ................................................................................................. 43 3.2.4 Audio Encoding ................................................................................................. 44 3.2.5 P2P .................................................................................................................... 46 3.2.6 DVB Transmission ............................................................................................. 50 3.2.7 Network Monitor ................................................................................................ 51 3.2.8 Audio-Visual Communication Overlay ............................................................... 52 3.2.9 Authentication, Registry and Security ............................................................... 53 3.2.10 User Generated Content ................................................................................... 54

3.3 Network Related Development ................................................................................. 55 3.3.1 Mobility .............................................................................................................. 55 3.3.2 Virtualisation ...................................................................................................... 57 3.3.3 Internet Resource and Admission Control Subsystem (IRACS) ....................... 58

ROMEO WP6 /68

4 CONCLUSION .................................................................................................................. 61

5 REFERENCES .................................................................................................................. 62

APPENDIX A: TASK LIST OF ROMEO ................................................................................... 63

APPENDIX B: GLOSSARY OF ABBREVIATIONS ................................................................. 65

ROMEO WP6 /68

LIST OF TABLES

Table 1 – Component Development Plan ................................................................................................ 10

Table 2 - Development plan for DVB reception module in test terminal ................................................... 13



Table 5 - Development plan for Topology Controller module ................................................................... 16

Table 6 - Development plan for depacketisation component ................................................................... 17

Table 7 – Development plan for the Chunk selection module .................................................................. 18

Table 8 - Development plan for P2P Transmitter/Receiver module ......................................................... 19

Table 9 - Development plan for Video Decoder module........................................................................... 21

Table 10 - Development plan for the video renderer ................................................................................ 22

Table 11 - Development Plan for Spatial Audio Decoder ......................................................................... 24

Table 12 - Development plan for audio rendering modules ...................................................................... 25

Table 13 – Synchronisation component development plan ...................................................................... 29

Table 14 – A/V Communication Overlay component development plan .................................................. 31

Table 15: Development plan for the audio-video adaptation .................................................................... 33

Table 16 - Development plan for NMS component .................................................................................. 34

Table 17 – User Interface Module development plan ............................................................................... 37

Table 18 - Development plan for Authentication Registration and Security component ........................... 38

Table 19 - Development plan for User Generated Content Capture component ...................................... 39

Table 20 - Development plan User Generated Content component ........................................................ 40

Table 21 - Development plan for Visual Attention modules ...................................................................... 42

Table 22 - Development plan for Transport Stream Adaptation Module .................................................. 43

Table 23 - Development plan for Video Encoder module ......................................................................... 44

Table 24 – Development plan for Audio Encoding Component ................................................................ 46

Table 25 - Development plan for Topology Builder module ..................................................................... 48

Table 26 - Development plan for Multicast Tree Manager module ........................................................... 49

Table 27 - Development plan for packetisation module ........................................................................... 49

Table 28 - Development plan for P2P Transmitter module ...................................................................... 50

Table 29 - Development plan for DVB transmission module .................................................................... 51

ROMEO WP6 /68

Table 30 – A/V Communication Overlay component development plan .................................................. 53

Table 31 - Development plan for Authentication Registration and Security module ................................. 54

Table 32 - Development plan for Authentication Registration and Security module ................................. 55

Table 33 – Development plan for mobility component ............................................................................. 57

Table 34 - Development plan for Virtualisation ........................................................................................ 58

Table 35 - Development plan for Internet Resource and Admission Manager module. ........................... 60

Table 36 - Development plan for Internet Resource Controller module. .................................................. 60

LIST OF FIGURES

Figure 1 - Block diagram of the Test Terminal (green: tested and evaluated, yellow: under development, grey: not started yet) ........................................................................................................................ 13

Figure 2 – DVB Reception component in fixed and portable terminal ...................................................... 14

Figure 3 – DVB Reception component in Mobile Terminal Device ........................................................... 15

Figure 4 - Block diagram for Topology Controller ..................................................................................... 16

Figure 5 - Interaction diagram for depacketisation module ...................................................................... 17

Figure 6 - Block diagram of the interaction between Chunk selection and other modules ....................... 17

Figure 7 – Block diagram of the Chunk selection module ........................................................................ 18

Figure 8 - Block diagram of P2P Transmitter/Receiver ............................................................................ 19

Figure 9 - Block diagram for decoder module .......................................................................................... 20

Figure 10 - Video Rendering block diagram ............................................................................................. 21

Figure 11 – Diagram of Audio Decoder Module ....................................................................................... 22

Figure 12 - Diagram of ABS Spatial Audio Decoder................................................................................. 23

Figure 13 - Audio Rendering Module Diagram ......................................................................................... 24

Figure 14 – Synchronisation Component Detailed Block diagram ........................................................... 26

Figure 15 – PCR information in transport stream ..................................................................................... 27

Figure 16 – A/V Communication Overlay block diagram .......................................................................... 29

Figure 17 - Audio-Visual Adaptation block diagram ................................................................................. 32

Figure 18 – Block Diagram for A/V Adaptation & Network Monitor .......................................................... 33

Figure 19 – NMS interactions ................................................................................................................... 34

Figure 20 – User Interface and Control Detailed Block diagram .............................................................. 35

Figure 21 - Block diagram of Authorization Registration and Security component ................................... 37

Figure 22 – User Generated Content Capture Detailed Block diagram.................................................... 38

ROMEO WP6 /68

Figure 23 – Block diagram of user generated content.............................................................................. 39

Figure 24 - Diagram of the Visual Attention Module ................................................................................. 41

Figure 25 - Detailed block diagram of the “disparity map computation” block .......................................... 41

Figure 26 - Detailed block diagram of the saliency estimation from the disparity ..................................... 42

Figure 27 - Color based saliency block .................................................................................................... 42

Figure 28 - Transport Stream Adaptation Module .................................................................................... 43

Figure 29 - Diagram of the Video Encoder block ..................................................................................... 44

Figure 30 - Diagram of Spatial Audio Encoder ......................................................................................... 45

Figure 31 - Diagram of ABS Spatial Audio Encoder ................................................................................. 45

Figure 32 - Block diagram for Topology Builder ....................................................................................... 47

Figure 33 – Block diagram for the Multicast Tree Manager ...................................................................... 48

Figure 34 - Block diagram for packetisation module ................................................................................ 49

Figure 35 - Block diagram of the DVB Transmission part (green: tested and evaluated, yellow: under development, grey: not started yet) .................................................................................................. 50

Figure 36 – A/V Communication Overlay block diagram .......................................................................... 52

Figure 37 - Block diagram for Authorization Registration and Security component ................................. 54

Figure 38 - Block diagram for user generated content ............................................................................. 55

Figure 39 – Block diagram of Mobility component ................................................................................... 56

Figure 40 – Block diagram of virtualisation component ............................................................................ 57

Figure 41 - Block diagram for Internet Resource and Admission Manager. ............................................. 59

Figure 42 - Block diagram for Internet Resource Controller. .................................................................... 60

Figure 43 - ROMEO platform development plan ...................................................................................... 61

ROMEO WP6 /68

1. INTRODUCTION

1.1 Purpose of the Document This document is prepared to describe the plan of development of ROMEO components and prepared in accordance with ROMEO Deliverable 6.1 - First report on system components development plan. This version contains more detailed module explanations and plan for their completion.

1.2 Scope of the Work ROMEO will focus on the delivery of live 3D content to a group of users all at the same time to enable them to collaborate, and jointly enjoy it. There will be fixed, portable and mobile users to view the content. Since there are several tasks and blocks to complete in creation of these users and servers, system development is divided into several components. During the development of ROMEO project, task dependencies should be carefully analysed and the development should be planned accordingly. This document will help to build up a system working properly at the end. System components are defined in ROMEO Deliverable 2.2 [1]. “ROMEO Deliverable 2.2 - Definition of the initial reference end-to-end system architecture and the key system components”

1.3 Objectives and Achievements This document aims to form a consensus on the system development plan, which will be used through the project as a reference for all project team. Development plan will define component development dates and related milestones. Second report also contains information on development of components with their internal modules.

1.4 Structure of the Document Document has several sections to concentrate on different aspects of the system development plan:

• Section 2 is concentrated on System Development plan’s general rules of operation.

• Section 3 dedicated to the detailed explanations of the system components’ development plans in three parts: Peer, Server and Network components.

• Section 4 is dedicated for a summary of what has been explained throughout the document

• References and appendices are placed at the end of the document

ROMEO WP6 /68

2 SYSTEM DEVELOPMENT PLAN In this section following items will be explained for proper development throughout the project:

• General System development Plan.

• General Integration Completion Procedure.

2.1 General System Development Plan There are three main parts in ROMEO project. These are:

• Server

• Peer / User Terminal

• Network-related modules

ROMEO system development contains development of these three parties. Development plan is divided into integration steps. This will enable system to be integrated periodically. Each partner is supposed to define outputs for each integration step. As a result all partners will be aware of the connections between their components and other components. Integrations are defined in correlation with workshops of ROMEO project.

2.1.1 Integration Steps Table-1 shows integration plan. It is based on the linkage with critical milestones and deliverables of ROMEO project. There are some modifications from deliverable 6.1. Project team has agreed to execute an intermediate integration phase before second workshop. This phase is added as Phase 1.5 into the table

ROMEO WP6 /68

Task Name Start Finish Explanation

MILESTONES 31.03.12 30.09.14

Draft first year annual project progress report for project review

30.09.12 30.09.12 Integration Phase 1 Milestone

Phase 1.5 Integration on Munich 11.03.13 15.03.13 Integration Phase 1.5 Milestone

Second workshop on ROMEO Technology

08.07.13 10.07.13 Workshop Integration Milestone

Draft second year annual project progress report for project review


First attempt to integrate early prototype system components


DELIVERABLES 30.10.11 30.09.14

First report on system components development plan

30.06.12 30.06.12

Report on the first ROMEO workshop 30.09.12 30.09.12 Integration Phase 1 Milestone

Report on developed final system components

04.06.14 04.06.14 Phase 3 Dev. Milestone

Demonstrator 31.07.14 31.07.14 Integration Phase 3 Milestone

SYSTEM COMPONENTS DEVELOPMENT PLAN 01.03.12 30.07.14

System Integration Phase 1 01.03.12 08.08.12

Phase 1 Integration 09.08.12 12.09.12

System Integration Phase 2 13.09.12 14.08.13 Integration Phase 1 Milestone


System Integration Phase 3 26.09.13 14.05.14 Integration Phase 2 Milestone


FINAL SYSTEM INTEGRATION 05.06.14 30.07.14 Integration Phase 3 Milestone

SYSTEM INTEGRATION Milestone 30.07.14 30.07.14

Table 1 – Component Development Plan

2.1.2 Integration Phase Relations with Milestones and Deliverables ROMEO development contains several iterations and these iterations are planned to cover critical milestones and deliverables of ROMEO project. Table-1 is dedicated to show this correlation between development plan, critical milestones and deliverables.

2.2 General Integration Step Completion Procedure Each integration phase is divided into two main sections:

• Component Development Time

ROMEO WP6 /68

• Integration Time

Components will be developed in development time and integrated in integration time. Each component has its own nature of development. Thus every partner is supposed to define their own methods within this document in Section 3. Releases will be uploaded to the configuration management system according to the defined folder structure.

There are three integration phases for ROMEO project. Each integration phase is completed with its corresponding milestone. Every partner is supposed to deliver planned outputs before integration time start date. Final integration of ROMEO platform starts when these three phases are completed. Final integration will be ROMEO system demonstration.

2.2.1 Component Release and Test Procedure Integration phase ends with tests and approvals. Each partner submits their modules after development time.

• Each release is supposed to have a release note [2]

• Each component is supposed to satisfy basic set of tests for integration [2]

• Every component has to be released after testing internally by the developing partner’s people.

2.2.2 Bug / Change Request Release Mechanism Integration phase ends with tests and approvals. During project component development some changes may occur or some bugs can be detected. These issues are supposed to be logged properly. ROMEO partners supposed to solve the bugs or update the module specification according to the change request. Each change request needs a detailed review by related partners to accept or deny the request.

ROMEO WP6 /68

3 COMPONENT DEVELOPMENT PLANS This section is dedicated to the components’ development plans in detail. Every sub section is in direct correlation with deliverable 2.2. In section 3.1 user terminal components’ development plans and in section 3.2 server components’ development plans are explained. Each component has defined their released features for specific integration phase.

3.1 User Terminal / Peer Development ROMEO platform has several end users. These are:

• Fixed

• Portable

• Mobile

Each party is expected to receive one form of 3D content. 3D content is delivered through:

• DVB

• IP

networks. Each type of terminal will have its own parameters to enable user to get best quality. There are several components to be developed for user terminal for different users. Following sections are dedicated to list the plans for these components

The test receiver platform is also capable of receiving, demodulating and decoding the DVB signal. In addition, it provides a multitude of Quality-of-Service (QoS) parameters that can be used to evaluate the system performance and its limitations. These QoS parameters are DVB-related and IP-related so that both main input signals can be measured and monitored.

3.1.1 DVB Reception

3.1.1.1 Test terminal: The DVB receiver of the test receiver platform is based on an implementation that can receive, demodulate and decode DVB-T, DVB-T2 and DVB-T2 Lite signals. The implementation allows access to the different components of the MPEG2 Transport Stream, i.e. the Elementary Streams for video and audio, but also to the data in the Service Information, Data Carousels and all tables.

3.1.1.1.1 Related Description of Work Tasks Task 6.4, Task 6.5.

3.1.1.1.2 Detailed Module Diagram of Component The block diagram of the test terminal in Figure 1 gives an overview of the state of the work by indicating which building blocks have been tested and evaluated (green), which are currently under development (yellow) and for which blocks the development work had not started yet. (grey).

ROMEO WP6 /68

Figure 1 - Block diagram of the Test Terminal (green: tested and evaluated, yellow: under

development, grey: not started yet)

3.1.1.1.3 Brief Module Explanations The block 'DVB-T/T2/T2 Lite frontend' has been tested for stationary, portable and mobile reception. The encapsulation of TS packets into IP packets for transport over an internal interface between test receiver and notebook is also working. The 'IP Frontend' is a placeholder which will be replaced by a WLAN or LTE frontend at a later stage. The current development work is focused on the 'Internal IP Interface' which allows for the transfer of all data and streams between the test receiver and the notebook, and the 'Synchronisation Engine' which synchronises the video/ audio streams from DVB and the broadband network. The dimensioning of the two buffers is carried out when the details of the properties of the streams that carry the DVB views and the P2P views, are known. The video players are provisionally implemented; changes are likely to be necessary.

3.1.1.1.4 Module Development Plan The blocks for the DVB reception path are planned to be fully operational at the beginning of the 6th quarter of the project, in-time for the preparation of the intermediate demonstrator.

The blocks '3D Decoder' and the buffers are likely to become available during the 8th quarter of the project.

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24

Development of sync engine and related sub-modules

Validation of sync engine etc.

Implementation 3D decoder

Validation of long sequence TS

Table 2 - Development plan for DVB reception module in test terminal

ROMEO WP6 /68

3.1.1.2 Fixed and Portable Terminal Development

3.1.1.2.1 Related Description of Work Tasks Task 5.1, Task 5.3, Task 6.4, Task 6.7

3.1.1.2.2 Detailed Module Diagram of Component

Figure 2 – DVB Reception component in fixed and portable terminal

3.1.1.2.3 Brief Module Explanations DVB reception component is going to work as a front end block for peer terminal. RF signal will be prepared as stream in the network for synchronisation component to demultiplex. DVB reception component will have the following modules: Demodulator, TS Packetiser.

3.1.1.2.3.1 Demodulator RF signal from DVB broadcasting system is going to be demodulated for extracting transport stream. Transport stream data is going to be delivered at a specific DVB frequency. Demodulator module will tune to this frequency according to user request from user interface component. Secondly it will demodulate the RF signal and send Transport stream data to TS packetiser

3.1.1.2.3.2 TS Packetiser TS packetiser is going to package transport stream data for synchronisation component. Since there will be transport stream from both DVB and P2P synchronisation block needs similar front end interface for receiving transport stream.

3.1.1.2.4 Module Development Plan

M13 M14 M15 M16 M17 M18 M19

Demodulator

TS Packetiser Table 3 – Component Development Plan

3.1.1.3 Mobile Terminal Development

3.1.1.3.1 Related Description of Work Tasks Task5.2, Task 5.3, Task 6.4

ROMEO WP6 /68


Figure 3 – DVB Reception component in Mobile Terminal Device

3.1.1.3.3 Brief Module Explanations In order to receive DVB-T signal in Mobile Terminal Device a DVB-T2 receiver is needed. Chosen model is “PCTV nanoStick T2” [3].This module is connected to the device via internal USB interface. Low level drivers and SW support are specially developed for Mobile Terminal Device. The output stream is transferred to other ROMEO module – TS-Packetiser. After that the data are sent to Synchronization unit. The module is controlled by User Interface.


M16 M17 M18 M19 M20 M21 DVB module integration

TS Packetiser Integration

Table 4 – Component Development Plan

3.1.2 P2P

3.1.2.1 Internet Resource and Admission Control Subsystem (IRACS) At the peer side, IRACS defines the Resource Controller (RC) module to be used to enforce control decisions on nodes inside the network. The decisions are made by another module of IRACS, namely the Resource and Admission Manager (RAM), which resides at the server side and is described in subsection 3.3.3.1. Further details on the development plan of this module are provided in subsection 3.3.3.2

3.1.2.2 Topology Controller This component is responsible for the communication with server to join the P2P overlay. Since a central tree management approach is followed for ROMEO project, most of the topology construction related tasks are performed at server side. However, in the peer side, the communication related tasks for joining or leaving the overlay and other messaging for the peer placement in the topology (like parent change messaging, informing the server about a disconnected parent etc.) and connection establishment task with the parents or children are included.

3.1.2.2.1 Related Description of Work Tasks Task 6.3

ROMEO WP6 /68


Figure 4 - Block diagram for Topology Controller

3.1.2.2.3 Brief Module Explanations This module includes Communication Interface, Tree Connection Manager and QoS Manager sub-modules as in Figure 4 and the messages corresponding to these sub-modules will be implemented within the development phase of this module. This module interacts with Topology Builder and IRACS-RAM at the server side, with P2P NMS and UI&Control modules at the peer side and with Topology Control module of other peers.

3.1.2.2.4 Module Development Plan Since this module is mainly designed for handling control messages related to P2P topology construction and maintenance, hence, only messaging functionality is planned to be implemented within the scope of this module. Both internal and external messaging infrastructure will be implemented till the end of M15, which is the end of December 2012. While the testing period will be on the M16, the integration phase will be completed by the end of M18 including the integration efforts in the 4th GM meeting in Munich.

M14 M15 M16 M17 M18

Msg implementation for interaction with the whole system Testing Integration with other components

Table 5 - Development plan for Topology Controller module

3.1.2.3 P2P Depacketisation This component is responsible for the decapsulation of the chunks received over P2P network generated by P2P Packetisation component of the server.

3.1.2.3.1 Related Description of Work Tasks Task 5.1, Task 6.2

3.1.2.3.2 Detailed Module Diagram of Component Packetisation component does not have a sub-module. It performs all of its operations in the same module. Interaction diagram for depacketisation module shows the interaction of packetisation component with the other components.

ROMEO WP6 /68

Figure 5 - Interaction diagram for depacketisation module

3.1.2.3.3 Brief Module Explanations This module gets P2P packets as input over P2P network and decapsulates them. It also interacts with Decryption module and provides it encrypted MPEG2-TS packets to be decrypted before sending them to Synchronisation component.

3.1.2.3.4 Module Development Plan Since this is an essential module for P2P data transfer, it is planned to be finalized soon. In order to verify its full-functionality, it is necessary to have P2P packets so implement P2P packetisation functionality. As can be seen from the table below, it is planned to implement the ROMEO system messages within the following two months. Starting from month 15, it is planned to work with P2P packets containing MPEG2-TS encapsulated NALUs which is the output of content generation part of ROMEO project. Testing of this component will be performed in M15 and in the following months it is planned to integrate it with the whole system.

M13 M14 M15 M16 M17

Msg implementation for interaction with the whole system Packetisation of ROMEO data provided by US and R&S Testing Integration with other components

Table 6 - Development plan for depacketisation component

3.1.2.4 P2P Chunk Selection P2P Chunk Selection module implements the process of selecting the video chunks to be forwarded to the children peers. The intelligence behind P2P chunk selection lies in the prioritization of more important chunks given the current overlay topology and the network status. In addition, the Chunk Selection module will include a failover mechanism in order to alleviate the impact of churn.

3.1.2.4.1 Related Description of Work Tasks Task 6.1, Task 6.3, Task 7.1


Figure 6 - Block diagram of the interaction between Chunk selection and other modules

ROMEO WP6 /68

Figure 7 – Block diagram of the Chunk selection module

3.1.2.4.3 Brief Module Explanations The basic operations executed by the Chunk selection module will: a) decide which chunks will be forwarded to children peers, b) report missing chunks to parent peers, c) request missing chunks from parent peers in case of temporal disconnection due to churn, d) request the initiation or termination of the forwarding of certain types of chunks from the parent peers. The Chunk selection module will communicate with the P2P Transmission/Reception module in order to send or receive control messages. It will communicate with the Topology Controller in order to receive information about the parent and children peers. Inside the Chunk selection module, the Decision maker will: a) make decisions regarding chunk forwarding and chunk requests, b) handle incoming control messages from other peers and c) send control messages to other peers. The Topology information will handle information about current parent and children peers retrieved from the Topology Controller.

3.1.2.4.4 Module Development Plan During months 13 and 14 the Communications interface will be developed. This interface will handle JSON message exchange with other models. Dummy modules will be used to evaluate these operations. During month 15, a first version of the Chunk selection module will be provided and put into testing procedures. Integration with other modules is planned for months 16 and 17.

M13 M14 M15 M16 M17 Msg implementation for interaction with the whole system First versio of Chunk selection module Testing Integration with other components

Table 7 – Development plan for the Chunk selection module

3.1.2.5 P2P Transmitter & Receiver This component is responsible for encapsulating the network related calls for message and content delivery over the IP network. All of the transmission and reception operations should be performed over this component.


ROMEO WP6 /68


Figure 8 - Block diagram of P2P Transmitter/Receiver

3.1.2.5.3 Brief Module Explanations This module contains sub-modules for TCP and UDP transmission / reception and interacts with the corresponding P2P Transmitter/Receiver modules of other peers and/or P2P Transmitter module of the server. While the content is planned to be delivered through UDP, the control messages is planned to be sent through TCP.

3.1.2.5.4 Module Development Plan The API of network related calls for message and content delivery will be implemented till the end of M13, which is the end of October 2012. While the unit tests will be performed on the M14, the integration phase will be completed before the 4th GM meeting, by the end of M17.

M13 M14 M15 M16 M17 API implementation for data transfer between computers Testing Integration with other components

Table 8 - Development plan for P2P Transmitter/Receiver module

3.1.3 Video Decoding This component has several purposes:

• Forwarding H.264/SVC elementary stream packets (NALUs) to separate decoding instances and decoding them in real time

• Merging decoded descriptions in real time and supplying raw video frames to the video renderer

• Extracting view-interpolation related metadata from the received data and carry out necessary decoding operation to pass the decoded metadata directly to the video renderer

• Extracting visual attention metadata from the received data and forwarding it directly to the video renderer

3.1.3.1 Related Description of Work Tasks Task 4.1, Task 6.4, Task 7.2

ROMEO WP6 /68

3.1.3.2 Detailed Module Diagram of Component

Figure 9 - Block diagram for decoder module

3.1.3.3 Brief Module Explanations Video decoder block in the user terminal is composed of two major building blocks: SVC decoder instances and multiple description merging instances.

3.1.3.3.1 SVC decoder SVC decoder is able to decode the NAL unit streams coming through the demultiplexing and decapsulation module. Each SVC decoder instance acts as a description decoder. It will be based on the OpenSVC [7] implementation. Expected input to each SVC decoder instance is a packet stream (UDP or TCP) consisting of NALU packets (SPS, SSPS, PPS, SEI messages, prefixes and coded slice NALUs) preceded by the corresponding presentation time stamp (PTS) extracted from the PES header. For each video frame or access unit that is identified with a particular presentation time stamp, base layer and enhancement layer coded slice NALUs need to be aggregated before sending to the SVC decoder. The output is the reconstructed video description (YUV4:2:0 format), or other decoded metadata that will be sent directly to the video renderer.

3.1.3.3.2 Multiple description merger This module receives as input the reconstructed description pairs from a group of SVC decoders and does processing in pixel domain by interleaving the vertical pixel groups of both received descriptions. The outputs are a couple of HD video frames, since each description is represented as a side-by-side combination of two HD video frames. The output video frame streams will be sent to the video renderer using TCP, or will be stored in a shared memory space with the video renderer in order to let the renderer fetch decoded frames.

3.1.3.4 Module Development Plan The first year of the project was dedicated to the specification of this module. The sub-modules have been defined and their interaction described in deliverable D2.2. The API of input NALU stream communication to the SVC decoder instance will be implemented by the end of M16. The API of output raw video frame communication with the descriptions merger will be implemented by the end of M17 along with the functional real-time SVC decoder capable of decoding two-layer HD stream at 25 fps. The first version of a working SVC decoder platform (excluding the description merger) is expected to be ready in M18 (end of 6th Quarter). As far as the integration with the video renderer is concerned, single description decoded outputs can directly be fed to the video renderer for testing.

Video Decoder

Demultiplexing and

Decapsulation

SVC decoder

Video Renderer

Multiple description merging

13011302

1302SVC decoder

SVC decoder

SVC decoder

SVC decoder

SVC decoder

SVC decoder




1305

SVC decoder

ROMEO WP6 /68

M13 M14 M15 M16 M17 M18 M19 M20

Open SVC – API implementation for NALU stream acquisition from network

Open SVC – API implementation for raw video frame transfer to the renderer

Open SVC – real-time decoding of 2-layer HD video at 25 fps

Description merging block Video decoder controller & messaging block

Table 9 - Development plan for Video Decoder module

3.1.4 Video Rendering The video renderer will receive incoming video and disparity maps from the video decoder. A maximum of 4 video and 4 disparity maps can be handled to generate the output video. Different display technologies can be addressed including multi-view auto-stereoscopic displays and stereoscopic displays. The video renderer will be able to address these displays providing the right video shuffling.

Multi-view displays are requiring a variable number of views from 2 and up to 28 (regarding the list of displays available among partners). The video renderer will then take incoming videos and their associated disparity maps to create new views in between existing ones using Depth Image Based Rendering (DIBR) techniques. The relative positioning of interpolated views compared to incoming ones as well as the range of disparity will be carefully managed to ensure an optimized quality of experience.

3.1.4.1 Related Description of Work Tasks Task 3.4, Task 6.4


Figure 10 - Video Rendering block diagram

3.1.4.3 Brief Module Explanations The video rendering block will provide the adapted video signal to the targeted display. It will handle the number of views required for the specific display unit. The “disparity map projections” and “View Interpolation” blocks are the computing blocks taking as input incomings disparity maps and video and generating interpolated views. The “Video Shuffling” block adapts the interpolation to the targeted display. The “Video renderer Control” converts user requirements and network conditions into video rendering parameters.

ROMEO WP6 /68

3.1.4.4 Module Development Plan The first year of the project was dedicated to the specification of this module. The sub-modules have been defined and their interaction described in deliverable D2.2. The specification of the architecture has been also finalized. It is mainly integrated into a General purpose GPU processor.

During the second year, the development of the CUDA code to achieve the video rendering part will be done. A first version of working software is expected at the end of the 6th quarter. The integration with the video decoder should start during the 7th quarter. It should lead to a common demonstrator (video decoder + video renderer) at the end of the second year.

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24

Development of CUDA code

Integration with video block

First video decoder + rendering demonstration

Table 10 - Development plan for the video renderer

The third year will be dedicated to the integration in the global demonstrator including all dedicated adaptation functionalities.

3.1.5 Audio Decoding This component is aimed at decoding back the compressed audio bit streams into mono, stereo or multichannel audio signals. The descriptions in this section will be closely related to the descriptions in the audio encoding section.



3.1.5.2.1 Multichannel Audio Decoder In Romeo, there are three different types of terminals – namely the fixed terminals, the portable terminals and the mobile terminals. The audio decoder module is a component resident on each terminals. It is responsible to decode audio bit stream compressed in AAC-compatible format into raw audio signals suitable for mono/stereo/multichannel rendering. Expected input to the audio decoder is a packet stream consisting of compressed audio data and corresponding synchronisation information (PTS and/or PCR) from the audio stream depacketizer submodule. A detailed diagram of the audio decoder module is shown in Figure 11

Figure 11 – Diagram of Audio Decoder Module

ROMEO WP6 /68

3.1.5.2.2 ABS Spatial Audio Decoder The diagram of the ABS Spatial Audio Decoder is shown in figure below.

Figure 12 - Diagram of ABS Spatial Audio Decoder

3.1.5.3 Brief Module Explanations

3.1.5.3.1 Audio Stream Receiver Sub-module The audio stream receiver sub-module always listens on a specified UDP (or TCP if required) port for incoming audio packets and put the newly received data (including PCR message for A/V synchronisation) into the receiving buffer.

3.1.5.3.2 Buffer Management Sub-module The received compressed audio stream data along with PCR message are first stored in the receiving buffer, which is managed by the buffer management sub-module. The output buffer of the audio decoder is also under the management of this sub-module. The buffer management sub-module is actually the interface between the decoder and the renderer, where the renderer gets access to the decoder’s output data from another thread.

3.1.5.3.3 AAC-compatible Audio Decoder The AAC-compatible Audio Decoder sub-module takes data from the receiving buffer, decodes the data into channel-based audio frames and put them into the output buffer for rendering.

3.1.5.3.4 T/F Transformation The encoding process will be carried out in the sub-band frequency domain. Thus the T/F Transformation module is used, for each audio channel, to decompose the time domain audio signal into frequency sub-band domain.

3.1.5.3.5 OTT-Modules One-to-two (OTT) module, as used in MPEG Surround to up-mix a single audio channel into two channels, is applied. A number of OTT modules will be applied to up-mix a single audio channel into multiple channels. For instance, to up-mix a single channel into 5 channels a total of 4 OTT modules may be needed.

3.1.5.4 Module Development Plan The core functionality of the multichannel audio decoder will be available from M16 with further improvement and optimisation in the following months. The ABS spatial audio decoder will also be available from M16. Detailed development plan for the audio decoder related sub-modules is shown in development plan table.

ROMEO WP6 /68

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24

Audio Stream Receiver

Buffer Management

AAC-compatible Audio Decoder

Closed-loop R-OTT Module

T/F Transformation

ABS optimisation algorithm

Table 11 - Development Plan for Spatial Audio Decoder

3.1.6 Audio Rendering This component accepts the decoded audio signals and delivers them to the users according to the control messages from the user and from the adaptation module, through various processing algorithms corresponding to the proposed configurations within the project.

3.1.6.1 Related Description of Work Tasks Task 3.4, Task 3.5, Task 4.2, Task 6.4


Figure 13 - Audio Rendering Module Diagram


3.1.6.3.1 Audio Object Description This module provides the Audio Description Language, which is sent to both the portable and the fixed renderer.

3.1.6.3.2 Decoded Audio Streams The already decoded audio streams are delivered to the different audio renderers. This can be 5.1 surround audio or single audio objects.

3.1.6.3.3 Additional Metadata This module sends additional metadata (e.g. headtracking information) to the mobile renderer.

ROMEO WP6 /68

3.1.6.3.4 Portable Device / Object-based Renderer This renderer is used on the portable device and will render the object-based audio.

3.1.6.3.5 Fixed Device / Spatial Renderer This renderer is used on the fixed device and will render the object-based audio with a multi-channel spatial renderer and the simplified WFS system respectively.

3.1.6.3.6 Mobile Device / Binaural Renderer This renderer is used on the mobile device and will render the 5.1 surround sound for binaural listening audio. The renderer will be controlled by the headtracking information of the user.

3.1.6.3.7 Headphone / Stereo Playback This module is the portable user’s headphone or stereo loudspeaker.

3.1.6.3.8 Large Number of Loudspeakers This module is a large number of loudspeakers required for multi-channel spatial audio rendering.

3.1.6.3.9 Headphone Playback This module is the mobile user’s headphone.

3.1.6.4 Module Development Plan The API for the mobile renderer will be available respectively finished in M23. The portable renderer will be implemented by M24. The fixed terminal multi-channel spatial renderer will be available in M24.

M16 M17 M18 M19 M20 M21 M22 M23 M24

Mobile Renderer

Portable Renderer

Fixed Renderer

Table 12 - Development plan for audio rendering modules

3.1.7 Synchronisation This component has two purposes: Synchronizing transport stream inputs from P2P network and DVB, synchronizing collaborative partners to receive content at the same time.


ROMEO WP6 /68


Figure 14 – Synchronisation Component Detailed Block diagram

Synchronisation component is a software component that is to be placed on peer terminals. It is responsible for receiving transport stream data from DVB and P2P and deliver to video and audio decoders. ROMEO has three types of terminal; fixed, portable and mobile. Synchronisation component’s structure will be different in these terminals. Fixed terminal, which is capable to receive multi-view 3D content is supposed to display all views. On the other hand portable and Mobile will be able to display only stereoscopic 3D content or one of the views with its depth map view..


3.1.7.3.1 Stream Input Interface – DVB DVB stream input will be delivered from DVB reception component. It is planned to send DVB-TS packets in IP packets as in P2P. This method will be used for unification of stream input ports of synchronisation component. Chunk structure of DVB-TS packets will be similar to P2P structure that is defined in ROMEO deliverable 5.1[4]. Stream Input Interface module delivers transport stream data to PCR Clock extraction block for reference clock generation. After clock extraction, transport stream is processed in demux block for audio and video separation.

3.1.7.3.2 Stream Input Interface – P2P P2P interface is going to deliver video and audio content of the side views and depth maps. Thus, content will enable end users to have more qualified 3D experience over terminals. Since display capabilities of fixed, portable and mobile terminals are different, user experience over the terminals will be limited to these capabilities. P2P chucks will be depacketized in P2P component. P2P streams may be encrypted on transmission side. As a result there will be a decryption module to prepare encrypted data for P2P input interface module. P2P input interface will also extract transport stream data from IP packets. Since ROMEO platform will use DVB clock as reference, transport streams from P2P side will directly be directly sent to demux block.

ROMEO WP6 /68

3.1.7.3.3 PCR Extract Block DVB transport stream has defined packet structure. Every transport stream packet has the header and a specific part of it is dedicated to sore PCR information.

Figure 15 – PCR information in transport stream

Demultiplexing all transport stream into elementary streams takes time. PCR information is planned to extract before demultiplexing to generate proper timing for all streams and other parts of ROMEO platform. Each transport stream packet will be examined for PCR data and delivered to demux block afterwards.

3.1.7.3.4 Demux Block DVB transport stream data consists of audio, video and metadata parts. Demux block is planned to separate these data parts. ROMEO platform has P2P and DVB transmission media. This will cause time differences in receiving packets at every peer terminal. As a result it is required to have a buffer mechanism to guarantee quality of service in smooth streaming. Demux Block is expected to separate each stream that has been received by input interface modules and deliver audio data to audio buffer and video data to video buffer.

3.1.7.3.5 Audio/Video Buffer ROMEO platform is organised such that DVB is always available and P2P is added to DVB whenever possible. Other views then DVB will let user to experience multi-view 3D content. Video buffer module is planned to collect all video packets properly.

Demultiplexed audio data will be also stored to keep smooth streaming. Audio data will not be as much as video, on the other hand it has multiple components to be separated for different channels. Synchronisation component’s main responsibility is to extract audio data and deliver to audio decoder.

Audio and video buffer shall be able to keep following memory area for convenient operation

• 1 stereoscopic view, 4 colour views and 4 depth map views (totally 9 views) • Bit rate of each stream is about 7.5 mbps • At least 50 seconds of video storage to keep dynamic operation for each view

This calculation roughly requires 0,5 GB memory space for proper operation

ROMEO WP6 /68

3.1.7.3.6 PCR Sync Block Whenever audio and video data are ready for decoders, system shall deliver these data at the same time. Synchronisation component has reference clock from PCR extract block. PCR sync block will use this reference and deliver audio and video elementary streams that have same IDs at the same time.

3.1.7.3.7 Stream Output Interface – Audio / Video Decoders of ROMEO platform receive their data from stream output interface. It is basically a UDP port to send audio and video elementary streams. Packets will be sent to decoders with time stamps that are set in sync block.

3.1.7.3.8 Communication Interface Synchronisation component communicates with other ROMEO components with communication interface module. This module receives JSON messages and decodes them. Decoded JSON messages are processed in proper modules to generate required data. Inter-module communication will be also through communication interface to have unified messaging interface.

3.1.7.3.9 Collaborative Synchronisation Module ROMEO platform has collaboration feature between users. This requires a synchronisation of users on receiving data at the same time. Collaborative synchronisation module checks for the delay of the peer and let network be aware about its capacity to receive 3D content. If the peer is fast to receive content on time, this means short delay in overall system. If the peer is unable to receive data on time, collaboration should be set according to this slow user. Collaboration synchronisation module checks the slowest peer at the beginning and let other peers to adjust themselves according to this slowest peer.

3.1.7.4 Module Development Plan Synchronisation component will be developed under following plan stated in synchronisation component development plan table. .It has 4 milestones: Demultiplexing of standard DVB transport stream, integrating with DVB and P2P front end, Collaboration and buffering completion, Component development completion.

ROMEO WP6 /68

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27

Stream Input Inerface - P2P

Stream Input Inerface – DVB

PCR extract Block

Demux Block

Audio/Video Buffer

PCR Sync Block

Stram Output Interface Audio / Video

Communication Interface

Collaborative Synchronisation

Table 13 – Synchronisation component development plan

3.1.8 Audio-Visual Communication Overlay Audio/Visual Communication Overlay component on peer terminal will be responsible for constituting an A/V communication channel between the local user and the users that experience the same 3D content.



Figure 16 – A/V Communication Overlay block diagram

ROMEO WP6 /68

3.1.8.3 Brief Module Explanations A/V Communication Overlay component on peer terminal aims to enable the local user to communicate with collaborative remote users when the collaborative group enjoy the same 3D content. That aim can be achieved by developing five modules for client side of the system.

3.1.8.3.1 Conference Controller Conference Controller module on a client is responsible for carrying out the operations that a collaborative user may need via communication with Conference Controller module on server. Those operations might be logging into server, logging out from server, joining a conference room, starting a conference session, ending a conference session and some other required user operations for audio/visual streaming.

3.1.8.3.2 A/V Content Receiver A/V Content Receiver module on a client is responsible for collecting the A/V streams of remote users to show on renderer later.

3.1.8.3.3 A/V Content Distributer A/V Content Distributer module on a client is responsible for delivering the user-specific A/V content to server.

3.1.8.3.4 Synchronisation Controller ROMEO platform aims to synchronise the content that users display on their screen. Every user in a conference session needs to receive 3D content simultaneously in order to enjoy good collaboration quality. Therefore, synchronisation controller on a client is mainly responsible for carrying out those operations about synchronisation. For example, if the delivery of the content delays over the collaborative network, the content to be displayed later is buffered by the Synchronisation Controller module.

3.1.8.3.5 Overlay Manager Overlay Manager on a client is going to check the display of audio-visual data in correlation with 3D data. The module will be in close relation with rendering and receiver part to display communication data with 3D content data simultaneously.

3.1.8.4 Module Development Plan Modules will be developed in parallel. On the other hand, synchronisation and conference controller modules are expected to last till integration with other components of the ROMEO platform.

ROMEO WP6 /68


Peer Conference Controller A/V Content Distributer A/V Content Receiver Synchronisation Controller Overlay Manager

Server Conference Controller A/V Content Distributer A/V Content Receiver

Table 14 – A/V Communication Overlay component development plan

3.1.9 Audio-Visual Adaptation and Network Monitor The audio-visual adaptation module is responsible for adapting the content to:

• the user requirement • the network conditions • the visualisation/hearing environment.

The video adaptation module will take into account the user requirement in term of selected viewpoint. This parameter will directly interact with the video renderer block to ensure view interpolation is applied the right way.

Network conditions will impact the video decoder as well as the video renderer. Enhanced layer could be transmitted or not but it can also happen that base layer of satellite views will not be transmitted as well. The video renderer will then have to adapt its processing to effectively transmitted information.

Then an adaptation will be required to properly address the end-user display. Several multi-views displays are addressed during this project. They are not requiring the same video shuffling at the end of the video rendering block. General information about the visualization environment will be then used in this video adaptation block.

The Network Monitoring Subsystem (NMS) is a software module residing in every peer (and super-peer) that periodically collects network information regarding the status of the network (such as delay, jitter and bandwidth). This information is then provided to the Topology Builder/Multicast Tree Manager (server or super-peer) in order to perform informed decisions on the multicast trees (ACTIVE, READY, and SLEEPING status). More specifically, in a P2P environment the NMS of a given peer will receive NMS’ data from its children peers, in the

ROMEO WP6 /68

selected multicast tree it will aggregate the received data with its own and then send the result to the NMS of its father (peer), see ROMEO D5.1 [4] and D6.2 [5] for more details. This process goes on until the data gets to a super-peer (or server in small environments). In addition, the Topology Builder can command the NMS of a peer to start collecting data or to connect with the NMS of another peer to collect data on a P2P link.

3.1.9.1 Audio-Visual Adaptation



Figure 17 - Audio-Visual Adaptation block diagram

3.1.9.1.3 Brief Module Explanations Both audio and video processing must be adapted to:

• The user requirement (user interface): e.g. the user can decide the free viewpoint he would like to have

• The user environment: e.g. the end user display/sound system can require a specific processing that has to be anticipated.

• The network conditions must be monitored to understand at the end of the chain the level of information that could be used. Fall-back modes must be defined in case some part of the full video + disparity maps streams are missing.

3.1.9.1.4 Module Development Plan During the first year of the project, some adaptation scenarios have been defined. They will be described in detail in the D3.4 deliverable. The adaptation of the video renderer has also been studied and taken into account in the specification.

During the second year, the specific video processing for video adaptation will be developed. A first display dependent processing will be available end of the 6th quarter. A free viewpoint requirement will be possible on demand at the end of the second year.

A full integration of user requirement (user interface) will be developed during the third year and available for the final demonstrator. The network conditions will be also integrated as an input parameter to the adaptation processing.

ROMEO WP6 /68

Audio adaptation:

The multichannel AAC decoder currently operates with the fundamental downmixing feature configurable by the user. Development will follow from M13 to M18, to implement functions of the audio adaptation to user configurations of the rendering environment. Information from the network monitoring module will be considered during the adaptation function development.

When the free viewpoint is changed by the user, audio will have to be adapted to the new viewpoint accordingly. This function will have to be included in the audio adaptation code in the first phase of the A/V adaptation development and be integrated with the video adaptation of free view point in the second phase (M19-M24). A full integration of user requirement (user interface) will be developed during the third year and available for the final demonstrator. The network conditions will be also integrated as an input parameter to the adaptation processing.


Development of AV adaptation code

Integration of Free view point

Full integration: UI + NMS

Table 15: Development plan for the audio-video adaptation

3.1.9.2 Network Monitoring Subsystem



Figure 18 – Block Diagram for A/V Adaptation & Network Monitor

ROMEO WP6 /68

Figure 19 – NMS interactions

3.1.9.2.3 Brief Module Explanations � Collector&Report - collects three types of information: (i) peer hardware capabilities;

(ii) peer stability and; (iii) peer network information regarding specific peer connections - towards a parent, a child or a super-peer. More details on these functions is given in ROMEO D5.1 [4]. Following what was defined in D2.2 [1], and as depicted in Figure 19, at a specific point in the network the NMS data report can be an aggregation of children NMS data plus parent’s NMS data. A complete description of the NMS data structures is provided in Deliverable 6.2 [5].

� Monitoring Agent - passively collects connection statistics, either by monitoring connection related information, such as connection history, start time and duration or by exploiting the libpcap [6] library, which enables implementation-independent access to the underlying packet capture facility provided by the operating system. This information can be used, for example, to compute the peer’s stability, as explained in ROMEO D5.1 [4]. All monitored information is then gathered by the Collector&Report sub-module.

� Link tester – provides link testing functions to determine link characteristics such as packet loss, jitter and delay. It also allows measuring the download and upload link capabilities. The results are then passed to the Monitoring Agent.

� Communications interface - is responsible for all communications related to either configuring the sub-modules or sending the indicated control messages to other components/modules of ROMEO platform. This block also listens on specific TCP and UDP ports, and acts both as a NMS server and client.

3.1.9.2.4 Module Development Plan This module has an estimated development plan of two months. 1 month for implementation, 1 for testing and 2 for Integration.

M15 M16 M17 M18 M19 M20

Algorithm Development

Msg implementation for interaction with the whole system

Testing Integration with other components

Table 16 - Development plan for NMS component

ROMEO WP6 /68

3.1.10 User Interface and Control This Component is responsible from user interface of the ROMEO system. It has two aims: User interaction mechanism with user, System Diagnostic parameter adjustments. It is planned to develop the component in three integration steps. This section contains information about: Linkage with the tasks in description of work, Integration step development items and brief test conditions.


3.1.10.2 Detailed Module Diagram of Component User interface and control module is a software module that is located on each peer terminals. It is responsible for providing system features to be controllable over this interface, as well as user preferences, user choices and dynamic parameter can be adjustable via this module by interacting component of the module. Fixed, portable and mobile terminals’ interface modules will be different according to features planned to be supported.

Communication Interface

User Interface & Control

Message Encoder

Syncronization

Video Decoder

A/V Adaptation and Network Monitor

DVB Reception Module

Audio Decoder

Authentication, Registration and Security

User Generated Content1015/

1016

1017/

1018

1002/1003

1004/

1005

1008

1006/1007

1000/

1001

1012/1013

1014

A/V Communication Overlay

1009/1010

Video Renderer

Audio Renderer1011

P2P

Message Decoder

Message Listener

Queue Control

Message

Dispatcher

User Interface and

Control

Message Sender

Figure 20 – User Interface and Control Detailed Block diagram

ROMEO WP6 /68


3.1.10.3.1 Communication Interface Component Communication interface component is a common purpose software component that is composed of several software blocks in order to provide effective communication and interaction between all other modules. Message exchange will be through this component. User interface module communicates almost every other modules in asynchronous way, thus message traffic can be potential problem. To overcome such kind of communication problems, multi process structure will be used. Receiving each request and forwarding message to related module will be different processes.

3.1.10.3.1.1 Listener Block

This software block is a basically port listener process which listens User Interface module port and catches incoming JSON messages that are sent from the other modules. And also it forwards the message to the Message Decoder Block.

3.1.10.3.1.2 Message Decoder Block Message Decoder Block parses incoming message and checks whether it is correctly

formed according to predefined rules in deliverable D2.3. It forwards decoded JSON message to the Queue Control Block.

3.1.10.3.1.3 Queue Control Block Because of using different process for receiving and sending message, there shall be

communication interface between these two processes. The bridge between these two processes is provided by using “Queue” data structure. The logic behind the “Queue” data structure can be described FIFO (First In First Out). First incoming message will be processed firstly. Therefore, incoming message will be inserted into “Queue” via help of Queue control block after message decoder block finishes its work.

3.1.10.3.1.4 Message Encoder Block After requested message process finishes, Message Encoder Block forms outgoing

message according to predefined rules in deliverable D2.3. It forwards encoded JSON message to the Message Sender Block.

3.1.10.3.1.5 Message Sender Block

Message Sender software block’s aim is sending JSON message to modules. Sender mechanism is basically sends response or JSON message to ports of related modules.

3.1.10.3.2 User Interface Control Component Creation of items which user will interact and event handler mechanism are controlled via User Interface Control software. Received requests from interface will be matched corresponding functionality and will be forwarded Message Dispatcher software block in order to start implementation.

3.1.10.3.3 Message Dispatcher Component Message dispatcher software component is the core software block which includes definition of functions and implementations that are prepared for responding JSON messages or user interaction. User interface module communicates almost every module thus size of this software block depends on other modules requirements.

ROMEO WP6 /68

3.1.10.4 Module Development Plan


Comm. Interface-Listener

Comm. Interface-MessageDecoder

Comm. Interface-Queue Control

Comm. Interface-MessageEncoder

Comm. Interface-Sender

User interface Control

Message Dispatcher

Table 17 – User Interface Module development plan

3.1.11 Authentication, Registration and Security This component is responsible for the following tasks:

• Storing user rights • Decryption of the content for protecting from unauthorized users. • Key retrieval for the user generated data. • Signing content so that the source can be checked for integrity.



Figure 21 - Block diagram of Authorization Registration and Security component

3.1.11.3 Brief Module Explanations With the scope of this module, it is planned to implement Security and User REST APIs. The Security REST API will be implemented to create, delete or update public/private key pairs for each ROMEO user. The User REST API, on the other hand, will be implemented to deal with the user authentication and user rights.

ROMEO WP6 /68

3.1.11.4 Module Development Plan The APIs will be implemented in M14, November 2012. Following the testing period within M15, the integration phase will start to enable seamless interaction of this module with other ROMEO modules and end at the end of M18.

M14 M15 M16 M17 M18

The API implementation for Authentication, Registration and Security Module Testing Integration with other components

Table 18 - Development plan for Authentication Registration and Security component

3.1.12 User Generated Content

3.1.12.1 User Generated Content Capture The function of this module is to prepare user content suitable for transmission, store and sharing in ROMEO system. This includes 3D/2D video and image capture, download from Internet or other devices, using communications channels. “User Generated Content “module provide to “User Generated Content Upload/Download” module the content that should be uploaded to the server. The work in this module is primary concentrated to 3D/2D video and image capture, because most of other tasks related to downloading from Internet and using other communications channels are natively supported by mobile and user terminal devices.

3.1.12.1.1 Related Description of Work Tasks Task 5.5, Task 6.3, Task 6.4, Task 6.7, Task 6.8


Figure 22 – User Generated Content Capture Detailed Block diagram

3.1.12.1.3 Brief Module Explanations To capture user generated content in Mobile Terminal Devise is used build in stereo camera. The camera consists of two identical image raw sensors to be able to capture stereo frames. The sensor raw output is processed by Image Processing Pipeline to YUV frames. Depending on the shooting mode, the frames are encoded in H264/AVC format for video or JPEG format for still image capture. Then the content is stored in internal device memory. At the same time frames are displayed on auto-stereoscopic display. Image processing on the captured raw frames includes: Defect pixel correction, Lens Vignetting correction, Lens Geometrical Distortion Correction, Chromatic Aberration Correction, Green Imbalance Correction, Raw Noise Filtering, Color Filter Array Interpolation, Color Correction, Gamma Correction, Luma and Chroma Noise Filtering, Temporal and Spatial Video Noise Filtering, Stereo Image and Frame Alignment, Video Stabilization. Automatic Control algorithms – Auto Exposure, Auto

ROMEO WP6 /68

White Balance and Auto Focus are based on statistics data from both image sensors. The stereo camera is controlled by Camera Application using User Interface.


M13 M14 M15 M16 M17 M18 M19

HW production and bring-up

Image Pipe Implementation

Image Quality Tuning

Testing Table 19 - Development plan for User Generated Content Capture component

3.1.12.2 User Generated Content Upload / Download / Search and Discover This module is responsible for searching/uploading/downloading content to/from server. On the user terminal side, this module interacts with the “2D/3D User Content Capture” module to upload the content requested by the user and with the “Authentication, Registration and Security” module to sign packet to the content be uploaded. At the server side uploaded/downloaded content can be searched and downloaded.

3.1.12.2.1 Related Description of Work Tasks Task 5.5, Task 6.3, and Task 6.8


Figure 23 – Block diagram of user generated content

3.1.12.2.3 Brief Module Explanations It is decided to implement a REST Server/Client Architecture to implement User Generated Content functionality within the scope of ROMEO system. While Apache Server and ZEND PHP Framework will be implemented on the server side, Java based Apache HTTP Client will be implemented on the client side. This module interacts with the UI&Control module at the client side and Authentication, Registration and Security module both at the server and client side, hence, messaging functionality will be also implemented in the development phase of this module.

3.1.12.2.4 Module Development Plan The implementation of the REST based Server/Client architecture has already started. It is planned to be finalized including streaming the content between client and server together with

ROMEO WP6 /68

the messaging infrastructure for the interaction with other modules is planned to be finalized by the end of M14, which is the end of November 2012. By the end of M15, UGC module is planned to be tested and the possible issues will be addressed to make it ready for the integration. In the following three-month period, it is planned to spend efforts on integration of this module into ROMEO system.

M14 M15 M16 M17 M18

Msg implementation for interaction with the whole system Testing Integration with other components Table 20 - Development plan User Generated Content component

3.2 Server Development

3.2.1 Content Generation The content generation aspects in the ROMEO project are mainly covered by Task 3.1 [5]. The main achievement of this task is the production of raw multiview/spatial (video/audio) content that will be used then by the 3D encoding modules. In addition to the capturing itself, post-acquisition processes are required in order to eliminate as much as possible unwanted defects or artifacts introduced during the shooting.

Content Generation also includes depth map generation and visual attention modelling. Both are obtained from video content.

Jointly with the raw video views, depth maps are generated and encoded. The main utilisation of the depth image is to allow view interpolation for the multiview rendering. One depth map stream will be provided with each captured view and will be used by the renderer to increase the number of views from 4 or 8 [5] to the N views required by the auto-stereoscopic display.

Additionally, the depth maps can be used for:

• Monitoring: as described in [5], during the multiview acquisition, coarse disparity maps are computed (at low frame rate) in order to provide monitoring information on the 3D configuration of the scene. Depth and disparity are inversely related, thus it is easy to find the depth thanks to the disparity.

• Object based description: the depth content provide key information in order to enhance the description of a scene. Especially, a depth map can be used as a main input for object based segmentation. Thus, the depth content will be used in the visual attention module to improve the regions of interest partitioning.

The Visual Attention Module has two main goals in the ROMEO project. First it provides to the encoder information about interesting regions in the video. This information will be used to improve encoding in the form of saliency maps.

The second purpose is related to visual adaptation. The aim is to optimise the video to different screen sizes and aspect ratios by dynamically cropping and scaling it. This process considers the previously found interesting regions.


ROMEO WP6 /68


Figure 24 - Diagram of the Visual Attention Module

3.2.1.3 Brief Module Explanations • Disparity map computation:

This module provides to the global rating function a saliency map estimated from the disparity. Disparity map computation figure details the different blocks of this module. First, the calibration parameters, corresponding to the intrinsic and extrinsic parameters, have to be extracted and injected to rectification algorithm. The rectified multi-view content is used to extract the disparity maps.

Figure 25 - Detailed block diagram of the “disparity map computation” block

The main block is the one that estimate the saliency from the disparity map. The saliency estimation from disparity figure shows the different steps of this processing from the disparity to the saliency. This process take into account two step of preprocessing before the saliency estimation: first the disparity map has to be converted to depth map and afterward the result is quantized. Finally, the saliency is estimated with a method based on macroblock averaging.

ROMEO WP6 /68

Figure 26 - Detailed block diagram of the saliency estimation from the disparity

• Color based saliency detection:

This module provides a saliency map based on the spatial information (color, orientation & intensity). As it can be seen on color based saliency block figure, the processing is divided into two main steps: the extraction of features and the selection of the relevant information based on a winner take all mechanism (WTA).

Figure 27 - Color based saliency block

3.2.1.4 Module Development Plan Concerning the disparity map computation aspects, the major part of the work has been done in the Task T3.1. Additional developments and tests are still needed in order to improve the quality of the results. More particularly, the method used to extract the camera parameters is always in progress.

In the side of saliency extraction form spatial information (Color based saliency detection block), some experiments have been done and preliminary saliency maps will be used for the first developments of the global rating process.

M14 M15 M16 M17 M18 M19 M20 M21 M22 M23

Saliency estimation from disparity

Done

Color based saliency detection

Integration - Rating

Table 21 - Development plan for Visual Attention modules

3.2.2 Transport Stream Adaptation for Additional Content Signalling The purpose of this module is to add additional signalling into the Transport Stream (TS) that is beyond normal Program Service Information (PSI) especially information for discovery or identification of additional contents that are not carried by the specific transport stream but available from other sources.

3.2.2.1 Related Description of Work Tasks Task 5.1, Task 5.2, Task 5.3, Task 6.2, Task 6.5

ROMEO WP6 /68

3.2.2.2 Detailed Module Diagram

Figure 28 - Transport Stream Adaptation Module

3.2.2.3 Brief Module Explanation The module will consist of four components: TS Source and Sink, TS Processor and Control/Configuration Interface. The input TS will be analysed and in turn modified to insert all required signalling information required to link to additional content not directly included within the stream. Thus the generated output TS is the original stream plus the added signalling information.

3.2.2.4 Module Development Plan A first version of the module is planned for M16. This implementation will be a file based solution. The final version will be available before M32.

M14 M15 M16 M17 … M23 M24 … M32

Development of module v1 … …

Development of module v2 … …

Finalizing module according to integration progress

… …

Validation of adapted streams

.. ...

Table 22 - Development plan for Transport Stream Adaptation Module

3.2.3 Video Encoding This component has several purposes:

• Reading multiple camera and multiple depth map uncompressed video files from a directory and generating descriptions in pixel domain

• Applying SVC encoder on each description to generate multi-layer video description streams and delivering them to MPEG-2 TS encapsulation block (alternatively, compressed video content can be recorded in a directory, where the encapsulation block can read them to initiate real-time play-out)

• Generating the video interpolation metadata compressed stream from the compressed video description streams


ROMEO WP6 /68


Figure 29 - Diagram of the Video Encoder block

3.2.3.3 Brief Module Explanations The video encoder block consists of an offline (pre-processing based) multiple description generator and SVC encoder instances that are based on JSVM encoder library. Input the block is the raw video frames (YUV4:2:0), either disparity map or colour texture views. Output is the SVC elementary stream for MPEG2 TS encapsulation block.

3.2.3.4 Module Development Plan Based on the delivered demonstration content, descriptions will be generated and long test sequences will be produced offline that will comprise two quality layers, before the end of M17. The bit-rate for the quality layers and disparity maps will be determined based on the subjective quality tests. Visual attention based quantiser selection task in the SVC encoder will be accomplished before the end of M16. View interpolation metadata generation and encoding block’s implementation will start in M17 and scheduled to be finished before the end of second project year. It is planned to embed the generated view interpolation metadata into the video elementary stream(s) and thus, transmit them along with the coded video information.

M14 M15 M16 M17 M18 M19 M20 M21 M22 M23

Description generation for long test sequences (two quality layers)

Visual attention based quantiser selection – SVC integration

View interpolation metadata generation and integration with the SVC codec

Table 23 - Development plan for Video Encoder module

3.2.4 Audio Encoding The goal of this component is to encode spatial audio content and represent it as audio bit stream for the purpose of efficient transmission. The 5.1-channel or object-based audio will be used as basic format while compatibility to stereo audio format, mainly for audio rendering with portable or mobile terminal, is desired.



SVC

SVC

SVC

SVC

Multiple description generator

( pre - processing )

0107

0107

0107

0107

Content Generation 0107

encapsulation

encapsulation

encapsulation

encapsulation

0202

0202

0202

0202

Video Encoder

ROMEO WP6 /68

The diagram of audio encoder for Romeo integration is shown in spatial audio encoder figure.

Figure 30 - Diagram of Spatial Audio Encoder

For the analysis-by-synthesis (ABS) spatial audio encoder, various techniques of spatial audio encoding can be applied within the ABS framework. The diagram of ABS encoder is shown in ABS spatial audio encoder figure. The R-OTT module, as used in MPEG Surround to down-mix two channels into a single channel, is applied in the ABS framework.

Figure 31 - Diagram of ABS Spatial Audio Encoder


3.2.4.3.1 Spatial Audio Encoder Module This module is used to encode spatial audio content generated in the audio post-production block within the content generation module. For basic scheme of delivering channel-based or object-based audio, AAC multichannel encoder will be applied. In the case of delivering channel-based audio signals, each channel of 5.1 audio signals: left (L), right (R), centre (C), low-frequency effects (LFE), left surround (Ls), and right surround (Rs) will be encoded separately and then transmitted as a single bit stream. For this scenario, the AAC multichannel will operate at its typical bit rate of 320 kbps (total bit rate for all channels). To maintain stereo backward compatibility both the L and R channels will be encoded at a bit rate of 128 kb/s. The audio bit stream is subsequently fed to the audio stream packetizer module.

3.2.4.3.2 T/F Transformation in ABS Encoder The ABS encoding process will be carried out in the sub-band frequency domain. Thus the T/F Transformation module is used, for each audio channel, to decompose the time domain audio signal into frequency sub-band domain.

3.2.4.3.3 OTT-Modules in ABS Encoder One-to-two (OTT) module, as used in MPEG Surround to up-mix a single audio channel into two channels, is applied in ABS. A number of OTT modules will be applied to up-mix a single audio channel into multiple channels. For instance, to up-mix a single channel into 5 channels a total of 4 OTT modules may be needed.

ROMEO WP6 /68

3.2.4.3.4 Closed-loop R-OTT Modules in ABS Encoder Reverse (R-OTT) Module, as used in the encoder side of MPEG Surround, is used in ABS encoder to perform down-mixing process, as a reverse block of the OTT module. To achieve higher quality of reconstructed audio signals a closed-loop, instead of the open-loop, R-OTT module will be applied.

3.2.4.3.5 Error Minimisation in ABS Encoder This module is used to compare the reconstructed audio signals with the original signals in ABS encoder. The error introduced in the encoding process will be calculated. During an ABS optimisation loop, the reconstructed audio signals which provide minimum error will be determined and then transmitted to the decoder.

3.2.4.4 Module Development Plan The development of spatial audio encoder module for ROMEO integration starts from M13 and working version will be available from M16, with further improvement and optimisation in the following months. The ABS spatial audio encoder will also be under development from M13.

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24

Spatial Audio Encoder Module

OTT Module

Closed-loop R-OTT Module

T/F Transformation

ABS optimisation algorithm

Table 24 – Development plan for Audio Encoding Component

3.2.5 P2P

3.2.5.1 P2P Topology Builder The Topology Builder (TB) component is responsible for: (i) the construction of the multiple multicast tree structure for the P2P network. The trees - one tree per each stream (view) - are used for the distribution of 3D media content in the P2P network. The TB is also responsible to find a suitable position to insert new joining peers (a procedure known as peer insertion) in the P2P network. As depicted in block diagram for topology builder figure, the TB interacts with other ROMEO components/modules; such interactions are described in section 3.2.5.1.3.

3.2.5.1.1 Related Description of Work Tasks Task 4.4, Task 5.1, Task 6.3

ROMEO WP6 /68


Figure 32 - Block diagram for Topology Builder

3.2.5.1.3 Brief Module Explanations In order to perform its functions, the TB: (i) operates as a server, by listening for new peer connection requests; (ii) as an authentication proxy (authenticator), for user authentication, and ; (iii) as a collector/aggregator of network monitoring data, to obtain on-the-fly network status. TB module has 4 sub-modules:

� Authenticator - is responsible to exchange messages with the Authentication, Registration and Security component in order to verify authenticity and authorization for new incoming peers.

� Peer Insertion Control - is responsible for managing the peer insertion at the tree. If the peer disconnects (graceful or ungraceful), it must perform the operations in order to remove the peer from the tree.

� P2P Overlay Constructor – uses the tree construction and maintenance algorithm described in ROMEO D5.1 [4] to compute the multiple trees used for the distribution of 3D media content in the P2P network. The tree construction is based on peer join request information received from each peer’s Topology Controller. The P2P Overlay Constructor is also capable to ask IRACS assistance for resource reservation availability.

� NMS Data Collector – interacts with the Network Monitoring Subsystem Multicast Tree Manager to obtain on-the-fly network statistics. This sub-module is also part of the Multicast Tree Manager module, see section 3.2.5.2, and the collected information is made available (internally) to the P2P Overlay Constructor for tree maintenance purposes.

� Communications interface - is responsible for all communications related to either configuring the blocks or sending the indicated control messages to other components/modules of ROMEO platform. This block also listens on specific TCP and UDP ports, and acts as both server and client

3.2.5.1.4 Module Development Plan It is planned to finalize the multiple multicast tree construction algorithm before the end of December. Following the algorithm implementation, messages to interact with the ROMEO system will be implemented within the month 16.Testing and system integration tasks will be held in months 17 and 18.

ROMEO WP6 /68

M14 M15 M16 M17 M18

Algorithm Development Msg implementation for interaction with the whole system Testing Integration with other components

Table 25 - Development plan for Topology Builder module

3.2.5.2 Multicast Tree Manager (MTM) This component receives, from the Network Monitoring Subsystem, the data describing the conditions of the links of the overlay managed by the Topology Controller. The MTM computes trees in the overlay along with their potential performance, considers the collected network data, and finally decides which trees are actively used for multimedia delivery (ACTIVE trees), which ones have enough potential to be actively monitored and kept ready for usage (READY trees) and finally which trees are too bad to be used - for the moment (SLEEPING trees). This strategy allows for fault tolerance by providing peers with backup connections in case of ACTIVE link failure.

3.2.5.2.1 Related Description of Work Tasks Task 4.4, Task 5.1, Task 5.2, Task 5.3, Task 5.4, Task 6.3


Figure 33 – Block diagram for the Multicast Tree Manager

3.2.5.2.3 Brief Module Explanations This module includes Communication Interface, Tree Manager, NMS Data Collector and QoS Manager sub-modules as depicted multicast tree manager block diagram. This module interacts with Topology Builder and IRACS at the server side, and with P2P Network Monitoring Subsystem, Topology Controller and Chunk Selection modules at the peer side. The messages corresponding to these sub-modules will be implemented within the development phase of this module

3.2.5.2.4 Module Development Plan The development of this module is expected to be finalized at the end of Month 18.The procedures for ACTIVE, READY and SLEEP trees are expected to be finalized till the end of Month 15. Internal and external messaging infrastructure will be implemented till the end of M17. The testing period will start on M15 and will have a duration of 2 months. The integration phase is expected to be completed by the end of M18.

ROMEO WP6 /68

M14 M15 M16 M17 M18

Procedures for ACTIVE, READY and SLEEP treees/links Msg implementation for interaction with the whole system Testing Integration with other components Table 26 - Development plan for Multicast Tree Manager module

3.2.5.3 Packetisation This component is responsible for the encapsulation of the chunks to be distributed over P2P network.

3.2.5.3.1 Related Description of Work Tasks Task 6.6 and Task 7.1


Figure 34 - Block diagram for packetisation module

3.2.5.3.3 Brief Module Explanations Since this is an essential module for P2P data transfer, it is planned to finalize it soon. In order to verify its full-functionality, it is necessary to have NALUs encapsulated in MPEG2-TS packets.

3.2.5.3.4 Module Development Plan Since this is an essential module for P2P data transfer, it is planned to finalize it soon. In order to verify its full-functionality, it is necessary to have MPEG2-TS encapsulated NALUs which are the output of content generation part of ROMEO project. Testing of this component will be performed in M15 and in the following months it is planned to integrate it with the whole system.

M13 M14 M15 M16 M17

Msg implementation for interaction with the whole system Packetisation of ROMEO data provided by US and R&S Testing Integration with other components

Table 27 - Development plan for packetisation module

3.2.5.4 P2P Transmitter This component is responsible for encapsulating the network related calls for message and content delivery over the IP network. All of the transmission and reception operations should be performed over this component.


ROMEO WP6 /68

3.2.5.4.2 Detailed Module Diagram of Component This module interacts with the corresponding P2P Transmitter/Receiver modules of other peers. While the content is planned to be delivered through UDP, the control messages is planned to be sent through TCP.

3.2.5.4.3 Module Development Plan The API of network related calls for message and content delivery will be implemented till the end of M13, which is the end of October 2012. While the unit tests will be performed on the M14, the integration phase will be completed before the 4th GM meeting, by the end of M17.

M13 M14 M15 M16 M17

API implementation for data transfer between computers Testing Integration with other components

Table 28 - Development plan for P2P Transmitter module

3.2.6 DVB Transmission The DVB transmission part uses pre-recorded MPEG2 Transport Streams as input to the DVB modulator and the IP encapsulator during the laboratory testing.

For the field trials, the MPEG2 TS that delivers the input to the DVB modulator consists of programmes recorded in ROMEO. The input to the IP network is based on P2P content that is encoded and encapsulated in real-time and provided through a dedicated server.



Figure 35 - Block diagram of the DVB Transmission part (green: tested and evaluated, yellow: under development, grey: not started yet)

ROMEO WP6 /68

Based on tests carried out with the Test Terminal frontend and a USB receiver, a suitable DVB-T2 mode with a sufficient data rate for the planned services, has been proposed for usage in the ROMEO project:

• FFT size: 16k ext. • Modulation: 16QAM • Pilot pattern: PP2 • Guard Interval: 19/128 • Code Rate: 3/5 • Useful data rate: 14.8 Mbps

3.2.6.3 Brief Module Explanations The DVB modulator/ transmitter and the channel simulator were validated for the DVB-T2 modes suitable for fixed, portable and mobile reception.

Development work concentrates on the encapsulation of the video/ audio material that was recorded in the ROMEO project, for the TS repository. At the beginning, a few test sequences of about 3 to 5 minutes length are encapsulated into an MPEG2 Transport Stream according to the requirements in the project. These test streams are then to be incorporated into the stream library.

3.2.6.4 Module Development Plan The Transport Streams with the test sequences need through testing by all partners involved in the design of terminals. This testing and the subsequent adaptations and modifications are planned to be completed in the 6th quarter of the project, i.e. in-time for the intermediate demonstrator set-up.

The encapsulation of MPEG2 Transport Stream packets into IP packets is being prepared. It depends on the availability of the correctly encapsulated MPEG2 Transport Streams.

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24

TS test streams development

TS test streams validation

Development of TS with long sequences

Validation of long sequence TS

Table 29 - Development plan for DVB transmission module

3.2.7 Network Monitor The Network Monitor Subsystem (NMS) residing in the server is a software module that periodically gathers network information (such as delay, jitter and bandwidth) from peers in the network. This information is then provided to the Topology Builder/Multicast Tree Manager (at the server) so that it can perform informed decisions on the multicast trees (ACTIVE, READY, and SLEEPING status). In the ROMEO environment, not all peers report to the NMS residing at the server, only super-peers or high-level parent peers send such report. Section 3.1.9, 3.1.9 better explains how this distributed monitoring is processed.

ROMEO WP6 /68

3.2.8 Audio-Visual Communication Overlay Audio-Visual Communication Overlay component on server will be responsible of creating a communication channel between the users that experience the same 3D content.



Peer

Server

Conference

Controller

A/V Communication

Overlay

A/V Content

Distributer

A/V Communication

Overlay

Conference

Controller

A/V Content

Distributer

0900

A/V Content

Receiver

0902 09030901

Overlay

Manager

A/V Content

Receiver

0904

0904

0905

910

0912

0910

0911

Renderer

Other Peers

UI & Control1009/

1010

Synchronisation

Control

A/V Adaptation

Module

Synchronisation

0913

0914

Figure 36 – A/V Communication Overlay block diagram

3.2.8.3 Brief Module Explanations The server side of A/V Communication Overlay component is briefly described here. The three main modules of the server component are Conference Controller, A/V Content Distributer and A/V Content Receiver.

3.2.8.3.1 Conference Controller Conference Controller module on server is responsible for room management of video-conference sessions between remote users. The module also controls the audio/visual data stream over communication channel.

3.2.8.3.2 A/V Content Receiver A/V Content Receiver module on server is responsible for collecting the A/V streams of remote users to distribute later.

3.2.8.3.3 A/V Content Distributer A/V Content Distributer module on server is responsible for delivering all A/V streams collected from remote users to all users on session.

ROMEO WP6 /68

3.2.8.4 Module Development Plan Modules will be developed in parallel. On the other hand, synchronisation and conference controller modules are expected to last till integration with other components of the ROMEO platform.


Peer

Conf. Controller

A/V Content Distributer

A/V Content Receiver

Synch. Controller

Overlay Manager

Server

Conf. Controller

A/V Content Distributer

A/V Content Receiver

Table 30 – A/V Communication Overlay component development plan

3.2.9 Authentication, Registry and Security This component is responsible for the following tasks:

• User authentication based on licences sent • Encryption of the content for protecting from unauthorized users. • Key management for the user generated data • Integration checking of the user generated data


ROMEO WP6 /68


Figure 37 - Block diagram for Authorization Registration and Security component

3.2.9.3 Brief Module Explanations With the scope of this module, it is planned to implement Security and User REST APIs. The Security REST API will be implemented to create, delete or update public/private key pairs for each ROMEO user. The User REST API, on the other hand, will be implemented to deal with the user authentication and user rights

3.2.9.4 Module Development Plan The APIs will be implemented in M14, November 2012. Following the testing period within M15, the integration phase will start to enable seamless interaction of this module with other ROMEO modules and end at the end of M18

M14 M15 M16 M17 M18 The API implementation for Authentication, Registration and Security Module Testing Integration with other components

Table 31 - Development plan for Authentication Registration and Security module

3.2.10 User Generated Content This component is responsible for registering, storing and discovering the requested content file.


ROMEO WP6 /68


Figure 38 - Block diagram for user generated content

3.2.10.1.3 Brief Module Explanations In the development phase of this module, the interactions with the other modules such as Authentication, Registry and Security and the mechanisms enabling the client and server streaming the content will be implemented.

3.2.10.1.4 Module Development Plan The implementation of the REST based Server/Client architecture has already started. . It is planned to be finalized by the end of M14, which is the end of November 2012. By the end of M15, UGC module is planned to be tested and the possible issues will be addressed to make it ready for the integration. In the following three-month period, it is planned to spend efforts on integration of this module into ROMEO system.

M14 M15 M16 M17 M18

Msg implementation for interaction with the whole system Testing Integration with other components

Table 32 - Development plan for Authentication Registration and Security module

3.3 Network Related Development

3.3.1 Mobility The Mobility component is responsible for ensuring seamless and uninterrupted 3D video streaming service to mobile and portable users, across heterogeneous wireless access networks. The synergy of Media Independent Handover (MIH), Media Aware Proxy (MAP) and Proxy Mobile IP (PMIP), provides the ability to optimize the Quality of Experience of video streaming according to the available radio networks resources by performing horizontal and vertical handovers, when possible or selecting multi-home transmission.

ROMEO WP6 /68

3.3.1.1 Related Description of Work Tasks Task 6.6 and Task 7.1


Figure 39 – Block diagram of Mobility component

3.3.1.3 Brief Module Explanations The operation of the Mobile component, shown in block diagram of mobility component figure, is threefold and it is related to the three main functionalities that it consists of: a) Media Aware Proxy, b) Media Independent Handover and c) Proxy Mobile IP. In particular, the IP traffic generated from the parent P2P transmitter, which is aimed for the wireless P2P client, is filtered by the transparent to the P2P overlay Media Aware Proxy. MAP parses the chunk headers and initiates an internal adaptation decision engine that will decide whether or not the video transmission rate can be supported by the currently available wireless link bandwidth. In the case where, the transmission rate exceeds the available radio resources, then less important enhancement layers or video descriptions will be filtered out. The adaptation decision engine is aided by the Media Independent Handover functionality. The latter periodically monitors all available access networks in the wireless client’s neighbourhood. The collected parameters from all networks and active clients are stored in a database. The database is also updated by the adaptation decision engine, which stores information from the chunk headers regarding the number of views, the video transmission rate, layer id, priority flags, etc. This information is passed to the Handover functionality that is responsible for deciding horizontal or vertical handovers or multi-home transmission, in order to keep the current Quality of Experience level or if possible increase it. In case of a handover decision, a flag message is sent to the Home Agent, which in turn executes the decided Layer 3 handover

ROMEO WP6 /68

and establishes the tunnel to the Proxy Mobility Agent that corresponds to the access network that will receive the handover.

3.3.1.4 Module Development Plan Mobility is a major part of the ROMEO use case; hence some of the functionalities that will ensure some kind of mobility have to be developed soon. MAP development will be ready by month 14. In parallel, MIH and PMIP functionalities that require also the procurement of LTE access and the development of the portable and mobile clients will be ready by month 16. The Testing phase is planned for month 16 when all functions will be developed and by that time the integration with other units will commence and will be concluded by month 17.

M13 M14 M15 M16 M17

Media Aware Proxy Media Independent Handover Proxy Mobile IP Testing Integration with other components

Table 33 – Development plan for mobility component

3.3.2 Virtualisation

3.3.2.1 Related Description of Work Tasks Task 5.1, Task 5.3, Task 5.4, Task 6.1, Task 6.2, Task 6.4


Figure 40 – Block diagram of virtualisation component


3.3.2.3.1 Optical Network Terminator (ONT) It is the subscriber equipment for the Gigabit Passive Optical Network (GPON) access

3.3.2.3.2 Optical Line Termination (OLT) It corresponds to the access node for the GPON access. From one side terminates the optical fibre from the subscriber and from the other side it connects to the Metropolitan Area Network (MAN) to transmit the traffic to the due service centres. The OLT employs 802.1ad encapsulation (stacked VLANs) for traffic destined to the NATIE. This encapsulation protocol allows multiple VLAN headers to be inserted in a single frame.

ROMEO WP6 /68

3.3.2.3.3 DHCP SERVER and DDBB It allocates IP addresses to all devices connected to the Ethernet ports of an ONT. The DHCP server is located in the MAN. The broadcast requests perform by Ethernet devices will cross the OLT and the MAN in order to reach the NATIE. Thus, in order to enable DHCP server to receive IP address requests from Ethernet home devices, the NATIE must be configured as DHCP relay pointing to the DHCP server.

3.3.2.3.4 Network Address Translation Implementer Equipment (NATIE) This equipment concentrates the user Ethernet Traffic to route it to the different services (voice, IPTV, Internet). The virtualization platform establishes the NATIE as the key point for hosting the HGW routing and HGW NAT/PAT functionalities. These functionalities will be implemented for each subscriber within a virtual routing forwarding (VRF) instance. In addition, the NATIE will receive the QoS rules that are to be applied to each user in order to enforce the flow prioritization.

3.3.2.3.5 Policy Charging and Rules Function (PCRF) The PCRF is the entity that defines the QoS rules that are to be applied to the user. It receives from the ROMEO Topology Builder module the physical parameters that identify a certain user and then it communicates with the NATIE in order to enforce the rules.

3.3.2.3.6 Configuration Portal The configuration portal is the entity that will provide to the user a tool for configuring the virtual router functionalities. Besides addressing the current configurable functionalities of a physical HGW (DHCP options, NAT/PAT, IP addressing scheme, etc), the configuration portal will provide a protocol prioritization tool to manually adjust the QoS rules that are to be enforced in the access network.

3.3.2.3.7 Virtual Software Execution Environment (VSEE) It corresponds to the module in charge of hosting the virtualized execution software capacities that currently runs within the RGW. The system provides a layer 2 visibility to the local area network (LAN) of the subscriber.

3.3.2.4 Module Development Plan

M13 M14 M15 M16 M17 M18 M19 M20 M21 M22 M23 M24 M25 M26 M27 M28

OLT Cmp.

NATIE Cmp.

DHCP Server and DDBB

Cmp.

PCRF

Conf. Portal

VSEE

Table 34 - Development plan for Virtualisation

3.3.3 Internet Resource and Admission Control Subsystem (IRACS) This component is responsible for efficient use of backbone network resources while assuring differentiated QoS guarantees. It is responsible for managing and providing bandwidth guaranteed paths for connectivity across the ROMEO core network. It encompasses two

ROMEO WP6 /68

modules: the Resource and Admission Manager (RAM) and the Resource Controller (RC) modules which are detailed respectively in subsections 3.3.3.1 and 3.3.3.2.

3.3.3.1 Resource and Admission Manager (RAM) The RAM is the module used for network resource and admission control decisions making in the network. It assures QoS and paths computation control decisions. Hence, connection requests to a backbone network are sent to the RAM which processes requests and provides appropriate candidate paths for communication.



Figure 41 - Block diagram for Internet Resource and Admission Manager.

3.3.3.1.3 Brief Module Explanations This module includes Communication Interface, and the Resource and Admission Manager (RAM). The RAM is composed of three sub-modules: the Control Information Base which maintains network resources and policies control information, the Resource Reservation Subsystem which defines reservations and manages bandwidth-aware IP multicast trees, and the Admission Control Subsystem assures services-to-QoS mapping depending on traffic requirements and network resource conditions. The Communication Interface allows for control information exchanges between RAM and external elements such as the P2P Topology Builder & Multicast Tree Manager, the P2P Topology Controller, and the Resource controller.

3.3.3.1.4 Module Development Plan Since the design of this module involves control messages handling, resource reservation & IP multicast trees control policies decisions, and traffic flows mapping to bandwidth-aware trees throughout the P2P network, these functionalities are planned to be implemented till the end of M15, that is, the end of December 2012. The M16 is for the overall testing and the integration phase will be completed by the end of M18 including the integration efforts in the 4th GM meeting in Munich.

ROMEO WP6 /68

M14 M15 M16 M17 M18

Msg implementation for interaction with the whole system. Resource reservation & IP multicast trees control policies and decisions making Traffic flows mapping to IP multicast trees. Testing Integration with other components

Table 35 - Development plan for Internet Resource and Admission Manager module.

3.3.3.2 Resource Controller (RC) The Resource Controller (RC) module is used by network nodes to enforce control decisions received from the RAM through signalling messages. It is also responsible for data forwarding inside the network.



Figure 42 - Block diagram for Internet Resource Controller.

3.3.3.2.3 Brief Module Explanations This module includes Communication Interface and the Resource Controller (RC) modules. The Communication Interface allows for control information exchanges between RC and external elements such as the Resource and Admission Manager and the A/V Adaptation & Network Monitor.

3.3.3.2.4 Module Development Plan Since the design of this module involves control messages handling, resource reservation & IP multicast trees control policies decisions enforcement, and traffic packets forwarding, these functionalities are planned to be implemented till the end of M15, that is, the end of December 2012. The M16 is for the overall testing and the integration phase will be completed by the end of M18 including the integration efforts in the 4th GM meeting in Munich.

M14 M15 M16 M17 M18

Msg implementation for interaction with the whole system. Resource reservation & IP multicast trees control policies and decisions enforcement, Traffic packet forwarding. Testing Integration with other components

Table 36 - Development plan for Internet Resource Controller module.

ROMEO WP6 /68

4 CONCLUSION This deliverable provides information on detailed description of components and their modules. Development plans are documented with considering each module’s timeline. This document will also let partners to check other components’ timelines for integrated solution. A summary figure is displayed below for all ROMEO components.

Figure 43 - ROMEO platform development plan

ROMEO WP6 /68

5 REFERENCES [1] “ROMEO Deliverable 2.2 - Definition of the initial reference end-to-end system architecture

and the key system components” [2] “ROMEO Deliverable 6.1 - First report on system components development plan.docx” [3] http://www.pctvsystems.com/Products/ProductsEuropeAsia/DVBTT2products/PCTVnanoSt

ickT2/tabid/248/language/en-GB/Default.aspx [4] “ROMEO Deliverable 5.1 - Interim report on 3D media networking and

synchronisation of networks” [5] “ROMEO Deliverable 6.2 - First Report on server, peer, user terminal, security and content

registration modules development” [6] The libpcap project. url: http://sourceforge.net/projects/libpcap/ [7] M. Blestel and M.Raulet, "Open source code of an svc decoder", SIGMultimedia Rec., vol.

1, no. 4, pp. 24, December 2009.

ROMEO WP6 /68

APPENDIX A: TASK LIST OF ROMEO WPs Title WPL/TL WP1 Management US

T1.1 Technical Management and Quality Control US T1.2 Administrative and financial management US T1.3 Coordination and cooperation between partners US

WP2 System Architecture TTA T2.1 System Architecture and Reference Scenarios ARCELIK T2.2 Fixed and Mobile Users’ Terminals Requirements ARCELIK T2.3 Audio-Visual Format Requirements TEC T2.4 Peer-to-Peer Overlay Specifications TTA T2.5 Joint content and network optimisation details TTA T2.6 Media Networking specification IT T2.7 Terminal Equipment Virtualisation TID

WP3 3D Audio-Visual Processing TEC T3.1 Content capture, Pre-Processing and Content Preparation VITEC T3.2 QoE Modelling for Compression US T3.3 Visual Attention Modelling IRT T3.4 3D Audio-Visual Rendering TEC T3.5 3D Audio-Visual adaptation TEC

WP4 Joint Content and Network Optimisation US T4.1 Scalable and Multiple Description Coding of Multi-View Video US T4.2 Spatial Audio Coding US

T4.3 Streaming and Networking Aspects for 3D Multi-View Video and Spatial Audio VITEC

T4.4 Content Aware Peer-to-Peer Overlay Design TTA T4.5 Optimisation of Audio-Visual Compression and Peer-to-Peer Overlay VITEC

WP5 Advanced Heterogeneous Media Networking IT T5.1 P2P and DVB interworking for fixed users TTA T5.2 P2P and DVB interworking for mobile users extension IT

T5.3 Playout Synchronisation for Real-time 3D Content Delivery in Heterogeneous Networks TTA

T5.4 Real-Time Audio-Visual Communication Overlay Between Remote Users TTA T5.5 User Generated Content Provisioning IT T5.6 Cross-Layering for Spectral Efficient Service Delivery IT T5.7 Terminal Equipment Virtualisation TID

WP6 System Components Development ARCELIK T6.1 System Components Development Planning and Validation ARCELIK T6.2 Main Server Development TTA T6.3 Peer Development IT T6.4 User Terminal Development ARCELIK T6.5 3D Multi-view DVB-T2/NGH Broadcast R&S T6.6 Hot-Spot Wi-Fi and 3G/LTE Cellular System UP T6.7 Overall System Configuration and Synchronisation TTA T6.8 Content Registration and P2P Security TTA

WP7 Integration and Demonstration R&S T7.1 Overall Integration and Demonstration Planning and Validation ARCELIK T7.2 Test Bed Integration R&S T7.3 Pilot set-up & Evaluation Methodologies IT T7.4 Trials and Results Analysis TID

ROMEO WP6 /68

WP8 Dissemination and Exploitation UP T8.1 Dissemination UP T8.2 Standardization IRT T8.3 Exploitation Plans ARCELIK

T8.4 ROMEO Workshops R&S, UP,

TTA

ROMEO WP6 /68

APPENDIX B: GLOSSARY OF ABBREVIATIONS

A

AAC Advanced Audio Coding

AbS Analysis by Synthetis

ALSA Advanced Linux Sound Architecture

AP Access Point

AVC Advanced Video Coding

AU Access Units

A/V Audio-Visual

B

BRAS Broadband Remote Access Server

C

CLD Channel Level Differences

CPC Channel Prediction Coefficients

CPE Consumer-Premises Equipment

CPU Central Processing Unit

CoS Class of Service

CUDA Computer Unified Device Architecture

D

DBIR Depth based Image Rendering

DHCP Dynamic Host Configuration Protocol

DPX Digital Picture Exchange

DVB Digital Video Broadcasting

E

eNodeB Enhanced Node B

ES Elementary Stream

F

fps Frames per Second

FTTH Fiber to the Home

G

GbE Gigabit Ethernet

GOP Group of Pictures

GPL General Public license

GPON Gigabit Passive Optical Network

GPU Graphics Processing Unit

GUID Global Unique Identification

ROMEO WP6 /68

H

HD High Definition

HDMI High Definition Multimedia Interface

HGW Home Gateway

I

ICC Inter-channel Coherences

IEEE Institute of Electrical and Electronics Engineers

IP Internet Protocol

J

JSON-RPC Javascript Object Notation – Remote Procedure Call

K

L

LAN Local Area Network

LGPL Lesser General Public license

LMA Local Mobility Anchor

LTE Long Term Evolution

M

MAG Mobile Access Gateway

MANE Media Aware Network Element

MAP Media Aware Proxy

MIH Media Independet Handover

MIP Mobile Internet Protocol

MME Mobility Management Entity

MN Mobile Node

MPEG Motion Picture Experts Group

MPS MPEG Surround

MTM Multicast Tree Management

N

NACF Network Atachment Control Functions

NALU Network Abstraction Layer Unit

NAT Network Address Translation

NATIE Network Address Translation Implementer Equipment

NMS Network Monitor SubSystem

NTFS New Technology File System

O

OLT Optical Line Terminator

ONT Open Network Terminator

ROMEO WP6 /68

OS Operating System

P

PCM Pulse Coded Modulation

PCRF Policy Charging and Rule Function

PES Packetised Elementary Stream

PMIP Proxy Mobile IP

P2P Peer to Peer

Q

QoE Quality of Experience

QoS Quality of Service

R

RAM Random Access Memory

RF Radio Frequency

RGW Residential Gateway

RTP Real Time Protocol

S

SBC Session Border Controller

SDI Serial Digital Interface

SI Service Information

SIP Session Initiation Protocol

SNR Signal to Noise Ration

SVC Scalable Video Coding

T

TCP Transmission Control Protocol

TS Transport Stream

U

UDP User Datagram Protocol

UI User Interface

UMTS Universal Mobile Telecommunications System

USB Universal Serial Bus

V

VCL Video Coding Layer

VoDSAR Video on Demand Service Access Router

VRF Virtual Router Forwarding

VSEE Virtual Software Execution Environment

W

WFS Wave Field Synthesis

ROMEO WP6 /68

WiFi Wireless Fidelity

WLAN Wireless - LAN

X

Y

Z

Numbers

Remote C ollaborative Real-Time Multimedia Experience over ... · Task Name Start Finish...

Documents

Transcript of Remote C ollaborative Real-Time Multimedia Experience over ... · Task Name Start Finish...