Standard approaches and a case study - UniNa STiDuEunina.stidue.net/Applicazioni...
Transcript of Standard approaches and a case study - UniNa STiDuEunina.stidue.net/Applicazioni...
Standard approaches and a case study
1
Corso di Sistemi MultimedialiA.A. 2010-11
Prof. Antonio Picariello
Corso di Applicazioni TelematicheA.A. 2010-11
Prof. Simon Pietro Romano
Università degli Studi di Napoli Federico IIFacoltà di Ingegneria
9/6/2011
Outline Web Conferencing Introduction History and technologies
IETF standardization efforts
Milestones and ongoing works in the RAI area Centralized Conferencing, Media Control, Session Recording,
Via-browser conferencing, Telepresence Systems
From theory to practice: the Meetecho platform A fully-fledged standard conferencing system made in
unina
2
A “Web Conferencing” definition The term “Conference” can be used to describe any meeting
of people that “confer” about a certain topic
“Web Conferencing” is used to conduct live meetings or presentations over the Internet
3
Features VoIP (Voice over IP)
Live video
Text chat
Slide presentations
Whiteboard with annotation
Screen/desktop sharing
Application sharing
Recording
Polls and surveys
4
History Tele-Conferencing
Conference calls (Audio Tele-Conferencing)
Video conferences (Video Tele-Conferencing)
IP-Conferencing
Text Conferencing
Audio and Video Conferencing
Data Conferencing
5
Audio and Video ConferencingEvolution Analog Phone Lines (PSTN)
Conference calls Three-way calling
Conference bridges
Video Telephony
Digital Telephony (ISDN) ITU-T H.320 umbrella recommendation
Next-Generation Networks & VoIP technologies
6
VoIP protocols SIP – Session Initiation Protocol, RFC 3261
Call signalling
Session intitiation, managing and termination
Codec negotiation
SDP – Session Description Protocol, RFC 4566
Session media stream description
RTP – Real-time Transport Protocol, RFC3550
Media streams transfer
7
Text Conferencing Evolution Asynchronous Meetings
Posted text messages (not live)
Message/Bulletin Boards
Forums
Network Newsgroups/Mailing lists
Synchronous Meetings
Live text communication
Internet Relay Chat (IRC)
Web-based Chat (CGI/Java)
Instant Messaging (Skype/MSN/XMPP/etc.)
8
A B
C
A+B+CA+B
Typical scenarios From Point-to-Point Calls to Multipoint Calls
Three-way calling Coaching scenario
Lecture-mode Conferences Presentation Question & Answers session
Ad-hoc and Reserved Conferences Conference-aware/-unaware participants Manage conference/users/media/policies Sidebars/Whispers
9
Client B
Client CClient A
Media A Media C
Me
dia
B
Server
A
B
B
A
A+B
Issues Call signalling
Gateway functionality
Control and Management Tone detection (DTMF)
Dedicated protocols
Mixing and Transcoding Terminal capabilities
User media profiling Coaching scenario
Videoswitching
10
Client BClient A
SIP H.323
Server
Standardization Efforts Geographically dispersed businesses, the rising cost of energy, and the
development of NGN lead to the proliferation of proprietary conferencing solutions
No standardization for many years Lack of interoperability Platform dependency Security issues Cost Market segmentation
Standardization Bodies ITU (International Telecommunication Union) 3GPP (3rd Generation Partnership Project) IETF (Internet Engineering Task Force)
11
Standardization Efforts: ITU Established to standardize and regulate international radio and
telecommunications
International Standards referred to as “Recommendations”
ITU-T: Telecommunication Sector
G: Transmission Systems and Media G.71x (Audio compression, mu-law and a-law) G.72x (Audio compression, ADPCM)
H: Audiovisual and Multimedia Systems H.320 (PSTN/ISDN, Telephone Systems) H.323 (IP, Packet-based Communication Systems)
T: Terminals for Telematic Services T.120 (Data Sharing Protocols) T.140 (RTP Interactive Text)
12
Standardization Efforts: IETF Under the umbrella of the Internet Society
Develops and promotes Internet Standards
Deals in particular with standards of the TCP/IP suite
Organization
Working Groups (WG)
Internet Drafts
Requests for Comments (RFC)
“Rough consensus, running code”
We will focus on some RAI area activities...
13
IETF RAI area RAI – Real time Applications and Infrastructure
“Legacy” standards Standard VoIP protocols
SIPPING
XCON
MediaCtrl
Brand-new standardization activities SIPREC
RTCWEB
CLUE
14
SIPPING Working Group Session Initiation Protocol Proposal INvestiGation Documents the use of SIP for several applications related to
telephony and multimedia SIP Conferencing Models
15
Loosely-Coupled Conference Fully Distributed Multiparty Conference
Tightly-Coupled Conference
Sip ConferencingFramework
Main actors: - Participants- Focus- Mixer- Policy Server- Notification Server
XCON Working Group Centralized Conferencing (“X”CON)
A star topology on the signalling plane
Advanced conferencing features VoIP + video & data sharing
“Signalling-agnostic” Not only SIP
H.323, IAX, …
Defines : Conference data model and lifecycle A suite of client-server protocols to realize sophisticated
conferencing scenarios
16
Conference Object &Data Model
17
firstjoin
Template conference object
(blueprint)
Registered conference object
Activeconference object
cloningcreation
lastleave
<conference-info>
<sidebars-by-val>
<sidebars-by-ref>
<conference-description>
<host-info>
<floor-information>
<conference-state>
<users>delete
delete
Conference object example
18
<info:conference-info entity="8773158"><info:conference-description>
<info:display-text>My Full-Media Conf</info:display-text><info:conf-uris>
<info:entry><info:uri>xcon:[email protected]</info:uri><xcon:conference-password>3903/0</xcon:conference-password>
</info:entry></info:conf-uris><info:available-media>
<info:entry label="audioLabel"><info:type>audio</info:type>
</info:entry><info:entry label="videoLabel"><info:type>video</info:type>
</info:entry></info:available-media>
</info:conference-description><info:conference-state>
<info:active>false</info:active></info:conference-state><info:users>
<info:user entity="13"><info:display-text>alex</info:display-text><info:endpoint entity="sip:[email protected]:5060"/>
</info:user><info:user entity="14"><info:display-text>user134</info:display-text><info:endpoint entity="sip:[email protected]:5080"/>
</info:user><xcon:join-handling>allow</xcon:join-handling>
</info:users><xcon:floor-information>
<xcon:floor-request-handling>confirm</xcon:floor-request-handling><xcon:conference-floor-policy><xcon:floor id="11"><xcon:media-label>audioLabel</xcon:media-label><xcon:max-floor-users>8</xcon:max-floor-users><xcon:moderator-id>13</xcon:moderator-id>
</xcon:floor><xcon:floor id="22">
<xcon:media-label>videoLabel</xcon:media-label></xcon:floor>
</xcon:conference-floor-policy></xcon:floor-information>
</info:conference-info>
Conference ID Title Conference state Password / PIN Allowed join modality Floor handling policy Participants info
User ID Nickname Signalling URI
Floor info Floor ID Associated media Moderator ID Max number of floor owners
XCON Protocols Signalling
SIP, H.323, IAX2, et al.
Floor Control BFCP
(Binary Floor Control Protocol)
Notification Xcon Event Package
Conference Control CCMP
(Centralized ConferencingManipulation Protocol)
University of Napoli highly active in this field
19
CCMP – Conference Control Conference CRUD Manipulation
Creation From a client-provided model or
by cloning system blueprints
Retrieval Update
User profiles, multimedia flow features (audio volume, video layout, …),...
Deletion
XML- based Carrying datamodel fragments
“Single verb HTTP + CCMP body” transport solution
20
BFCP – Floor Control Aimed at coordinating access to a set of shared resources
A “Floor” is a token, a temporary permission to access or manipulate a specific shared resource or set of resources
Standardized in RFC 4582 Identifiers (Conferences/Floors/Users) Floor Control Server Floor Control Participant Floor Chair
Negotiation of BFCP connections within SIP/SDP standardized in RFC 4583
Only existing implementation to date: COMICS/Ericsson
21
BFCP in action
22
1) Floor Request
2) Notify
3) Chair decision
4) Decision
5) Floor Granted/Denied
6) Notify
Chair Decision
1
2
3
4
5
6
Floor Request
Floor GrantedOr Denied
Decision Notify
Notify
MediaCtrl Working Group Defines an architecture in which an Application
Server can control a Media Server
A Media Server could be completely devoted toperform onerous online/offline operations on media flows: Media Processing
Mixing/Transcoding Playing/Recording Storing/Retrieving Detecting Tones (DTMF) Interactive Voice Response (IVR)/VoiceXML Text-to-Speech/Speech Recognition
RTP Streams Manipulation
Exploitable approach within conferencing frameworks
23
Client B
SIP SIP
Application
Server
Media
Server
SIP Control
Channel RTPRTP
SIPREC Working Group SIPREC - SIP-based RECording
Designing a protocol based on SIP-extension and a metadata model to deal withmultimedia session recording
A recorded session can play an important role in different scenarios E-learning, minutes taking, … Business analytics Media processing (online/offline) …
Some media processing actions may require separation between differentmedia streams E.g.:
Word-spotting, Speaker identification Emotion-detection …
24
SIPREC - Architecture Session Recording Client
(SRC) Logical entity able to
acquire the communicationsession to be recorded
Sends the recording media streams to the SRS
Session Recording Server (SRS) Records the received session
Recording Session (RS) ismade of: CS Multimedia Content CS Metadata
25
SIPREC - Requirements
“Active” recording The SRC proactively demands for registration No passive recording / sniffing / wiretapping
Session participants MUST be recording-aware Notification mechanisms are needed to inform participants
Start, Pause, Resume and Stop controls
Define a protocol between SRC and SRS to manage the recording session
26
SIPREC – Recording Metadata
Information that describes recorded media and the Communication Session (CS) to which they relate
They have to allow for:
Searching and filtering the archived session
Performing a “meaningful” playout
e.g., with correct synchronization between media streams
27
Recording Metadata Example<recording-metadata xmlns='...:siprec'>
<recording id=""> </recording>
<group id="" recording=""></group>
<session id=""group=""></session>
<participant id=""session=""></participant>
<stream id=""session=""></stream>
<extensiondata id="" parent="">
</extensiondata>
</recording-metadata>
Recording Metatada Elements Recording Session
Features: ID, recording type (selective, persistent), recording requestor (SRC or SRS), … One or mode associated CSG
Communication Session Group (CSG) Identified by a unique ID Group of one or more CS belonging to the same macro-session
Communication Session (CS) Features: ID, call termination reason, start time, end time, … Can involve one or more Participant elements and one or more Media Stream elements
Participant Features: name, AoR, participant role, …
Media Stream Features: start time, stop time, codec, SDP m-line reference, ..
Extension Data Envisioned the optional presenxe of additional data not specified as part of SIPREC
29
RTCWEB Working Group Real Time Communication on the World Wide Web
Goal: standardizing basic components enabling interactive rich communication using audio, video, collaboration, games, etc. between two peers' web-browsers Direct browser-to-browser communication
Nowadays several different methods exists, but they are not interoperable since they rely on incompatible browser plug-in platform
Collaboration with the W3C WEBRTC group : IETF architecture and requirements W3C Javascript API development
The idea: client-side applications, typically written in HTML and Javascript, interacting with web browser through a yet-to-be-defined API
30
RTCWEB IETF issues Identification and definition of network-related
aspects
Control protocols
Connection establishment and management
Connection-less transport
Selection of the most suitable encoders/decoders
Codec negotiation
New threats and security issues
Firewall and NAT traversal
Interoperation with legacy VoIP equipment, keeping the balance with practicality of browser deployment
31
CLUE Working Group
32
ControLling mUltiple streams for tElepresence
Telepresence systems provide high definition, high quality audio/video enabling a "being-there" /”immersive” experience
Use of multiple displays permitting life size imagereproduction using multiple cameras, encoders, decoders, microphones and loudspeakers
CLUE issues specifications for SIP-based conferencing systems
to enable communication of information about media streams necessary to make reasonable decisions about transmitting, selecting, and rendering media streams: Spatial relationships of cameras, displays, microphones, and
loudspeakers
Viewpoints - field of view/capture for camera/microphone/display/loudspeaker Senders and intermediate devices use them to understand how best to
compose streams for receivers
Receivers can know the characteristics of the received streams
Usage of the stream (presentation, document camera output, …)
Aspect ratio of cameras and displays
Sources selection receiver-driven
33
Meetecho Platform Web conferencing and on-line collaboration tool
Academic spin-off since Spring 2011 Developed by COMICS members and Unina students
Main features: Scalable and Performing Client-Server Solution IETF standards compliant (XCON, Media Ctrl) Open-source Java-based Desktop Client Mobile device access support
Android iPhone, iPad BlackBerry Symbian Windows Phone
Web browser access support (Meetecho Weblite)
34
Meetecho capabilitiesPSTN audio
Voice over IP audio
Live video
Audio & Video Moderation
Presentation delivery
Desktop/screen sharing
Text chat
Presence
File sharing
Whiteboard Sharing
Basic security
Remote Control
Session Recording
Polls & Surveys
Mobility support
Web access support
Audio/Video controls panel
Mirror On/Off
Pause Audio
Pause Video
Listen/Talk Volume
Selection of the desiredcontribution
Moderated Scenarios Realized by means of BFCP
The conference chair decides to grant or not audio and/or video floor to the requestor
The Meetecho server allows only granted users tocontribute their media within the conference media mix
Adaptive video layout depending on granted participantnumber
Slide sharing Slides are converted in background to a series of images made available on a
web server
All kind of presentation are supported Microsoft Office (ppt, pptx, doc, docx, xls, xlsx) OpenOffice (odp) Pdf Slides previously exported in image
formats as (gif, jpg, png, ecc.)
The presenter is the one who controlsslide progress
Slides can be sent to the whiteboardfor online collaborative modifications
Whiteboard Shared between conference participants
Allows to
Draw illustrative sketches
Modify slides
Insert clip-art
Save file modifications
Text Chat Based on XMPP (eXtensible Messaging and Presence
Protocol)
Allows instant messaging during conference sessions between conference participants
Features
Roster management
Presence info
Single file sharing
Conversation chronology
Desktop sharing Meetecho users can
Show to other participants their desktop activities
Render desktop control to other participants
Set visualization quality
Video mixing (1) Up to 16 parallel video streams in a single conference
Mixing strategy:
The mixer receives participant video contributions
Video streams are composed in a mosaico stream
The mixer sends to each participant the single mosaico stream and not each single video contribution That allows for
Constant load
Constant bandwidth consumption
Video mixing (2)
Server
MeetechoServer
Other tools overwhelm clients with as many flows as the contributing participants number…
…Meetecho mixes video streams and sends always a single flow to each users
Users
Users
User
User
Meetecho codecs
Supported audio codecs:
GSM
G.711 u-law
Supported video codecs:
H.323 Q-CIF (176 x 144 pixel)
H.323 CIF (352x288 pixel)
46
Managing Scalability Centralized case
A single focus for N users
Distibuted case Cascaded Conferencing
Load-balancing users between more interoperating focuses
Each focus is seen as a participant by the other focus
Orchestration of a set of XCON “clouds”
Experimental campaign showed performance benefits whit the increase of the participant number
47
Meetecho Session Recording Meetecho Session Metadata are encoded using SMIL
(Synchronized Multimedia Integration Language) Well known and established standard specification
Recordings are available in a standard fashion
Media timing – aware Enabling synchronized playout
Allowing media stream separation
A Recording Server is envisioned To receive all conference media streams and record them
To encode stream-related metadata info in the standard SMIL format
48
Server-side recording Client-side recording may be realized by means of ordinary desktop
recording softwares Available tools: Adobe Captivate, Wink, Istanbul, Byzanz,… Simple and fast Conferencing platfom independent
Client-side recording drawbacks: Only a single perspective is taken into account Recorded media “flattened” to a single, very large video Output files usually in proprietary formats
Flash, OGGTheora, …
Meetecho server-side recording approach uses standard formats is more suitable for multimedia session with several media and etherogeneous
devices involved
49
Recording audio & video Both Audio and video from
each participant are aggregated in the centralmixer the mixer can record and
store them locally or on a remote storage
Transcoding, adaptation and other operation may be carried out on the streams
The output is a single file containing both audio and video (AVI file with specific codecs)
Mixer UA-BUA-A RTPRTP
XCON AS
Recorder
MEDIACTRL
SIP SIP
RTP UA-A(Rx + Tx)
RTP UA-B(Rx + Tx)
Recording IM & events Chat contributions are
saved together withtimestamps and info on the employed IM protocol
51
Recorder UAC-BUAC-AB (IRC)
‘10:11:26 – Hey C’
A (XMPP)
UAC-C
C (MSRP)
‘10:11:30 – Hello’
‘10:11:24 – Hi!’
10:11:24 <User C> Hi!
10:11:26 <User A> Hey C
10:11:30 <User B> Hello
[…]
[…]
Textual event notificationmay be treated as well Slide changes
Join /leave events
Poll announcement
Recording slide presentations Presentations are considered as simple slideshows of
static images
Slide change events are triggered and notified via XMPP, registered as text
Slides are registered as a series of images togetherwith metadata info (associated events)
52
Recording whiteboarding
53
Whiteboard Server
UAC-BUAC-AB (XMPP)
10:10:56 – circle
A (XMPP)
UAC-C
C (XMPP)
10:12:30 – text
10:11:20 – line
10:10:56 <User A> circle
10:11:20 <User C> line
10:12:30 <User B> text
[…]
[…]
Putting It All Together: SMIL Standard XML-based markup language for describing
presentations involving heterogeneous media
SMIL metadata file
<head>
Generic metadata and layout composition details
<body>
References to media to be included (URL)
Layout regions info
Timing info
54
Media metadata SMIL allows for both standard and non-standard media to
be included in a presentation
Standard media streams have dedicated standard tags in the media part of the <body> <video> the A/V Meetecho recording AVI file
<textstream> IM, event notifications, converted in Real-Time Text docs
<img> slides
Non-standard media are included in the <ref> tag whiteboarding case
55
Playing out the recording (1)
57
Sessiondatabase
Web server
SMIL-enabled player
1. START
7. SHOW
2. get conf45.smil
video
text
slides
whiteboard
3. get audio/video
4. get RealText file
5. get slide images
6. get whiteboard packets
Playing out the recording (2) General approach:
1) Retrieval of the SMIL file (HTTP or out-of-bandmechanism)
2) SMIL file is passed to a compliant media player preparyingthe presentation layout
3) The player retrieves and reproduces each supported media
SMIL players examples : RealPlayer, Ambulant,… Standard stream support
Don’t play Meetecho whiteboarding…
Meetcho players!
58
Playing out the recordings (3) 3 options
1) Watch it online by means of a JNLP-based custom player
2) Watch it online withHTML5
3) Download the registration and watch it offline(jar file)
59
Session recordings available at:https:\\ietf.conf.meetecho.com
60