41570904 Remote Media Immersion
-
Upload
shreejith-nair -
Category
Documents
-
view
19 -
download
0
description
Transcript of 41570904 Remote Media Immersion
-
5/24/2018 41570904 Remote Media Immersion
1/17
2010
Remote
Medi a
Imm
ersion
The Remote Media Immersion (RMI) system is the resultof a unique blend of multiple cutting-edge mediatechnologies to create the ultimate digital mediadelivery platform. The main goal is to provide animmersive user experience of the highest quality. RMI
encompasses all end-to-end aspects from mediaacquisition storage transmission up to their finalrendering. !pecifically the "ima streaming mediaserver delivers multiple high band#idth streamstransmission error and flo# control protocols ensuredata integrity and high-definition video combined #ithimmersive audio provide highest quality rendering. TheRMI system is operational and has been successfullydemonstrated in small and large venues. Relying on thecontinued advances in electronics integration andresidential broadband improvement RMI demonstratesthe future of on-demand home entertainment
-
5/24/2018 41570904 Remote Media Immersion
2/17
ABSTRACT
The Remote Media Immersion (RMI) system is the result of a uniqueblend of multiple cutting-edge media technologies to create the ultimate digital
media delivery platform. The main goal is to provide an immersive user
experience of the highest quality. RMI encompasses all end-to-end aspects
from media acquisition storage transmission up to their final rendering.
!pecifically the "ima streaming media server delivers multiple high
band#idth streams transmission error and flo# control protocols ensure data
integrity and high-definition video combined #ith immersive audio provide
highest quality rendering. The RMI system is operational and has been
successfully demonstrated in small and large venues. Relying on the continuedadvances in electronics integration and residential broadband improvement
RMI demonstrates the future of on-demand home entertainment.
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
3/17
INTRODUCTION
The charter of the Integrated Media !ystems &enter (IM!&) at the
'niversity of !outhern &alifornia ('!&) is to investigate ne# methods and
technologies that combine multiple modalities into highly effective immersivetechnologies applications and environments. ne of the results of these
research efforts is the Remote Media Immersion (RMI) system. The goal of the
RMI is to create and develop a complete aural and visual environment that
places a participant or group of participants in a virtual space #here they can
experience events that occurred in different physical locations. RMI technology
can effectively overcome the barriers of time and space to enable on demand
the realistic recreation of visual and aural cues recorded in #idely separated
locations.
The focus of the RMI effort is to enable the most realistic recreation of
an event possible #hile streaming the data over the Internet. Therefore #e
push the technological boundaries much beyond #hat current video-on-demand
or streaming media systems can deliver. s a consequence high-end rendering
equipment and significant transmission band#idth are required. The RMI
pro*ect integrates several technologies that are the result of research efforts at
IM!&. The current operational version is based on four ma*or components that
are responsible for the acquisition storage transmission and rendering of high
quality media.
STAGES OF RMI
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
4/17
Acquisition of high-quality !"ia st#!as
This authoring component is an important part of the overall chain to
ensure the high quality of the rendering result as experienced by users at a later
time. s the saying +garbage in garbage out, implies no amount of qualitycontrol in later stages of the delivery chain can mae up for poorly acquired
media.
R!al-ti! "igital sto#ag! an" $lay%ac& of ulti$l! in"!$!n"!nt st#!as
"ima !calable !treaming Media rchitecture provides real-time
storage retrieval and transmission capabilities. The "ima server is based on a
scalable cluster design. $ach cluster node is an off-the-shelf personal computer
#ith attached storage devices and for example a ast or /igabit $thernet
connection. The "ima server soft#are manages the storage and
net#or resources to provide real-time service to the multiple clients that are
requesting media streams. Media types include but are not limited to M0$/-1
at 2T!& and 34T5 resolutions multichannel audio (e.g. 67.1 channel
immersive audio) and M0$/-8.
'#otocols fo# synch#oni(!") !ffici!nt #!alti! t#ansission of ulti$l!
!"ia st#!as
selective data retransmission scheme improves playbac quality #hilemaintaining realtime properties. flo# control component reduces net#or
traffic variability and enables streams of various characteristics to be
synchroni9ed at the rendering location. Industry standard net#oring protocols
such as Real-Time 0rotocol (RT0) and Real-Time !treaming 0rotocol (RT!0)
provide compatibility #ith commercial systems.
R!n"!#ing of i!#si*! au"io an" high #!solution *i"!o
Immersive audio is a technique developed at IM!& for capturing the
audio environment at a remote site and accurately reproducing the completeaudio sensation and ambience at the client location #ith full fidelity dynamic
range and directionality for a group of listeners (6: channels of uncompressed
linear 0&M at a data rate of up to 6;.:Mb Mb
-
5/24/2018 41570904 Remote Media Immersion
5/17
si9es may range from a single person or family at home to a large venue seating
hundreds. or the visual streams #e decided that #e required at least high-
definition (34) resolution as defined by T!&6. The highest quality T!&
modes are either 6?17 @ 67=7 pixels at an interlaced frame rate of 1?.?; per
second or 61=7 @ ;17 pixels at a progressive frame rate of >?.?8 per second.or the audio rendering #e rely on the immersive audio technology developed
at IM!& #hich utili9es a 67.1 channel playbac system. The rendering
capabilities of immersive audio goes much beyond current stereo and >.6
channel systems. The combination of 67.1 channels of immersive audio and
high-definition video is the next step in audio-visual
fidelity. $ach presentation session retrieves and plays bac at least one high-
definition visual and one immersive aural stream in synchroni9ation. 2ote that
this choice #as imposed by the available media content and is not an inherent
limitation in the "ima design. The streams are stored separately on the server
for t#o reasons. irst the RMI system is designed to be extensible such that
additional video or other streams may become part of a presentation in the
future. !econd allo#ing streams to be separately stored enables RMI to
retrieve different components of a presentation from different server locations.
The final fine-grained synchroni9ation is achieved at the client side. The on-
demand delivery of the streams that form anRMI presentation is enabled
through our streaming media architecture called "ima. Aith features such as
scalability multi-stream synchroni9ation transmission error and flo# control it
is uniquely suited for RMI-style media delivery.
DEI+ER. OF /IG/-RESOUTION MEDIA
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
6/17
n important component of delivering isochronous multimedia over I0
net#ors to end users and applications is the careful design of a multimedia
storage server. The tas of such a server is t#ofoldB (6) it needs to efficiently
store the data and (1) it must schedule the retrieval and delivery of the data
precisely before it is transmitted over the net#or. RMI relies on our "imastreaming media architecture. "ima has been designed from the ground up to
be a scalablemedia delivery platform that can support multiple very high
band#idth streams.
igure sho#s the server cluster architecture #hich is designed to harness
the resources of many nodes and many dis drives per node concurrently.
ur current implementation the server consists of a 8- #ay cluster of rac-
mountable 0&s running Red 3at Cinux ho#ever larger configurations arepossible to increase the number of concurrent RMI sessions supported. $ach
cluster node is attached to a local net#or s#itch #ith a ast or /igabit
$thernet lin. The nodes communicate #ith each other and send the media data
via these net#or connections. The local s#itch is connected to both a A2
bacbone (to serve distant clients) and a C2 environment #ith
local clients. &hoosing an I0 based net#or eeps the per-port equipment cost
lo# and is immediately compatible #ith the public Internet. The media data is
stored in segments (called blocs) across multiple say four high performance
hard dis drives. There are t#o basic techniques to assign the data blocs of amedia ob*ect D in a load balanced manner D to the magnetic dis drives that
form the storage systemB in a round-robin sequence or in a random manner.
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
7/17
"ima uses a pseudo-random data placement in combination #ith a deadline-
driven scheduling approach.
This combination enables short startup latencies and can easily support
multimedia applications #ith non sequential data access patterns including
variable bit rate (5ER) video or audio and interactive operations such pauseand resume etc. It also enables efficient data reorgani9ation during system and
storage scaling. or the delivery of media data #e tae full advantage of this
high performance storage layout.
E##o# Cont#ol 0ith S!l!cti*! R!t#ansissions f#o Multi$l! S!#*!# No"!s
ne of the characteristics of continuous media streams is that they require datato be delivered from the server to a client location at a predetermined rate. This
rate may vary over time for streams that have been compressed #ith a variable
bit rate (5ER) media encoder. 5ER streams enhance the rendering quality
ho#ever they #ill generate bursty traffic on a pacet s#itched net#or such as
the Internet. This in turn can easily lead to pacet loss due to congestion. !uch
data loss adversely affects compressed audio
and video streams because much of the temporal or spatial redundancy in the
data has already been removed by the compression algorithm. urthermore
important data such as audio
-
5/24/2018 41570904 Remote Media Immersion
8/17
the transmission bet#een the server and a rendering location. The "ima cluster
architecture taes advantage not only of the distributed storage resources
among the multiple nodes but also of the multiple net#or connections that
lin all the nodes together. Media data is transmitted via the real-time protocol
(RT0) encapsulated in connection-less '40 datagrams. To avoid trafficbottlenecs each node transmits the data blocs that it holds directly to the
clients via RT0. 3ence each client #ill receive RT0 data pacets fromeach
server node#ithin the cluster. 2ote that the current Internet infrastructure #as
not designed for streaming media and provides only best-effort pacet delivery.
The RMI test environment. server is located at the Information !ciences Institute in rlington 5
#hile the rendering is performed in Cos ngeles.
Therefore RT0
-
5/24/2018 41570904 Remote Media Immersion
9/17
source I0 address the client can easily establish the complete set of server
nodes. nce a server receives a request it checs #hether it holds the pacet
and either ignores the request or performs a retransmission. &onsequently this
approach #astes net#or band#idth and increases server load.
Unicast R!t#ansissions The second more efficient and scalable method of
sending retransmission requests requires that the unique server node that holds
the missing pacet be identified. This could be accomplished in several #ays.
or example the client could reproduce the pseudo-random number sequence
that #as originally used to place the data cross multiple server nodes. This
approach has several dra#bacs. irst identical algorithms on both the clients
and the servers must be used at all times. If
the server soft#are is upgraded then all clients must be upgraded immediately
too. The logistics of such an undertaing can be daunting if the clients are
distributed among thousands of end users. !econd during
scaling operations the number of server nodes or dis drives changes and hence
ne# parameters need to be propagated to the clients immediately. ther#ise
the server nodes #ill be misidentified. Third if for any reason the client
computation is ahead or behind the server computation (e.g. the total number
of pacets received does not match the number of pacets sent) then any future
computations #ill be #rong. This could potentially happen if the client has only
a limited memory and pacets arrive sufficiently out-of-sequence. more
robust approach is as follo#s. The client determines the server node from
#hich a lost RT0 pacet #as intended to be delivered by detecting gaps innode-specific pacet sequence numbers. Ae term these local sequence numbers
(C!2) as opposed to theglobal sequence number (/!2) that orders all pacets.
lthough this approach requires pacets to contain a node-specific sequence
number along #ith a global sequence number the clients require very little
computation to identify and locate missing pacets.
Cli!nt-S!#*!# A"a$ti*! Flo0 Cont#ol
RMI relies on the efficient transfer of multimedia streams bet#een aserver and a client. These media streams are captured and displayed at a
predetermined rate. or example video streams may require a rate of 18 1?.?;
G7 >?.?8 or :7 frames per second. udio streams may require 88677 or
8=777 samples per second. n important measure of quality for such
multimedia communications is the precisely timed playbac of the streams at
the client location. chieving this precise playbac is complicated by the
popular use of variable bit rate (5ER) media stream compression. 5ER
encoding algorithms allocate more bits per time to complex parts of a stream
and fe#er bits to simple parts to eep the visual and aural quality at nearconstant levels. or example an action sequence in a movie may require more
bits per second than the credits that are displayed at the end. s a result
different transmission rates may be required over the length of a media stream
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
10/17
to avoid starvation or overflo# of the client buffer. s a contradictory
requirement #e #ould lie to minimi9e the variability of the data transmitted
through a net#or. 3igh variability produces uneven resource utili9ation and
may lead to congestion and exacerbate display disruptions. The focus of the
RMI flo# control mechanism is on achieving high quality media playbac byreducing the variability of the transmitted data and hence avoiding display
disruptions due to data starvation or overflo# at the client. Ae designed a novel
technique that ad*usts the multimedia traffic based on an end to end rate control
mechanism in con*unction #ith an intelligent buffer management scheme.
'nlie previous approaches #e consider multiple signaling thresholds and
adaptively predict the future band#idth requirements. Aith this Multi-
Threshold Flow Control (MT&) scheme 5ER streams are accommodated
#ithout a priori no#ledge of the stream bit rate. urthermore because the
MT& algorithm encompasses server net#or and clients it adapts itself to
changing net#or conditions. 4isplay disruptions are minimi9ed even #ith fe#
client resources (e.gB a small buffer si9e). The MT& protocol is currently
implemented in our "ima streaming media server and clients. This allo#s us to
measure and verify its effectiveness in an end-to-end application such as RMI.
The design of a high performance rate control algorithm #as motivated in part
by our continuous media server implementation. Ae required an adaptive
technique that could #or in a real-#orld dynamic environment #ith minimal
prior no#ledge of the multimedia streams to be served. Ae identified the
follo#ing desirable characteristics for the algorithmB
Onlin! o$!#ationB Required for live streaming and also desirable for storedstreams.
Cont!nt in"!$!n"!nc!B n algorithm that is not tied to any particularencoding technique #ill continue to #or #hen ne# compression algorithms
are introduced.
Minii(ing f!!"%ac& cont#ol signalingB The overhead of online
signaling should be negligible to compete #ith offline methods that do not needany signaling.
Rat! soothingB The pea data rate as #ell as the number of rate changesshould be lo#ered compared #ith the original unsmoothed stream. This #ill
greatly simplify the design of efficient real-time storage retrieval and transport
mechanisms to achieve high resource utili9ation. &onsidering these ob*ectives
#e designed our novel Multi-Threshold lo# &ontrol (MT&) technique.
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
11/17
It distinguishes itself from previously proposed algorithms by incorporating the
follo#ingB (6) a multi-threshold buffer model (1) a consumption prediction
component and (G) a modular rate change computation frame#or in #hich
ne# consumption prediction algorithms and feedbac delay estimation
algorithms can easily be incorporated.
RENDERING
The rendering side of the RMI system is composed of several parts. The video
and audio streams are received over a sufficiently fast I0 net#or connection
on a personal computer running t#o instances of the "ima playbac soft#are.
This media player is structured into several components and only one of them
interfaces #ith the actual media decoder. This allo#s us to plug-in multiple
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
12/17
soft#are and hard#are decoders and hence support various media types. or
RMI one of the players interfaces #ith a &ine &ast
34 M0$/-1 decompression board manufactured by 5ela Research. This
decoder accepts M0$/-1 compressed high definition video at data rates in
excess of >7Mb7 Mb
-
5/24/2018 41570904 Remote Media Immersion
13/17
deliver sound over loudspeaers these methods also require precise
cancellation of the crosstal signals resulting from the opposite side
loudspeaer in order to deliver the desired sound to each ear. s a result they
#or #ell only for a single listener in a precisely-defined position.
Multichannel surround sound systems have also been developed that use threefront channels and t#o surround channels to provide a sense of envelopment
for multiple listeners. lthough these systems may #or #ell in accompanying
movies they are not suitable for immersive systems. The main reason is that
they leave significant spatial gaps in the a9imuth plane (e.g. at ?7 degrees to the
side of the listeners) and provide no coverage in the median plane (no elevation
cues). In order to minimi9e locali9ation errors the number of loudspeaers must
increase linearly #ith the #idth of the listening area. listener that moves *ust
a fe# cm from the designated listening spot is sub*ected to high imaging
distortion and no longer experiences the correct locali9ation cues. This problem
can be addressed by increasing the number of loudspeaers. It is no#n from
the psychoacoustics literature that human listeners can locali9e sounds very
precisely in the front hemisphere and less precisely to the sides and in the rear
hemisphere.
Therefore locali9ation errors from the front channels that arise from offsets in
the listener position from the desired center location #ill be particularly
evident. In our implementation #e allocate five channels to the front hori9ontal
plane by augmenting the traditional three front loudspeaers #ith t#o
additional #ide channels. The advantages of this are t#ofoldB (i) locali9ation
errors in the front hemisphere are reduced and (ii) the #ide channels providesimulated side-#all reflection cues that have been sho#n to increase the sense
of spatial envelopment. In addition to the > front channels a rear surround
channel is added to fill the gap directly behind the listener. $levation
information is reproduced from t#o height channels placed above the front left
and right loudspeaers. $arly experiments that #e have performed #ith this
configuration have sho#n that these 67 channels significantly increase the
sense of locali9ation and envelopment for listeners. ma*or challenge in multi-
listener environments arises from room acoustical modes particularly in small
rooms that cause a significant variation in the responses measured at differentlistener locations. Responses measured only a fe# cm apart can vary by up to
6>-17 dE at certain frequencies. This maes it difficult to equali9e an audio
system for multiple
simultaneous listeners. urthermore it maes it impossible to render a remote
performance for a large audience and have them experience the sound exactly
as it is being produced at the remote event. 0revious
methods for room equali9ation have utili9ed multiple microphones and spatial
averaging #ith equal #eighting. This approach tends to smooth out large
variations due to standing #aves but does not tae into account the effects ofroom modes that for some frequencies may be more concentrated in one region
and not in another. Ae have developed a ne# method that derives a
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
14/17
representative room response from several room responses that share similar
characteristics. This response can then be used to equali9e the entire class of
responses it represents.
DISTRIBUTED IMMERSI+E 'ERFORMANCE
DEMONSTRATION
ur current #or is a real-time multi-site interactive specific reali9ation of the
immersive environment called 4istributed Immersive 0erformance (4I0). The
Remote Media Immersion experiments and demonstrations primarily utili9ed
uni-directional transmissions and off-line audio and video processing. The 4I0
pro*ect leverages the RMI technology and extends its capabilities to multi-directional multi-participant and real-time interaction in synchronous and
collaborative music performance. The goal of the 4I0 pro*ect is to develop the
technology for performances in #hich the participants - subsets of musicians
the conductor and the audience - are in different physical locations and are
interconnected by high fidelity multichannel audio and video lins as sho#n in
igure.
There are generally t#o classes of participants #ith different sets of
ob*ectives and requirements. Ae label the participants acti*! or $assi*!depending on their level of interaction and the latency they can tolerate. orexample in a tele-conference application all participants are generally active.
or a performance event there is a mix of active participants (musicians in a
concert players in a sports event panelists at a meeting #hose primary actions
are those of doing of physically engaging in the activity) and passive
participants (the audience #hose primary actions are
seeing and listening).
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
15/17
The Remote Media Immersion system is the next step in audio-visual
fidelity for streaming media delivered on-demand over the Internet. The goal of
the RMI is to push the boundaries beyond #hat is currently available in any
commercial system or other research prototype. Its emphasis is on the highest
quality of audio-visual experiences and most realistic immersive rendering. Toachieve goal #e #ere faced #ith a number of unique challenges and had to
design novel techniques to mae RMI a reality. In this report #e presented the
error and flo# control algorithms as #ell as the immersive audio aspects of
RMI. The current RMI setup is out of reach for most home users. or
#idespread adoption a number of technological advances #ill be necessary
#hich subsequently #ill lead to more affordable prices and mae the RMI
system feasible for high-end home use. or example #e are currently using the
M0$/- 1 algorithm at a lo# compression ratio to achieve our target visual
quality. n improvement in compression algorithms and affordable hard#are
availability #ill most certainly mae the same quality available at lo#er bit
rates in the future (M0$/-8 is a candidate here). 3ence #e envision a cross-
over point in the next fe# years #hen the band#idth required for RMI is belo#
the bit rates that ne# high-speed residential broadband technologies can
deliver (for example very high-speed 4!C or 54!C). In the more distant
future one might envision a portable device #ith a high resolution screen and a
personal immersive audio system. or a single listener it is possible to achieve
very enveloping audio rendered #ith only t#o speaers as long as the position
and shape of the listenerFs ears are no#n. The 04of the future #ill at some
point have enough processing capabilities to combine visual tracing #ithadaptive rendering of both audio and video.
CONCUSION
The Remote Media Immersion system is the next step in audio-visual fidelityfor streaming media delivered on-demand over the Internet. The goal of the
RMI is to push the boundaries beyond #hat is currently available in any
commercial system or other research prototype. Its emphasis is on the highest
quality of audio-visual experiences and most realistic immersive
rendering. To achieve our goal #e #ere faced #ith a number of unique
challenges and had to design novel techniques to mae RMI a reality. In this
report #e presented the error and flo# control algorithms as #ell as the
immersive audio aspects of RMI. The current RMI setup is out of reach for
most home users. or #idespread adoption a number of technological advances#ill be necessary #hich subsequently #ill lead to more affordable prices and
mae the RMI system feasible for high-end home use. or example #e are
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
-
5/24/2018 41570904 Remote Media Immersion
16/17
currently using the M0$/- 1 algorithm at a lo# compression ratio to achieve
our target visual quality. n improvement in compression algorithms and
affordable hard#are availability #ill most certainly mae the same quality
available at lo#er bit rates in the future (M0$/-8 is a candidate here). 3ence
#e envision a cross-over point in the next fe# years #hen the band#idthrequired for RMI is belo# the bit rates that ne# high-speed residential
broadband technologies can deliver (for example very high-speed 4!C or
54!C). dditionally #e are #oring to#ards simplifying and automating the
setup and calibration procedures that are currently necessary. !ubsequently
these signal processing algorithms can be incorporated into multichannel home
receivers at moderate cost. In the more distant future one might envision a
portable device #ith a high resolution screen and a personal immersive audio
system. or a single listener it is possible to achieve very enveloping audio
rendered #ith only t#o speaers as long as the position and shape of the
listenerFs ears are no#n. The 04 of the future #ill at some point have
enough processing capabilities to combine visual tracing #ith adaptive
rendering of both audio and video.
REFERENCES
&oncert 3all coustics ". ndo !pringer 5erlag 2e# "or
Hournal of the coustical !ociety of merica
&onference on Multimedia &omputing and 2et#oring (MM&2)
!anta &lara &alifornia
R. immermann J. u &. !hahabi !.-". 4. "ao and 3. hu. "imaB
4esign and $valuation of a !treaming Media !ystem for Residential
Eroadband !ervices
###.ho#stuff#ors.com
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
http://www.howstuffworks.com/http://www.howstuffworks.com/ -
5/24/2018 41570904 Remote Media Immersion
17/17
###.docstoc.com
$lectronics $ngineering !eminar Topic % Instrumentation !eminar
http://www.docstoc.com/http://www.docstoc.com/