Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000
description
Transcript of Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000
Video Coding for Streaming Media Delivery on the InternetConklin, G. et al, 2000
Jake Wires
Motivation
Original standards designed for constant bit rate encoding
Constant bit rate encoding does not account for the channel variability inherent in many network applications
Large demand for streaming multimedia—over 350,000 hours of live content broadcast per week
History of Streaming Media Technologies
VivoActive 1.0HTTP server-client transactions via TCPH.263 video codec, G.723 audio codecMultiplexing protocol to combine audio &
videoVery simple
History: VivoActive
ProblemsHTTP’s lack of flow control and TCP’s
guaranteed delivery mechanisms resulted in large fluctuations of fragment delivery times
Necessitated large preroll buffer (5 – 20 sec)Rebuffering (filling empty buffer) diminished user
expereinceSession Control (i.e. f.f., rew.) difficult
History: RealAudio 1.0
Introduced in March 1995Relied on Progressive Networks
Architecture—a suite of dedicated network protocolsUsed 2 connections:
TCP-based session control connectionUDP-based audio transmission connection
History: RealAudio 1.0
Use of UDP-based transmission introduced possibility for lost, delayed, and out of order packets Automatic Repeat-Request mechanism allowed
clients to request retransmission of lost packets Frame interleaving technique minimized perceptual
damage Send packets of interleaved (rather than continuous)
audio frames Dropped packet results in multiple small disruptions rather
than single large disruption
History: RealVideo
RealVideo design features: Produced compressed data that can be streamed
at a fixed bit rate Non-uniform nature of video clips (scene changes,
transitions, etc.) results in unequal distribution of bits between frames when encoded at same level of distortion
To maintain fixed bit rate, encoder must introduce unequal distortion between frames, neglect encoding some frames, or both
Use of preroll buffer increases bit rate flexibility—only average rate must be kept constant
Variable-bit-rate (VBR) rate-control algorithm
History: RealVideo
Facilitated random access by inserting Intra frames
Employed unequal error protectionLarge video frame size precluded simple
interleaving techniques Instead, RealVideo combined forward error
correction codes for the most sensitive data with various error concealment mechanisms
History: RealSystem G2
SureStream technology improved adaptive streaming capabilities: A single SureStream file contains multiple streams
of the original content Different streams optimized for different channel
capacities Client of stream monitors connection performance,
dynamically requests optimal stream from server
Used IETF standardized RTP and RTSP protocols
History: Network Strategies
Multicasting reduces amount of redundant data transferred
Streaming media delivery networks allow for multiple, geographically scattered servers (achieved either by splitting or caching)
Dedicated networks guarantee bandwidth Multiple-access transmissions over Internet—
requested info sent from several different locations simultaneously
Streaming Media
Three modes in which video information can be streamed:Live broadcastingOn-demand streamingConferencing
Streaming Media: Live Broadcasting System Structure
Streaming Media: Live Broadcasting
Live video encoded on-the-fly, passed to server, which disperses compressed info to all splitters and/or clients Splitters are additional servers that can be used to
distribute server workload over a number of machines
Push splitting: initiated by server Pull splitting: initiated by a client connecting to a local
splitter; request propagated to nearest active splitter Multiple-access splitting: lower-tier splitters can connect to
multiple upper-tier splitters
Streaming Media: On-demand System Structure
Streaming Media: On-demand
Distribution system similar to live broadcasting system, except:No direct connection between encoder and
serverRemote proxy servers can cache popular
filesClients allowed to rewind, fast forward, etc.Server proxy transfers only initiated by client
Video Coding Problems
Channel capacity fluctuations Adaptive client-driven serving
Allow client to inform server of network performance Congestion control Error control: use bandwidth prescribed by congestion
control algorithm to minimize effects of packet loss
Dynamic prediction Combined with sufficient preroll buffer, a broad set of
statistical techniques can (possibly) be used to throttle transfer rate to improve user experience
Video Coding Problems
Random accessCan provide random access by inserting
I-frames at the boundaries of small video intervals
Reduces achievable compression ratesProcessing resources
Diverse range of client machinesLive encoding often done on computer with
limited resources
Video Coding Algorithms for Streaming
Original motion-compensated hybrid video coding algorithms produced single-rate encoded versions of input signals Inadequate for many network applications
Need scalable coding algorithms
Scalable Streaming Techniques
Simulcast: produce several, redundant, independently coded streams, each optimized for specific channel conditions Not suitable for splitting, where the combined
representation of all streams should be as small as possible
Scalable coding Based on spatio-temporal pyramid decomposition
of source signal Temporal, spatial, and Signal to Noise Ratio (quantizer
size) scalability modes
Scalable Streaming Techniques
Multiple-description coding for use in multiple-access systemsAchieved by replacing scalar quantization
with multiple-description quantization scheme
RealSystem 8
Adaptive channel encoding of streaming content to match bit rate with bandwidth and loss statisticsChannel statistics not available until time of
streaming, so server must manage adaptive strategies
Infeasible for server to encode on demand, so encoding techniques are required which allow server to efficiently choose between different bit rates
RealSystem 8
SureStream components:Adaptive Stream Management (ASM)SureStream file formatSource and channel coding algorithms
RealSystem 8
ASM Allows clients to communicate which type of
encoding should be ‘synthesized’ by server File format plug-in uses ASM rules to produce
various combinations of encoded streams from compressed media files
Rules expressed by fully programmable syntax Client subscribes to appropriate rule based on
channel characteristics ASM is format-independent
RealSystem 8
RealVideo 8 encoding module:
RealVideo 8 Algorithm
Input filters: remove noise and potential artifacts introduced by edits and conversions of video signal Remove low-energy spatial noise De-interlace analog signals to produce progressive
frames without “jaggy”-shaped artifacts Inverse telecine filter: remove redundant 5.97 frames
introduced by NTSC video
RealVideo 8 Algorithm
Spatial resampler: downscale input frames to spatial resolutions suitable for streaming
Codecs: motion-compensated hybrid scheme Rate control determines which frames to
encode Low bit rates: perceived quality impacted by
relationship between framerate and frame quality High bit rates: must maintain framerate, want to
maintain appearance of uniform high quality Parameterized preroll buffer size Rate control important in providing resiliency and
random access
RealVideo 8 Algorithm
CPU scalability control module: throttles complexity of different components of algorithm to maintain best quality level for encoding live presentations
RealVideo 8 Algorithm
Client-side video postprocessing: filters aimed at improving subjective quality of decoded signals Frame rate upsampler: special temporal filter that
attempts to interpolate intermediate frames Especially useful when many original frames were
regularly skipped to decrease bit rate of encoding
Pointers
Helix Community “The Helix community is a collaborative
effort among Real, independent developers, and leading companies to extend the Helix DNA™ platform, the first open multi-format platform for digital media creation, delivery and playback.”
https://helixcommunity.org/
Conclusion
Streaming multimedia poses many challenges to video encoders Variability of channel performance High server workloads
Need to adapt coded bit rate to match channel capacities; must be done efficiently and on the fly
“RealSystem 8 provides an open and extensible platform, capable of accommodating various future needs of streaming media infrastructure on the Internet, and in particular, new demands for improved video-coding techniques.”