February 2000 1

School of Computing ScienceSchool of Computing ScienceSimon Fraser University, CanadaSimon Fraser University, Canada

Rate-Distortion Optimized Streaming of Fine-Grained Scalable Video Sequences

Mohamed Hefeeda & ChengHsin Hsu

2 February 2007

2

MotivationsMotivations

Multimedia streaming over the Internet is becoming very popular- More multimedia content is continually created

- Users have higher network bandwidth and more powerful computers

Users request more multimedia content

And they look for the best quality that their resources can support

3

Motivations (cont’d)Motivations (cont’d)

Users have quite heterogeneous resources (bandwidth)- Dialup, DSL, cable, wireless, …, high-speed LANs

To accommodate heterogeneity scalable video coding:

Layered coded stream- Few accumulative layers

- Partial layers are not decodable

Fine-Grained Scalable (FGS) coded stream- Stream can be truncated at bit level

4

Motivations (cont’d)Motivations (cont’d)

Goal: Optimize quality for heterogeneous receivers

In general setting- FGS-coded streams

- Multiple senders with heterogeneous bandwidth and store different portions of the stream

Why multiple senders?- Required in P2P streaming:

• Limited peer capacity and Peer unreliability

- Desired in distributed streaming environment:• Disjoint network path Better streaming quality

5

Our Optimization ProblemOur Optimization Problem

Assign to each sender a rate and bit range to transmit such that the best quality is achieved at the receiver.

Consider a simple example to illustrate the importance of this problem

6

Example: Different Streaming SchemesExample: Different Streaming Schemes

Non-scalable Layered

7

Example: Different Streaming SchemesExample: Different Streaming Schemes

FGS Scalable Optimal FGS Scalable

8

Problem FormulationProblem Formulation

First: single-frame case- Optimize quality for individual frames

Then: multiple-frame case- Optimize quality for a block of frames

- More room for optimization

9

Input ParametersInput Parameters

T : fixed frame period

n : number of senders

bi : outgoing bandwidth of sender i

bI : incoming bandwidth of receiver

si : length of (contiguous) bits held by sender i

Assume s1 <= s2 <= …… <= sn

10

Allocation: A = {(Δi , ri) | i=1, 2, ……, n}

- Δi : number of bits assigned to i

- ri : streaming rate assigned to i

Specifies:

- Sender 1 sends range [0, Δ1 -1] at rate r1

- Sender 2 sends range [Δ1 , Δ1+Δ2 -1] at rate r2

- …

- Sender i sends range at rate ri

OutputsOutputs

]1,[1

1

1

i

tt

i

tt

11

Integer Programming ProblemInteger Programming Problem

Minimize distortion

Subject to:

- on-time delivery

- assigned range is available

- assigned rate is feasible

- Aggregate rate not exceeds receiver’s incoming BW

12

How do we Compute Distortion?How do we Compute Distortion?

Using Rate-Distortion (R-D) models- Map bit rates to perceived quality

- Optimize quality rather than number of bits

Approaches to construct R-D models- Empirical Models: Many empirical samples expensive

- Analytic Models: Quality is a non-linear function of bit rate, e.g., log model [Dai 06] and GGF model [Sun 05]

- Semi-analytic Models: A few carefully chosen samples, then interpolate, e.g., piecewise linear R-D model [Zhang 03]

Detailed analysis of R-D models in [Hsu 06]

13

Within each bitplane, approximate R-D function by a line segment

Line segments of different bitplanes have different slopes

The Linear R-D ModelThe Linear R-D Model

14

Visual Validation of Linear R-D ModelVisual Validation of Linear R-D Model

Mother & Daughter, frame 110

Foreman, frame 100

15

Rigorous Validation of Linear R-D ModelRigorous Validation of Linear R-D Model

Average error is less than 2% in most cases

16

Let yi be number of bits transmitted from bitplane i

Distortion is:

- d : base layer only distortion

- gi : slope of bitplane i

- z : total number of bitplanes

Using the Linear R-D ModelUsing the Linear R-D Model

z

hhh ygd

1

17

Integer Linear Programming (ILP) ProblemInteger Linear Programming (ILP) Problem

Linear objective function

Additional constraints

- number of bits transmitted from bit plane h does not exceed its size lh

- bits assigned to senders are divided among bitplanes

18

Solution of ILP is a Valid FGS StreamSolution of ILP is a Valid FGS Stream

Lemma 1: - An optimal solution for the integer linear program produces a

contiguous FGS-encoded bit stream with no bit gaps

Proof sketch- minimizing

- Since g1 < g2 < …… <gn<0 (line segment slopes),

- the ILP will never assign bits to yi+1 if yi is not full

z

hhh ygd

1

19

Solving ILP problem is expensive

Solution: Transform it to Linear Programming (LP) problem - Relax variables to take on real values

Objective function and constraints remain the same

Linear Programming RelaxationLinear Programming Relaxation

20

Solve LP

- Result is real values

Then, use the following rounding scheme for solution of the ILP

Efficient Rounding SchemeEfficient Rounding Scheme

21

Correctness/Efficiency of Proposed RoundingCorrectness/Efficiency of Proposed Rounding

Lemma 2 (Correctness) - Rounding of the optimal solution of the relaxed

problem produces a feasible solution for the original problem

Lemma 3 (Efficiency: Size of Rounding Gap) - The rounding gap is at most nT + n, where n is the

number of senders and T is the frame period

- Example: T=30 fps, n=30, the gap is 32 bits

- Indeed negligible (frame sizes are in order of KBs)

22

FGSAssign: Optimal Allocation AlgorithmFGSAssign: Optimal Allocation Algorithm

Solving LP (using Simplex method for example) may still be too much- Need to run in real-time on PCs (not servers)

Our solution: FGSAssign- Simple yet optimal allocation algorithm

- Greedy: Iteratively allocate bits to sender with smallest si (stored segment) first

23

Pseudo Code of FGSAssignPseudo Code of FGSAssign

1. Sort senders based on si, s1 ≤ s2 ≤ …… ≤ sn;

2. x0 = …… = xn = 0; Δ1 = …… = Δn = 0; ragg = 0;

3. for i = 1 to n do

4. xi = min(xi−1 + biT, si);

5. ri = (xi − xi−1)/T ;

6. if (ragg + ri < bI ) then

7. ragg = ragg + ri;

8. Δi = xi − xi−1;

9. else

10. ri = bI − ragg;

11. Δi = T × ri;

12. return

13. endfor

24

Optimality of FGSAssignOptimality of FGSAssign

Theorem 1- The FGSAssign algorithm produces an optimal

solution in O(n log n) steps, where n is the number of senders.

Proof: see paper

Experimentally validated as well.

25

Multiple-Frame OptimizationMultiple-Frame Optimization

Solve the allocation problem for blocks of m frames each

Objective: minimize total distortion in block

Why consider multiple-frame optimization?- More room for optimization

- Less computation overhead: solve the problem less often

26

Multiple-Frame Optimization: Why?Multiple-Frame Optimization: Why?

More room for optimization: higher quality and less quality fluctuation

27

Multiple-Frame OptimizationMultiple-Frame Optimization

Formulation (in the paper): - Straightforward extension to single-frame with lager

number of variables and constraints

- Computationally expensive to solve

Our Solution: mFGSAssign algorithm - Heuristic (close to optimal results)

- Achieves two goals: • Minimize total distortion in a block

• Reduce quality fluctuations among successive frames

28

mFGSAssign: High-Level DescriptionmFGSAssign: High-Level Description

1. Estimate a target distortion D that is feasible and achieves the two goals (binary search)

2. Compute for each frame f in the block its bit budget Bf

3. For each frame f, call FGSAssign to allocate Bf among senders

29

Computing Target DistortionComputing Target Distortion

Is bit budget enough to transmit all frames at distortion level D ?

- Du : distortion upper bound

- Dl : distortion lower bound

- D : distortion estimate

30

Efficiency of mFGSAssignEfficiency of mFGSAssign

Lemma 5- mFGSAssign terminates in O(m n log n) steps,

where n is the number of senders and m is the number of frames in a block

Much more efficient than linear programming approach

31

Experimental SetupExperimental Setup

Software used - MPEG-4 Reference Software ver 2.5

• Augmented to extract R-D model parameters

Algorithms implemented (in Matlab)- LP solutions using Simplex for the single-frame and

multiple-frame problems

- FGSAssign algorithm

- mFGSAssign algorithm

- Nonscalable algorithm for baseline comparisons

32

Experimental Setup (cont’d)Experimental Setup (cont’d)

Streaming scenarios- Four typical scenarios for Internet and corporate

environments

Testing video sequences - Akiyo, Mother, Foreman, Mobile (CIF)

- Sample results shown for Foreman and Mobile

33

Single Frame: Quality (PSNR)Single Frame: Quality (PSNR)

Foreman, Scenario I Mobile, Scenario III

Quality Improvement: 1--8 dB

FGSAssign is optimal

34

Multiple Frame: Quality (PSNR)Multiple Frame: Quality (PSNR)

Foreman, Scenario II Mobile, Scenario III

Scalable: higher improvement than single frame

mFGSAssign: almost optimal (< 1% gap)

35

Fluctuation ReductionFluctuation Reduction

Foreman, Scenario II Mobile, Scenario III

Small quality fluctuations in successive frames

36

Running TimeRunning Time

Foreman, Scenario I Foreman, Scenario IV

Small and stable running times for mFGSAssign, unlike mOPT (Simplex)

mFGSAssign can be used in real time

37

Reconstructed PicturesReconstructed Pictures

Nonscalable

mFGSAssign

38

ConclusionsConclusions

Formulated and solved the bit allocation problem for single and multiple frame cases

Nonlinear problem integer linear program- Using linear R-D model

Integer linear program linear program - Using simple rounding scheme

Efficient Algorithms - FGSAssign: Optimal and efficient

- mFGSAssign: close to optimal in terms of average distortion, reduces quality fluctuations, runs in real time

Significant quality improvements shown by our experiments

39

Thank You!Thank You!

Questions??

All programs/scripts/videos are available:

http://www.cs.sfu.ca/~mhefeeda