Rate-Distortion Optimized Motion Estimation for Error Resilient Video Coding

Hua Yang and Kenneth Rose

Signal Compression Lab

ECE Department

University of California Santa Barbara, USA

Mar. 2005

Mar. 2005 ICASSP 2005 2

Outline

Motion estimation (ME) for coding efficiency

– Conventional ME

– Rate-constrained ME & rate-distortion (RD) optimized ME

Motion estimation for error resilience

Proposed end-to-end distortion based RDME

– Intuition behind

– End-to-end distortion analysis

Simulation results

Conclusions

Mar. 2005 ICASSP 2005 3

Motion Estimation for Coding Efficiency

Motion compensated prediction (MCP)– To remove inherent temporal redundancy of video signal – Both the motion vector and the prediction residue are encoded.

Coded frame n-1 Original frame n

Mar. 2005 ICASSP 2005 4

21ˆminmin

MBi

mvin

in

mvres

mvffD

Motion Estimation for Coding Efficiency

Conventional motion estimation

– ME Criterion: minimize prediction residue

• Ignoring the motion vector bit-rate cost

Mar. 2005 ICASSP 2005 5

• However, not yet the ultimate rate-distortion optimization for the best overall coding performance.

Motion Estimation for Coding Efficiency

Motion estimation in low bit rate video coding– In low bit rate video coding, motion vectors may occupy a

significant portion of total bit rate.

– Efficient bit allocation between motion vector and prediction residue coding is necessary for better overall coding efficiency.

mvresmv

RD min : Lagrange multiplier

– Rate-constrained motion estimation

Mar. 2005 ICASSP 2005 6

headerresmvQPmv

RRRD },{

min

Motion Estimation for Coding Efficiency

Motion estimation for low bit rate video coding (cont’d)

– Rate-distortion optimized motion estimation (RDME)

QMBi

in

in

MBi

in

in DeeffD

22ˆˆ

– Some references• [Girod `94] Theoretical analysis of rate-constrained ME

• [Sullivan `98] Summary of rate-constrained ME

• [Chung `96] Low complexity RDME for each MB using RD modeling

• [Schuster `97] Joint RDME for multiple MB’s

Mar. 2005 ICASSP 2005 7

Motion Estimation for Error Resilience

In the presence of packet loss:– Packet loss & error propagation

• Internet – no QoS guarantee

Wireless – inherent error-prone channel

• Error propagation due to MCP

No mv for Inter-mode!

– Error resilient video coding• RD optimization with end-to-end distortion

• Coding mode selection: {Intra/Inter, QP}

Error resilience via motion compensation – Multi-frame motion compensation (MFMC) [Budagavi `01]

– Reference picture selection (RPS) [H.263+]

– Error resilient rate-constrained ME [Wiegand `00]

Not comprehensively attack the RD optimization problem!

Mar. 2005 ICASSP 2005 8

Motion Estimation for Error Resilience

We propose end-to-end distortion based RDME

[accounting for packet loss]

The exact RD optimal ME solution for error resilience

Critical:

accurate pixel-level end-to-end distortion estimation• Build on: recursive optimal per-pixel estimate (ROPE) [R. Zhang, S. Regunathan, and K. Rose `00]

Mar. 2005 ICASSP 2005 9

Conventional motion estimation completely ignores the error resilience information.

– This error resilience information should be exactly considered for each pixel.

Proposed RDME

Intuition for “error resilience via ME”

For coding efficiency For error resilience

I

I

P1

P3

P4

P2

P2

P1

P1

Best trade-off

Mar. 2005 ICASSP 2005 10

– DEP is explicitly affected by mv, whose minimization favors mv’s that point to reference areas with less encoder-decoder mismatch.

Proposed RDME

ROPE-based end-to-end distortion analysis

Error concealment

Error propagated distortion

ECQEP

ECMBi MBi

in

in

mvin

mvin

MBi

in

in

in

mvin

in

MBi

in

in

DpDpDp

DpeeffEp

ffEpeffEp

ffEDE

11

ˆ})~ˆ{(1

})~

{(})ˆ~

{(1

})~

{(}{

2211

21

21

2

MBi

in

in

in

in

MBi

in

in fEfEffffEDE }])

~{(}

~{2)[(})

~{(}{ 222

ROPE

Mar. 2005 ICASSP 2005 11

Proposed RDME

The proposed RDME solution

– Comparing with existent RDME• Source coding distortion end-to-end distortion

• mv affects not only the Rmv vs. Rres trade-off, but also more importantly, the coding efficiency vs. error resilience trade-off.

Packet loss impact

headerresmvQPmv

RRRDE }{min},{

– Comparing with existent RD optimized coding mode selection• Extended Inter mode with the mv parameter

• Further optimize the Inter-mode performance

headerresmvQEP

QPmvRRR

pDD

1min

},{

Mar. 2005 ICASSP 2005 12

Simulations

Objective: to check upper-bound performance– Joint {mv, QP} optimization

– RD calculation via actual encoding

Simulation settings– UBC H.263+

– Encoding: I-P-P-……

– Transmission: independent packet loss, with a uniform p

– Decoding: 50 different packet loss realizations for each p

– Performance: average luminance PSNR

Mar. 2005 ICASSP 2005 13

Simulations

Simulation settings (cont’d)– Testing methods

• Conventional ME (cME)• The proposed RDME (RDME)

– Testing scenarios• Random Intra updating (rI):

arbitrarily assigns MB’s to 1/p groups, and cycles through them updating one group per frame.

• Optimal Intra updating (oI):

RD optimized Intra/Inter mode selection.

Mar. 2005 ICASSP 2005 14

Simulation Results Random Intra

PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s]

0 5 10 15 20 25 3030

31

32

33

34

35

36

37

38

39

40

Packet los s rate (%)

PS

NR

(dB

)

cMERDME

0 5 10 15 20 25 3021

22

23

24

25

26

27

28

29

30

Packet los s rate (%)P

SN

R (

dB)

cMERDME

Miss_am Foreman

Mar. 2005 ICASSP 2005 15

Simulation Results Optimal Intra

0 5 10 15 20 25 3031

32

33

34

35

36

37

38

39

40

Packet los s rate (%)

PS

NR

(dB

)

RDME-rIcME-oIRDME-oI

0 5 10 15 20 25 3021

22

23

24

25

26

27

28

29

30

Packet los s rate (%)P

SN

R (

dB)

RDME-rIcME-oIRDME-oI

PSNR vs. Packet loss rate [QCIF, 10f/s, 48kb/s]

Miss_am Foreman

Mar. 2005 ICASSP 2005 16

Simulation Results Random Intra

30 40 50 60 70 80 90 100 110 12031

32

33

34

35

36

37

38

Total bit rate (kb/s )

PS

NR

(dB

)

cMERDME

30 40 50 60 70 80 90 100 110 12021.5

22

22.5

23

23.5

24

24.5

25

25.5

26

26.5

Total bit rate (kb/s )P

SN

R (

dB)

cMERDME

PSNR vs. Total bit rate [QCIF, 10f/s, p=10%]

Miss_am Foreman

Mar. 2005 ICASSP 2005 17

Simulation Results Optimal Intra

30 40 50 60 70 80 90 100 110 12033.5

34

34.5

35

35.5

36

36.5

37

37.5

38

Total bit rate (kb/s )

PS

NR

(dB

)

RDME-rIcME-oIRDME-oI

30 40 50 60 70 80 90 100 110 12023

23.5

24

24.5

25

25.5

26

26.5

27

27.5

Total bit rate (kb/s )P

SN

R (

dB)

RDME-rIcME-oIRDME-oI

PSNR vs. Total bit rate [QCIF, 10f/s, p=10%]

Miss_am Foreman

Mar. 2005 ICASSP 2005 18

Simulation Results

Miss_am: QCIF, 10f/s, 48kb/s, p=10%, random Intra

Conventional ME [29.58dB]

RDME[33.83dB]

Mar. 2005 ICASSP 2005 19

Simulation Results

Foreman: 1st 200f, QCIF, 10f/s, 112kb/s, p=10%, random Intra

Conventional ME[23.92dB]

RDME[26.92dB]

Mar. 2005 ICASSP 2005 20

Besides Intra updating, RDME presents another good alternative for error resilience.

Conclusions

Identify the new opportunity of achieving error resilience via motion estimation.

Propose an RD optimal ME solution, which further optimizes the Inter-mode performance.

Investigate the upper-bound performance.– With random Intra: substantial gain– With optimal Intra: significant gain at low bit rates.

Mar. 2005 ICASSP 2005 21

Conclusions

Future work I: more comprehensive tests– Inaccurate p, bursty loss, or over actual networks, etc.

Future work II: complexity reduction– RD modeling, separate mv and QP optimization,

sophisticated ME strategies, etc.

Originally, the power of Intra coded MB’s is only recognized as stopping past error propagation, while the proposed RDME reveals their new potential on reducing future error propagation.

Mar. 2005 ICASSP 2005 22

References

[Girod `94] B. Girod, ``Rate-constrained motion estimation,'' Nov. 1994. [Sullivan `98] G. J. Sullivan and T. Wiegand, ``Rate-distortion optimization for video

compression,’’ Nov. 1998. [Chung `96] W. C. Chung, F. Kossentini, and M. J. T. Smith, ``An efficient motion

estimation technique based on a rate-distortion criterion,'' May 1996. [Schuster `97] G. M. Schuster and A. K. Katsaggeslos, ``A theory for the optimal bit

allocation between displacement vector field and displaced frame difference,'' Dec. 1997.

[Budagavi `01] M. Budagavi and J. D. Gibson, ``Multiframe video coding for improved performance over wireless channels,'' Feb. 2001.

[H.263+] ITU-T, Rec. H,263, ``Video codeing for low bitrate communications'', version 2 (H.263+), Jan. 1998.

[Wiegand `00] T. Wiegand, N. Farber, K. Stuhlmuller and B. Girod, ``Error-resilient video transmission using long-term memory motion-compensated prediction,'' June 2000.

[Zhang `00] R. Zhang, S. L. Regunathan, and K. Rose, ``Video coding with optimal intra/inter mode switching for packet loss resilience,'' June 2000.

Mar. 2005 ICASSP 2005 23

The End