Katto lab.1

Overlapping in Video Signal Overlapping in Video Signal ProcessingProcessing

Jiro Katto

Dept. of Computer Science, Waseda University

E-Mail: katto@waseda.jp

2

Katto lab.

Outline

Introduction“Overlapped” Transform“Overlapped” Motion Compensation“Overlapped” PredictionDenoising by “Overlapping”Conclusions

3

Katto lab.

What I am ...

90 95 00 05

Ph.D Student Company University

Image/Video Compression

Multimedia Communication Systems

Multimedia Signal Processing

MPEG, ITU-T

4

Katto lab.

What I did ...

90 95 00 05

Student Company University

Image/Video Compression

Multimedia Communication Systems

Multimedia Signal Processing

MPEG, ITU-T

Overlapped MC(IEEE Trans CSVT-94 & IEEE ICASSP-95)

IPB Prediction & Rate Control

(IEEE ICIP-95)

Wavelet Image Coding (SPIE VCIP-91)

Denoising using Motion Estimation & Pixel Shift(IEEE ICASSP-2008)

5

Katto lab.

Transform Quantization

InverseQuantization

InverseTransform

+

−

FrameMemory

MotionCompensation

MotionEstimation

EntropyCoding

Rate Control

input

local decoder

+

MemoryMotionCompensation

EntropyDecoding

InverseQuantization

InverseTransform

output

Encoder Decoder

Video Coder

OverlappedTransform

OverlappedTransform

OverlappedMotion

Compensation

OverlappedMotion

CompensationBi-directional(Overlapped)Prediction

Bi-directional(Overlapped)Prediction Denoising by

OverlappingDenoising byOverlapping

6

Katto lab.

1. “Overlapped” Transform

J.Katto and Y.Yasuda: "Performance Evaluation of Subband Coding and Optimization of Its Filter Coefficients", SPIE VCIP'91, Nov.1991.

Block TransformDCT (Discrete Cosine Transform)DFT (Discrete Fourier Transform)KLT (Karhunen-Loeve Transform)

Overlapped TransformWavelet transform / Subband codingLOT (Lapped Orthogonal Transform)MDCT (Modified DCT)

7

Katto lab.

Wavelet Transform

2h0(n)x(n)

2h1(n)

2h0(n)

2h1(n)

2h0(n)

2h1(n)

LLL

LLH

LH

H2分割フィルタバンクのツリー接続

π

角周波数

0

LLL LLH LH HLL

H

LH

LPF

HPF tree structured filter bank

frequency

8

Katto lab.

Problems (at that time)

DCT was widely used in image/video compression (JPEG/MPEG) due to its suboptimal compression capability, but suffers from blocking artifacts.Wavelet transform is a promising alternative because it can alleviate the blocking artifacts due to its overlapping properties. However, there was no theoretical compression performance measure of Wavelet transform.

original JPEG

9

Katto lab.

Optimum Bit Allocation Problem

N.S.Jayant and P.Noll: “Digital Coding of Waveforms”, Prentice Hall, 1984.

minimize reconstruction error variance,

under constant bit rate constraint

Methods: e.g. Lagrange multiplier method

( ))( 22qr f σσ =

constR ≤

quantization error variance

10

Katto lab.

Coding Gain

N.S.Jayant and P.Noll: “Digital Coding of Waveforms”, Prentice Hall, 1984.

NN

kk

N

kk

TCNG 1

1

0

2

1

0

21

⎟⎟⎠

⎞⎜⎜⎝

⎛=

∏

∑−

=

−

=

σ

σN ×N matrix(orthogonal)

N ×N matrix(orthogonal)

optimum bit allocationproblem

“Coding Gain”σk

2: coefficient varianceTC : transform coding

SIZE (N)

3

4

5

6

7

8

9

10

11

0 2 4 6 8 10 12 14 16

GA

IN (

dB)

OPTIMUM (ρ=0.95)

KLT, DCT

DFT

ρ : correlation factor

[Theory] DCT can approximate KLT when ρ → 1

11

Katto lab.

Coding Gain of Wavelet Transform ?

multirate filter bank

optimum bit allocationproblem

∏−

=⎟⎟⎠

⎞⎜⎜⎝

⎛=

1

0

1K

k k

kk

wavelet kBAG α

α

2h0(n)x(n)

2h1(n)

2h0(n)

2h1(n)

2h0(n)

2h1(n)

LLL

LLH

LH

H2分割フィルタバンクのツリー接続

α0

α1

αK-1

tree-structured filter bank

22xky A

kσσ ⋅=

∑−

=

⋅=1

0

22K

kqkr k

B σσ

{ })(nhk←

{ })(ngk←

sampling ratio

12

Katto lab.

Advantages & Evaluation Example

Proposed coding gain includes classical coding gain as its special case, and enables integrated performance comparison with block transform.Proposed coding gain can be applied to non-orthogonal filter bank (e.g. bi-orthogonal case).

provides theoretical validity on JPEG-2000’s 9x7 filter (bi-orthogonal filter)

13

Katto lab.

2. “Overlapped” Motion Compensation

J.Katto, J.Ohki, S.Nogaki and M.Ohta: "Wavelet Codec with Overlapped Motion Compensation for Very Low Bit Rate Environment", IEEE Trans. on Circuit & Systems for Video Technology, June.1994.J.Katto and M.Ohta: "An Analytical Framework for Overlapped Motion Compensation", IEEE ICASSP'95, May.1995.

(Gradient Method: Optical Flow)Block Motion Compensation

16x16 macroblock (until MPEG-4)4x4 ~ 16x16 macroblocks (H.264/AVC)

Overlapped Motion Compensation32x32 macroblock(Motion estimation may use 16x16 macroblock)

14

Katto lab.

Overlapped Motion Compensation

Block MC:Copy a macroblock in the previous frame to build a prediction frame.

Overlapped MC:Overlap and add multiple macroblocks in the previous frame with weights.

M.Ohta and S.Nogaki: “Hybrid Picture Coding with Wavelet Transform and Overlapped Motion Compensated Interframe Prediction Coding", IEEE Trans. SP, Dec.1993.H.Watanabe and S.Singhal: “Windowed Motion Compensation", SPIE VCIP-91, Nov.1991.

∑∑ −−=i nm

yixiii dydxznmwyxz,

,, ),(),(),(ˆ

),(),(ˆ yx dydxzyxz −−=

15

Katto lab.

Problems (at that time)

What is the optimum weighting coefficient for overlapped motion compensation ?

“Raised cosine” and “trapezoid” window functions were heuristically tried. Only one work (below) exists.

Can we theoretically compare overlapped motion compensation with fractional-pel (e.g. half-pel) motion compensation ?

At that time, MPEG-1 adopted half-pel motion compensation. Later, overlapped motion compensation was adopted by H.263.

M.T.Orchard and G.J.Sullivan: “Overlapped Block Motion Compensation: An Estimation-Theoretic Approach”, IEEE Trans. IP, Sep.1994.

16

Katto lab.

Problem Formulation

minimize

under

[ ]2)ˆ( zzE −

⎪⎪⎩

⎪⎪⎨

⎧

=

=

∑

∑−

=

−

=

1

ˆ1

0

1

0N

kk

N

kkk

w

zwzN : # of macroblocks to beoverlappedzk : k-th mackroblock

Wiener-Hopf equation

bw 1−= AoptA : correlation matrixb : correlation vector

17

Katto lab.

Optimum WindowsTheoretical: AR(1) model Experimental

Trapezoid Raised Cosine Proposed

Salesman +0.90 +0.90 +0.92Blue Jacket +0.42 +0.41 +0.46

Simulations

18

Katto lab.

Motion Model (1)

corresponding pixels (between current and previous) do not change from frame to frame (though previous pixel may not be on sampling grid)

spatial correlation between pixels decreases exponentially

Assumptions:

),(),( ,0,0,0,00 yyxx ddyddxzyxz Δ−−Δ−−=

[ ][ ]

[ ]0

20

,0,000

),(),(),( dEyx

yxzEdydxzyxzE Δ=

Δ−Δ−ρ

19

Katto lab.

Motion Model (2)Probability density function of motion estimation errors

[ ] max,0,0 21

41 VsdEa ii ⋅

−+=Δ=

[ ] ⎟⎠⎞

⎜⎝⎛ −⋅

−+=Δ=

43

21

81

max,0,0 VsdEa hh

reliableunreliable

[ ][ ]2

20 )(

zEzzE − (prediction error reduction)

Interger-pel ia ,01 ρ−

Half-pel ha ,01 ρ−

Overlapped 22/)(

2/)( 21

111

,1,0,1,0

,1,0

,0 ⎥⎦

⎤⎢⎣

⎡ +−−−

−−+

+

iiii

ii

i

aaaa

aaa ρρρ

ρρ

Half-pel + Overlapped 22/)(

2/)( 21

111

,1,0,1,0

,1,0

,0 ⎥⎦

⎤⎢⎣

⎡ +−−−

−−+

+

hhhh

hh

h

aaaa

aaa ρρρ

ρρ

overlapping effect

20

Katto lab.

Results

Overlapped motion compensation was originally proposed to reduce blocking artifacts in block motion compensation.Proposed theory validates the prediction error reduction effect of overlapped motion compensation. The theory proves that overlapped motion compensation performs well when motion estimation does not work well, though half-pelmotion compensation works well when motion estimation works well. They can be combined and can help each other.

21

Katto lab.

3. “Overlapped” Interframe Prediction

IPB prediction structureI-picture: Intraframe codingP-picture: Interframe codingB-picture: Bi-directional interframecoding

Adopted in MPEG-1 standard

J.Katto and M.Ohta: "Mathematical Analysis of MPEG Compression Capability and Its Application to Rate Control", IEEE ICIP'95 Oct.1995.

22

Katto lab.

Problems (at that time)

Why does the IPB prediction structure work better than the IP prediction structure ?

MPEG-1 was indeed better than H.261. One reason is half-pel motion compensation, and another is IPB prediction. But, why ?

TM5 (famous rate control for MPEG-2 standard at that time) is the best rate control method ?

Theory on IPB prediction might provide better rate control strategy.

23

Katto lab.

Coding Gain of IPB Prediction (1)

Prediction error variancesP-picture

B-picture( ) 22 )(12 xP Mcor σσ ⋅−⋅=

22 )()()(21

23

xB jMcorjcorMcor σσ ⋅⎟⎠⎞

⎜⎝⎛ −−−⋅+=

j M-j

cor(k) : correlation between k-frame apart frames

24

Katto lab.

Coding Gain of IPB Prediction (2)

( ))(12/)( 22 McorMP xP −⋅== σσ

⎟⎠⎞

⎜⎝⎛ −−−⋅+== )()()(

21

23/)( 22 jMcorjcorMcorMB xB σσ

IPB prediction structure

optimum bit allocationproblem

MM

LML

IPB

MMBMP

G 11

1)()(

1−

−

⎥⎦⎤

⎢⎣⎡

−⋅

=

wkkcor ρ=)(

25

Katto lab.

Rate Control for IPB Prediction (1)

HeuristicsbQaR +⋅= loglog

R : rateQ : quantization step sizea, b : constants

XRQ =⋅ α

Rate-distortion theory: 2222 2 xR

q σεσ −⋅=

difficult to apply to rate control directly ...

26

Katto lab.

Rate Control for IPB Prediction (2)

Optimum target bit assignment minimize average quantization step size

under rate constraint

BPI

mBB

mPP

mII

NNNQNQNQN

++⋅+⋅+⋅

constRNRNRN BBPPII =⋅+⋅+⋅

NI, NP, NB : # of pictures in GOP

27

Katto lab.

Rate Control for IPB Prediction (3)

Solution

αα mm

I

BB

mm

I

PPI

I

XXN

XXNN

RR++

⎟⎟⎠

⎞⎜⎜⎝

⎛+⎟⎟

⎠

⎞⎜⎜⎝

⎛+

=11

αmm

P

BBP

P

XXNN

RR+

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=1

αmm

B

PPB

B

XXNN

RR+

⎟⎟⎠

⎞⎜⎜⎝

⎛+

=1

III XRQ =⋅ α

PPP XRQ =⋅ α

BBB XRQ =⋅ α

observable

Simulation

28

Katto lab.

Summary of Formulations

29

Katto lab.

Results

Coding gain of IPB prediction quantitatively proves its efficiency against IP prediction.

(However, though omitted, coding gain based rate control is not practical)

Rate control for IPB prediction based on heuristics between rate and quantization step size provides smarter bit assignment than TM5.

TM5 is included as a special case of the proposal.

30

Katto lab.

4. Denoising by “Overlapping”

Reconsiderations on video codingComplexity shift to decoders

Distributed video codingApplication to wireless sensor networks

Multiple description codingApplication to heterogeneous networks

Super-resolutionSDTV/HDTV conversion at a decoder

Image restorationDenoising

Reduction of blocking artifacts and flicker noisesDeblocking filters, motion compensated temporal filtering, re-application of JPEG, and so on ...

J.Katto, J.Suzuki, S.Itagaki, S.Sakaida and K.Iguchi: “Denoising Intra-Coded Moving Pictures using Motion Estimation and Pixel Shift”, IEEE ICASSP 2008, Apr.2008..

31

Katto lab.

Problems

Quality enhancement of compressed video streams becomes important

Enormously rapid increase of compressed visual contents over storages and networksPeople prefer better quality contents if they don’t pay much money for them Decoders (PCs) become powerfulSmart quality improvement of old visual contents is impressive Smart denoising might contribute to smarter compression in future

32

Katto lab.

Auxiliary Experiment

encode &decode

encode &decode

encode &decode

pixel shift(0,0)∼(7,7)

+

originalimage

reconstructedimage

encode &decode

encode &decode

encode &decode

pixel shift(0,0)∼(7,7)pixel shift

(0,0)∼(7,7)

+

originalimage

reconstructedimage

Pixel shift, JPEG compression and picture “overlapping” brings PSNR improvement.

0

0.5

1

1.5

2

2.5

3

1 10 100number of aligned images

PS

NR

gai

n [d

B]

Airplane (JPEG)Airplane (JPEG-2000)Whale (JPEG)Whale (JPEG-2000)theory (a=0.8)theory (a=0.6)theory (a=0.4)theory (a=0.2)theory (a=0.0)

a=0.6

33

Katto lab.

Formulation (1)

Statistical reduction of quantization errors

minimize

where

under constraint

∑∑==

+==K

k

K

kkqkwxkxkwx

11)()()(ˆ)(~

1)(1

=∑=

K

kkw

( )[ ]⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛=− ∑

=

2

1

2 )()(~ K

kkqkwExxE

K : # of overlapped picturesq(k) : k-th quantization errorw(k) : k-th weight

34

Katto lab.

Formulation (2)

Solutionvariant of Wiener-Hopf equation

where , and

uw 1−⋅= RCopt

t)1,,1,1( L=u)(sum

11 u−=

RC

[ ]⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢

⎣

⎡

=⋅=

])([)]2()([)]1()([

)]()2([])2([)]1()2([)]()1([)]2()1([])1([

2

2

2

KqEqKqEqKqE

KqqEqEqqEKqqEqqEqE

ER t

OM

L

qq

correlation matrix of quantization errors

35

Katto lab.

Formulation (3)

Special caseswhen independent, i.e.

when completely dependent, i.e.no gain (corresponding to no pixel shift)

[ ] )(0)()( lklqkqE ≠=

∑=

⋅= K

l lqkq

opt kw

12

)(

2)(

11)(

σσ

∑=

= K

k kq

q

12

)(

2min, 1

1

σ

σ

Kkwopt

1)( =

Kq

q

22

min,

σσ =

22)( qkq σσ =esp. when

[ ] 2)()( qlqkqE σ=

)(log10 10 dBKGain:

36

Katto lab.

Motion JPEG Extension

pixelshift

MJPEGencode

MJPEGdecode

inverse pixel shift

motionestimation

alignment& mixture

encoder decoder

Constraint by quantization step sizes

reference frames

Motion estimation at a decoderIntentional pixel shift at an encoder and decoder (though breaking compatibility)

37

Katto lab.

Experiments (1)

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

fwdME biME shift+fwdME shift+biMEMotion Estimation

PSN

R g

ain

[dB]

akiyo (34.7dB)children (30.2dB)coastguard (29.2dB)container (28.7dB)flower (26.8dB)mobile (26.0dB)news (32.3dB)sean (30.2dB)weather (26.4dB)

motion shift and pixel shift cooperate

motion estimation effect

pixel shift effectfwd: forwardbi: bi-directionalshift: pixel shift

Motion estimation works well for moving regionsIntentional pixel shift works well for static regions

38

Katto lab.

Experiments (2)

Subjective quality comparisonMotion JPEG Proposed method

impressive reduction of blocking artifacts and flicker noises

39

Katto lab.

Results

Overlapping similar regions with different quantization noises contributes to statistical reduction of quantization noises.Subjective quality improvement in blocking artifacts and flicker noises is also observed.

Extension to interframe coding is ongoing.Extension to super-resolution is also ongoing.

40

Katto lab.

Conclusions

“ Overlapping” had brought many performance improvements in video compression.“Optimum bit allocation” is one of important key formulations.“Wiener-Hopf equation” is another important key formulation.