Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof....

Deblocking Algorithms in Video and Image Compression Coding

Speaker: Wei-Yi Wei

Advisor: Prof. Jian-Jung Ding

Digital Image and Signal Processing Lab

GICE, National Taiwan University

Outline

Introduction Observations of Blocking Artifact Four Categories of Deblocking Algorithms Block-Edge Impairment Metric Comparison Simulation Conclusions and Future Work References

2009/04/11

Deblocking Algorithms in Video and Image Compression Coding 2

Introduction What is Blocking Artifacts ?

– Discontinuity over the block boundaries– Result from coarse quantization

2009/04/11 3

Original Image

Highly Compressed Image


A Simple Coarse Quantization Example

02

46

8

0

2

4

6

8-100

0

100

200

300

400

500

2009/04/11 4

02

46

8

0

2

4

6

80

20

40

60

80

100

02

46

8

0

2

4

6

80

100

200

300

400

500

02

46

8

0

2

4

6

852.5

53

53.5

54

54.5

55

DCT

IDCT

Q/Q-1


Observations of Blocking Artifacts

Two observations of blocking artifact in block-based transform coding– The blocking artifact are more noticeable in flat

areas than in complex areas– The motion compensation propagates the

blocking artifacts into the next frames

2009/04/11 5


Four Categories of Deblocking Algorithms

In-loop Deblocking Filter Post-processing Deblocking Algorithms Pre-processing Algorithms Overlapping Block Methods

2009/04/11 6


In-loop Deblocking Filter

Mainly used in video compression coding Add deblocking filtering to the coding loop Not compatible with the existing standards Case Study

– H.264/AVC– Optimal Post-Process/In-Loop Filtering

2009/04/11 7


H.264/AVC In-loop Filter (1/2)

2009/04/11 8

IntraInter

Transform Quantization

MotionEstimation

FrameBuffer

CoefficientScanning

EntropyCoding

MotionCompensation

Intra FramePrediction

InverseQuantization

InverseTransform

In-LoopFilter

Video Source

Motion Vector

Bitstream

In-LoopFilter


H.264/AVC In-loop Filter (2/2)

BS Pixels to be modified

4 p0, p1, p2, q0, q1, q2

3 p0, p1, q0, q1

2 p0, p1, q0, q1

1 p0, p1, q0, q1

0 No filtering is applied

2009/04/11 9

p3 p2 p1 p0

q0 q1 q2 q3

H.264/AVC encoder must determine the boundary strength (BS) before filtering.

Apply the filters with different strength based on the boundary strength


Optimal Post-process/In-loop Filtering (1/3)

2009/04/11 10

Transform Quantization

Multiplex

Find OptimalFilter Parameters

for Kernel G

Video SourceDCT

CoefficientsEntropyCoding

CoderControl

InverseQuantization

InverseTransform

FrameBuffer

Filter withG

Buffer

MotionCompensation

MotionEstimation

Buffer Fullness

EnablePrediction

QP

PreviousDecodedFrame

Motion Vector

Residue

Data out

OPF

ILOF0



2009/04/11 11

arg min || || G

I I G

Input Frame with size W × H

Filter Kernel with size l × l

Reconstructed Frame with size W × H

I

G

I

I I GThe relationship between , and can be expressed asIGI

The optimal filter coefficient matrix must satisfy

( ) ( , ), 0 , 1m nl m n m n l g GRow-stacked form of G



2009/04/11 12

0,0

0,1

, ( ) (1)

T

T

TW H WH

a

a

a

A

Ag b

Ag b

Tl l A A A T

1l b A b

( ) ( , ), 0 1 and 0 1i jl i j i W j H b IRow-stacked form of I

Window Matrix

Now the equation can be expressed as I I G

Multiply both side of with , we can obtainAg b

Solve this equation, then we

can obtain the optimal filter coefficients


G

l

l I(i,j)^

TA

Post-processing Deblocking Algorithms

The most popular deblocking methods Improve the quality of the compressed

image and video without the source data Case Study

– Reduction of Blocking Artifacts in DCT Domain– Deblocking Using Weighted Sums of Symmetrically

Aligned Pixels– Deblocking Using Offset and Shift Technique

2009/04/11 13


Reduction of Blocking Artifacts in DCT Domain (1/2)

2009/04/11 14

,, , ,

1( , ) ( , )

h hk l

m n k l m nk h l h

B i j w B u vW

,

h h

k lk h l h

W w

, 1, , 2,..., 2k lw k l

,

3, for ( , ) (0,0)

1, k l

k lw

otherwise

1,m nb

,m nb

1, 1m nb

, 1m nb

1, 1,m nb

, ,, ,( , ) { ( , )}k l k l

m n m nB u v DCT b u v

Low Activity

High Activity

DCT-domain filtering

where


Reduction of Blocking Artifacts in DCT Domain (2/2)

The neighboring DCT coefficients at the same frequency do not vary radically within a small range

Apply lowpass filter to each frequency to filter the high frequency components resulting from blocking effect

2009/04/11 15


Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (1/4)

Deblocking Frame

2009/04/11 16

{ , }, 0 -1, 0 -1i jI p i R j C

0,0,r cB 0,2

,r cB

2,0,r cB 2,2

,r cB

,r cB ,r cB

0,0,r cB

1,0,r cB

0,1,r cB

1,1,r cB

,, 1 1

8 ( ) ,8 ( )2 2

{ }, 0 1, 0 1f f

m nr c f f

r m S i c n S jB p i S j S



2009/04/11 17

' ' ' ' ' ' ' ',

', , , , , , , ,i j f f f f f f f f

i j i j S i j S i j i S j i S j S i S j S i S jp p p p p

, ( ) ( )i j i j

'

'

,( , ) ( )

ffS i j

S i j j

'

'

,( ) ( , )

ffi S j

i i S j

' '

' '

,( ) ( )

f ff fS i S j

S i S j

The filtered pixel

where the filter coefficients are

2,1p1,3p 1,4p

2,6p

6,3p 6,4p5,6p

5,1p



2009/04/11 18

Linear Solution

Quadratic Solution

[0, 1]( ) , if x [0, 1]kL x x k

k

[0, 1] (2 )( ) , if x [ , 2 1]k

L

kx x k k

k k

2 ( )(1 2 )( )

( (Q

kx x x

k k k k

2fS

k [0,1] whereDeblocking Algorithms in Video and Image

Compression Coding


2009/04/11 19

Linear Solution Quadratic Solution

05

10

05

100

0.5

1

05

10

05

100

0.2

0.4

05

10

05

100

0.2

0.4

05

10

05

100

0.1

0.2

05

10

05

100

0.5

1

05

10

05

100

0.2

0.4

05

10

05

100

0.2

0.4

05

10

05

100

0.1

0.2


Deblocking Using Offset and Shift Technique (1/3)

Estimate the block of activity

2009/04/11 20

6

0H k

k

ACH C

6

0V k

k

ACH R

7

, , 10

| |k i k i kk

C q q

7

, , 10

| |k k j k jk

R q q

1,1g1,0g 1,2g

0,1g0,0g 0,2g

2,1g2,0g 2,2g

1,4g1,3g 1,5g

0,4g0,3g 0,5g

2,4g2,3g 2,5g

1,6g 1,7g

0,6g 0,7g

2,6g 2,7g

4,2g4,1g

3,1g3,0g 3,2g

5,1g5,0g 5,2g

4,5g4,4g

3,4g3,3g 3,5g

5,4g5,3g 5,5g

4,7g4,0g

3,6g 3,7g

5,6g 5,7g

6,1g6,0g 6,2g

7,1g7,0g 7,2g

6,4g6,3g 6,5g

7,4g7,3g 7,5g

6,6g 6,7g

7,6g 7,7g

4,6g4,3g

Block boundary

Horizontal Activity

Vertical Activity



Filtering for Uniform Deblocking Blocks

2009/04/11 21

3 4offset p p

' | |( ) ( ), for i=1,2,3i i

i

offsetp p sign offset

' | |( ) ( ), for i=4,5,6i i

i


{8,4,2,2,4,8}i

p1 p2 p3 p4 p5 p6

offset

Block boundary

p1 p2 p3 p4 p5 p6

Block boundary

Offset/8Offset/4

Offset/2

Offset/2Offset/4

Offset/8



Filtering for Directional Deblocking Blocks

Filtering for Complex Deblocking Blocks

2009/04/11 22

'3 3

| |( ) ( )

4


'4 4

| |( ) ( )

4


' 1 9 3 7 2 8 4 65 0 5 1 2 3 42 2 2 2

g g g g g g g gg w g w w w w

5g4g 6g

2g1g 3g

8g7g 9g


Pre-processing Algorithms (1/2) Achieve deblocking and enhance the coding

efficiency

2009/04/11 23

S

p3 p2p1

p0

q0q1

q2

q3

'S ''S

pS 'pS ''

pS

Encoding Decoding

Encoding Decoding


Pre-processing Algorithms (2/2)

Our objective is to minimize the difference between I and V and obtain the coefficients

2009/04/11 24

3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q

V I

2( ) 0T

V I VV I

Q Q Q

' ' ' '0 1 2 3( , , , )Q q q q q

The difference between I and V

Original (I)

Pre-processed (R)' ' ' ' ' ' ' '3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q

Filtered (V)' ' '' '' '' '' ' '3 2 1 0 0 1 2 3( , , , , , , , )p p p p q q q q

Taking derivative

Solve this equation, then we

can obtain the optimal filter coefficients


Overlapped Block Methods

Source of Blocking Artifact – Block-based transform does not take the correlation

across the block boundary– Motion Estimation in video coding will propagate the

blocking artifact to the next frame How to solve these two problems

– Lapped Orthogonal Transform– Overlapped Block Motion Compensation

2009/04/11 25


Lapped Orthogonal Transform (1/3) The blocks overlap slightly and the redundant

information is transmitted for the samples in the block boundaries

Reduce the blocking artifact

2009/04/11 26

0 0y Tx

1

0

0

2

0

0

P

P

T

P

P

N N

LCurrent Block Neighboring Block

Transform CoefficientsAssume the 1D discrete time signal is a sequence of MN samples

Segments the signal into M blocks and each block with size N


Lapped Orthogonal Transform (2/3)

2009/04/11 27

0 0 0T

xxR P R P

2

11/

2

1

1 N

ii

TC NN

ii

NG

Energy Compaction Measurement

Where is the i-th diagonal entry of the matrix 2i 0R

2

1

1

2

1

1

1

1

L

L

xx

L

L

R

where


Lapped Orthogonal Transform (3/3)

2009/04/11 28

cos i

cos i

sin i

sin i

i

0

1

2

3

4

5

6

7

0

2

4

6

1

3

5

7

0

1

2

3

4

5

6

7

0

2

4

6

1

3

5

7

Y(0)Y(2)Y(4)Y(6)

Y(1)Y(3)Y(5)Y(7)

1

2

3

DCT

DCT

-

-

-

-

-

-

-

-

DCTE

O

Z

DCTE

O

Z

DCTE

O

Z

eJH

eH

Z

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

1/2

2

2

-

-

-

-

-

-

-

… …

Block 1

Block 2

Block M

LOT ofBlock 1

LOT ofBlock 2

LOT ofBlock M


Overlapped Block Motion Compensation (1/2)

2009/04/11 29

( , ) ( , ) ( , )i iv vE x y P x y S x y

( , ) ( , ) ( , )i iv W vE x y E x y W x y

21 1

1( , )

i

N N

v Wx y

MAE E x yN

EvaluationTarget Block

Overlapped Block

Reference blocks

Motion Vector OutputCurrent Frame

Reference Frame

Window Function


Overlapped Block Motion Compensation (2/2)

2009/04/11 30

( , ) ( , ) ( , )vW vP x y P x y W x y

Overlapped Block

Motion Vectors

Prediction Frame

Reference Frame

Window Function


Block-Edge Impairment Metric (1/2)

The disadvantages of PSNR– It does not always reveal the real quality

perceived by the HVS– It can not real the blockiness

Several block-edge impairment metrics have been proposed– GBIM– It can reveal the real blockiness and evaluate

the performance of deblocking algorithms

2009/04/11 31


Block-Edge Impairment Metric (2/2)

2009/04/11 32

Reconstructed Image Reconstructed Image after deblocking

=3.11GBIMM =1.05

PSNR=21.75 dB

GBIMM

PSNR=21.86 dBDeblocking Algorithms in Video and Image Compression Coding

Comparison (1/2)

In-loop Filter– Achieve better improvement because it can

refer to the new input video frame– Not compatible with the existing standards

Post-processing Deblocking Algorithms– Good potential to be integrated into existing

standards– Blind to the target image and video

2009/04/11 33


Comparison (2/2)

Pre-processing Deblocking Algorithms– Modify the source image and video in advance

and reduces the bit rate– Achieve the quality close to the direct

compressed image and video Overlapped Block Methods

– Prevents the blocking artifacts from happening in advanced

2009/04/11 34


Simulation (1/3)

2009/04/11 35

Original Image Highly Compressed Image

Offset and Shift

WSSAP DCT DomainDeblocking Algorithms in Video and Image Compression Coding

Simulation (2/3) LOT and DCT

2009/04/11 36

0 2 4 6 8 10 12 14 16 18 2024

26

28

30

32

34

36

38

40

42

Quantization step

PS

NR

(dB

)

DCT

LOT

LOT with Q=10

DCT with Q=10

PSNR under different quantization strength


Simulation (3/3) Overlapped Block Motion Compensation

2009/04/11 37

0 10 20 30 40 50 60 70 80 90 10029.5

30

30.5

31

31.5

32

32.5

Frame Number

PS

NR

(dB

)

weather-slow motion video :PSNR

Video Compression

Video Compression with OBMC

0 10 20 30 40 50 60 70 80 90 10027

28

29

30

31

32

33

Frame Number

PS

NR

(dB

)

stefan-fast motion video :PSNR

Video Compression


0 10 20 30 40 50 60 70 80 90 1000.4

0.6

0.8

1

1.2

1.4

1.6x 10

7

Frame Number

Energ

y

weather-fast motion video :Energy

Video Compression


0 10 20 30 40 50 60 70 80 90 1000

1

2

3

4

5

6

7

8

9

10x 10

7

Frame Number

Energ

y

stefan-fast motion video :Energy

Video Compression



Conclusions and Future Work

Many conventional methods do not exploit the characteristics of the blocking artifacts

The new algorithms apply different filter with different strength to different areas

Adopting the blockiness metric to measure the blockiness will be the future trend

2009/04/11 38


References (1/2)In-loop Filter

[1] Lain E.G. Richardson, “H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia”, John Wiley & Sons, Jan. 2004.

[2] Lee, Jae-Beom, Kalva and Hari, ”The VC-1 and H.264 Video Compression Standards for Broadband Video Services”, Springer, 2008.

[3] “Text of ISO/IEC FDIS 14496-10/Draft ITU-T H.264: Information Technology – Coding of Audio-Visual Objects: Advanced Video Coding”, International Organization for Standardization, 2003.

[4] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu Chang, “Optimal Post-Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007.

[5] S. Romero, and L.F. Romero, “An Optimized Preconditioned Conjugate Gradient Algorithm”, Technicah Report No: UMA-DAC-02/11, University of Malaga, Sept. 2002.

Post-Filtering [6] Tao Chen, Hong Ren Wu and Bin Qiu, “Adaptive Postfiltering of Transform Coefficients for the

Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 11, no. 5, Dec. 2001.

[7] Shizhong Liu and Alan C. Bovik, “Efficient DCT-Domain Blind Measurement and Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 12, no. 12, May 2002.

[8] Ci Wang, Wen-Jun Zhang and Xiang-Zhong Fang, “Adaptive Reduction of Blocking Artifacts in DCT Domain for Highly Compressed Images”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 50, no. 2, May 2004.

[9] A. Z. Averbuch, A. Schclar and D. L. Donoho, “Deblocking of block-transform compressed images using weighted sums of symmetrically aligned pixels,” IEEE Trans. on Circuits Syst. Video Technology, vol.14, pp.200-212, Feb. 2005.

2009/04/11 39


References (2/2) [10] Jongho Kim, Minseok Choi, and Jechang Jeong, “Reduction of Blocking Artifact for HDTV using

Offset-and-Shift Techniques”, IEEE Trans. on Consumer Electronics, vol. 53, no.4, November 2007. [11] Zixiang Xiong, Michael T. Orchard, and Ya-Qin Zhang, “A Deblocking Algorithm for JPEG-

Compressed Images Using Overcomplete Wavelet Representations”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 7, no. 2, April 1997.

[12] Jongho Kim and Jechang Jeong, “Adaptive Deblocking Technique for Mobile Video”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007.

Pre-Filtering [13] Sheng-Ho Wang, Sung-Wen Wang, Yi-Shin Tung, Ja-Ling Wu, Jau-Hsiung Huang, "Pre-Process for

maximizing the effect of in-loop deblocking filtering in H.264/AVC encoding," 5th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services (EC-SIP-M 2005), Smolenice, Slovak, 29 June - 2 July, 2005.

Overlapped Block Methods [14] H. S. Malvar and D. H. Staelin, “The LOT: transform coding without blocking effects”, IEEE Trans. on

Acoustic., Speech, Signal Processing, vol. 37, no. 4, pp. 553-559, April 1989. [15] M. T. Orchard and G. J. Sullivan, “Overlapped block motion compensation: an estimation-theoretic

approach”, IEEE Trans. on Image Processing, vol. 3, no. 4, pp. 693-699, Sep. 1994.

Block-Edge Impairment Metric [16] H. R. Wu and M. Yuen, “A Generalized Block-Edge Impairment Metric for Video Coding”, IEEE Signal

Processing Letters, vol. 4, no. 11, Nov. 1997. [17] Athanasios Leontaris, Pamela C. Cosman and Amy R. Reibman, “Quality Evaluation of Motion-Compensated

Edge Artifacts in Compressed Video”, IEEE Trans. on Image Processing, vol. 16, no. 4, April 2007.

2009/04/11 40


Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof....

Documents

Transcript of Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof....