Dimitry Kletsel, Yair Kuszpet

Supervisors: Yair Moshe, Dr. Avi Levy

In association with Intel-Oplus

18/07/2007

Outline Video Coding. MPEG2 and H.264

Coding Artifacts

Temporal Artifacts in MPEG2

The Temporal Flicker Artifact in H.264

Nonadaptive Approach to Flicker Reduction

Adaptive Approach to Flicker Reduction

Results, Conclusion, and Future Work

2

Project Goal Characterization of H.264-encoded video artifacts

Development and examination of suitable Post-Processing techniques

3

4

MPEG2 Block Diagram

5

•ICT

•Intra prediction

•Deblocking filter

•Motion Estimation

H.264 Block Diagram

H.264 Deblocking Filter Reduces blocking artifact Operates in-loop therefore more effective than post processing Adaptive filtering Significant computational complexity 5-15% improvement in bitrate per given quality compared to

unfiltered video

6

H.264 Intra Prediction Exploit spatial correlation between adjacent blocks in

intra frames

7

H.264 Coding Artifacts Blocking – reduced due to deblocking filter

Ringing – not observed, due to 4x4 blocks & filter

Blurring – only at very low bitrates

Color bleeding – wasn’t observed (4x4 blocks)

Temporal discontinuities

Most noticeable artifact in H.264

Include a wide range of phenomena. No agreement in the literature on terminology or causesMobile 250Kbps

8

Blocking

Ringing

Blurring

Color bleeding

Temporal discontinuities

MPEG2 - Temporal Post Processing Methods Delcorso et al., Mosquito Noise Reducer, 2002

Atzori et al., Adaptive Anisotropic Filter, 2002

Coudoux et al., Temporal Busyness Post Processor, 2003

Most literature deals with MPEG2 temporal discontinuities

Temporal discontinuities characterization not relevant to H.264 Practically no ringing in H.264

“Temporal discontinuities” is too wide a term

9

H.264 Flicker Noticeable temporal discontinuity around intra frames

Intra frame may be sharper or more blurred than preceding inter frame

Intra frame requires different bitrate than inter frames

Most noticeable in low-medium bitrates

10

0

100000

200000

300000

400000

500000

600000

0 15 30 45

Bits / Frame

Frame # [JVT Encoder]

Flicker in the literature Fan et al., 2002

All-intra video sequences

Attribute flicker to changes in intra prediction modes

Propose non-compliant encoder modification

Referenced measure Compare differences in adjacent frames in encoded and

original videos

11

1orig I If f 1ˆ ˆ

encoded I If f

orig encodedflicker avg

ˆ (# )If coded frame I

(# )If original frame I

Flicker in the literature – cont’d Later works treat videos with periodically-inserted

intra frames

Most use encoder modifications: Sakaida et al., 2004 – change intra prediction mode

selection, and encode repeatedly with finer quantization

Chun et al., 2006 – change intra pred. mode selection

Chono et al., 2006 – modify quantization levels

Yang, Park, Jeon, 2006 – preprocessing by Kalman filter (all intra)

All works use (roughly) same objective measure

12

Examination of Flicker Reasons Different Intra Prediction Modes with SKIP

By 4 different papers

Not satisfactory because DC-only prediction also exhibits flicker (By DQ – similar test by us, with similar results)

Grid Movement Objects are broken to different blocks due to movement, each block

handled differently, thus flicker is caused

Not satisfactory – non moving parts in videos exhibit flicker

Spirals half with DC modes only

13

Flicker – Further Examination Different Coding Error Patterns (suggested by DQ article)

Generalized, includes several components

Inter – Temporal prediction & strong quantization of residuals or SKIP

Intra – Spatial prediction & weak quantization of residuals

14

MSE Coded vs. Original (per pixel)

Inter vs. Intra MSE

MSE

frame

Flicker Post-Processing Novel treatment for flicker

Doesn’t necessitate changes in the encoder

Complements suggested encoder modifications as no single method eliminates flicker completely

No Post Processing method for flicker reduction was found in literature

Difference of Coding Error Patterns

Reduction – to alleviate flicker

Estimation – to measure flicker, crucial for adaptive filtering

15

Flicker Reduction - Main Idea Estimate motion vectors between every two consecutive frames Reconstruct frame X from X-1 by MV, to get MCP(X) X is the Intra frame which is Original + spatial prediction error MCP(X) estimates Original + temporal prediction error

The better the estimation the better the results No motion vectors for Intra frames

Average X and MCP(X)

16

Frame X

MCP(X)Motion

Estimator

Average

Frame X-1

Out Frame

Flicker Reduction – cont’d Need to filter only around I frames, to avoid

unnecessary blurring

Jump is steep – need to filter across more than one frame, to smooth the jump

Use weighted average by distance from I frame

17

I IP P…

t

Subj.

quality

Flicker Reduction Post Processing Scheme

18

+

GOP (n-1) GOP (n) ...

Displayed

Frames

Received

Frames

Filter

IX

1IX

1IX

IMCP X

ID

1ID

1ID

Filter

Flicker High Pass Filter

19

DC

Use weighted average by distance from I frame

Only filter low frequencies, to avoid flicker in fine details

Low

Freq.

Low

Freq.

Low

Freq.

High

Freq. Displayed

Frames

Received

Frames

Motion

Compensated

Frames

ICTInv.

ICTFilter

I jD

I jX

I jMCP X

ICT

Adaptive Filtering Filtering k frames reduces intra frame jump by ~1/k

Need to decide how many frames to filter in each GOP

Measure flicker in the intra frame

Earlier works only provide a referenced measure

Novel non-reference measure was developed, based on empirical flicker characteristics

Constructs a ‘flicker map’ for an intra frame using its motion-compensated counterpart

‘flicker map’ is derived from estimated difference of coding error patterns (same as X – MCP(X))

20

Adaptive Filtering – cont’d

21

Flicker is more noticeable in smooth areas

Identify ‘smoothness’ bycalculating 1/(1+std) of 3x3 block centered on the pixel

1/(1+std) < 0.5 means a pixelis in a non-smooth area

Adaptive Filtering – cont’d Identification of smooth areas

22Intra frame Smooth areas

Adaptive Filtering – cont’d Large differences between intra frame and motion-

compensated counterpart may indicate high flicker

23Intra frame Absolute difference

Adaptive Filtering – cont’d Smooth areas with large differences relative to motion-

compensated image will display most flicker

Multiply smoothness map by difference map

24Abs. diff * smoothness

Adaptive Filtering – cont’d Isolated changing pixels are not perceived as flicker

Use morphological opening by reconstruction to detect clusters of pixels

Where the result is not zero,copy pixel from difference image

25Final flicker map

Adaptive Filtering – cont’d Flicker map indicates presence and strength of flicker

Flicker is measured for the entire frame

Need to determine the worst flicker, not the average

Pick lowest integer that is greater than 75% of the non-zero pixels in the flicker map

Indicates the number of frames to filter in the GOP

26

Adaptive Filtering - Diagram

27

FlickerMap

AbsoluteDifference

MultiplySmooth

AreasMorphological

Opening

Copy if Morphological

Opening not zero

Lowest75%

Out

Coding Error

Difference Estimator

Frame X

MCP(X)

PSNR vs. Bitrate

28

33

34

35

36

37

38

39

40

200 250 350 500

PSNR[dB]

Bitrate [Kbps]

Container PSNR-vs-Bitrate

PSNR Unfiltered

PSNR Filtered

27

29

31

33

35

37

39

41

200 250 350 500 1000

PSNR[dB]

Bitrate [Kbps]

Paris PSNR-vs-Bitrate

PSNR Unfiltered

PSNR Filtered

20

22

24

26

28

30

32

34

200 250 350 500 1000

PSNR[dB]

Bitrate [Kbps]

Mobile PSNR-vs-bitrate

PSNR Unfiltered

PSNR Filtered

Flicker vs. Bitrate

29

3

5

7

9

11

13

15

200 250 350 500 1000

Fan'sFlicker

Bitrate [Kbps]

Mobile Flicker-vs-Bitrate

Flicker Unfiltered

Flicker Filtered

2

2.5

3

3.5

4

4.5

5

5.5

200 250 350 500

Fan'sFlicker

Bitrate [Kbps]

Container Flicker-vs-Bitrate

Flicker Unfiltered

Flicker Filtered

2

4

6

8

10

12

200 250 350 500 1000

Fan'sFlicker

Bitrate [Kbps]

Paris Flicker-vs-Bitrate

Flicker Unfiltered

Flicker Filtered

Summary Flicker is prevalent in H.264

Wasn’t studied extensively in the past

Existing solutions require encoder changes

Innovative post processing technique and non-reference objective measure suggested

Complements encoder modifications

Shows good results, objectively and subjectively

Paper submitted to PCS 2007

Patent-pending by Intel-Oplus

30

Future Directions Better frame reconstruction

Results affected by motion vector accuracy

Adaptive filtering in the frame

Use flicker map to select areas where flicker is particularly noticeable

Might lead to edge artifacts

31

Thank you!

32

Backup

33

34

Coding Artifacts - examples

Ringing /

Mosquito noise

Color bleeding

35

Coding Artifacts – examples

Blurring

Blocking

36

Deblocking Filter

Filter decisions based on block type

and position in macroblock:

Blockiness across a 4x4 block boundary:

Adaptive filtering along horizontal and

vertical edges:

37

Deblocking FilterFilter Strength can be altered on the:

Slice level

Macroblock level

Sample level

I

I

P

I P

Bs=4

Bs=3

Flicker – Suggested Reasons

38

Intra Inter Prediction

Intra

Block -

IntraModes

Selection

ICT Q

Grid movement

Skip Mode

Different Intra

Prediction modes

Different Coding

Error Patterns

Backup2

39

References[1] Sandra Delcorso, Carolina Miro, and Joel Jung MNR:

A Novel Approach to Correct MPEG Temporal Distortions

[2] Atzori, L.; De Natale, F.G.B.; Granelli, F Adaptive anisotropic filtering (AAF) for real-time visualenhancement of MPEG-coded video sequences.

[3] Francois-Xavier Coudoux, Marc Georges Gazalet, Patrick Corlay A post-processor for reducing temporal busyness in low-bit-rate video applications

40

Blurring Experienced at low-medium bitrates

Happens due to low bitrate and due to the de-blocking filter. Annoying blocking artifact is replaced by less annoying blurring

Some details were simply lost (due to bitrate)

It is not clear that something can be done about it

41

Temporal Busyness Post Processor Coudoux, Gazalet, Corlay, 2003

Deals with temporal busyness resulting from ringing and DCT basis images

DCT basis images not present in H.264 due to deblocking filter

Ringing is not a problem in H.264

42

MNR – Drawbacks Simple motion identification:

Doesn’t use motion vectors Uses only absolute difference between same blocks in

adjacent frames

Weak filter: Filters only DC coefficients Uses only 2 frames for filtering (preceding and

following)

No perceivable improvement in our videos.

43

AAF – Drawbacks Doesn’t do temporal filtering

Assumes that mosquito noise comes only from ringing

Doesn’t mention other temporal artifacts besides “mosquito noise”

There are other temporal artifacts

44

H.264 Encoders A variety of H.264 encoders in the market

x264 is the leading encoder according to benchmarks (MSU)

Chosen encoder for project

The JVT reference encoder is considerably inferior

Also exhibits motion jerkiness (at low bitrates)

45

Artificial Example: Spirals Based on Fenimore, Libert, Roitman, 2000

Propose a metric for MPEG2 MN measurement

Propose a test pattern for subjective MN measurement: still spirals video

We used a similar pattern (800x530x64Kbps)

Still video exhibits slight PI Jumps

Much worse jumps with movement

<Moving video example>

46

From Artificial to Real World Video Need to filter only around I frames, to avoid

unnecessary blurring

PI Jump is steep – need to filter across more than one frame

Use weighted average by distance from I frame

47

I IP P…

t

Subj.

quality

From Artificial to Real World Video – cont. I frame doesn’t have motion vectors

So we don’t use them…

We generated our own MVs from the original video, and used them in the reconstruction

In real applications, can use H.264 MVs, and generate I frame MVs by motion estimation with MSE

48

From Artificial to Real World Video – cont. Motion estimation is not perfect

Filtering high frequencies (=textures and edges) will cause an edge jump when we stop filtering

Solution: filter only low frequencies

49

Approach Summary Generate MVs for entire original video

fout(Intra-1)=f(Intra-1) % don’t process pre-I frames

For each frame j in frames: Intra to Intra+k-1

fc(j) = compensate_motion{fout(j-1)}

fout(j)= low_freq {j/k*f(j) + (k-j)/k*fc(j)}+high_freq{f(j)}

50

Video Examples Mobile, unfiltered

Mobile, filter low frequencies

Shields, unfiltered

Shields, filter all frequencies

Shields, filter low frequencies

Ballroom, unfiltered

Ballroom, filter low frequencies

51

Future Directions I frame motion vectors:

Generate by exhaustive search

Interpolate I-1 and I+1 MVs

Objective quality metric

Optimal thresholds

52

MNR – Mosquito noise reducer Delcorso, Jung, 2002

Defines Mosquito Noise as temporal fluctuation near edges of moving objects

Identifies moving blocks (LPF on frame difference)

DC median filter (temporal & spatial) on still blocks

Drawbacks

Simple motion estimation

Weak filter

No perceivable improvement in our videos

53

AAF – Adaptive Anisotropic Filter

Atzori, De Natale, Granelli, 2002

Defines Mosquito Noise as ringing near edges of objects

Identifies the types of blocks

Applies a set of spatial filters on different types of blocks

Drawbacks

No temporal filtering

Not all temporal artifacts are due to ringing

54

0

100000

200000

300000

400000

500000

600000

0 15 30 45 60 75 90 105 120 135 150

Bits/frame

Frame #

[JVT Encoder]

55

Objective Measurement Results Modified version of Fan’s flicker measure

Apply measure only to k frames following I-frame (k=2, 4, 6)

For our filter, low k values expected to give better results

56

Distortion Reduction

6 frames4 frames2 framesVideo

38.83%46.16%58.67%mobile175i15

-25.05%38.25%ballroom300i15

-17.86%22.50%shields512i15

Adaptive Filtering - Summary Locate smooth areas in image

Calculate absolute difference between intra frame and motion-compensated counterpart

Multiply images and do opening by reconstruction

Where the result is not zero, retain values from absolute difference image

Determine strength of flicker in the resulting image

Draw Instead!!!

57

Modified Flicker Measure Fan’s flicker measure designed for all-intra videos

Averages flicker for entire video

Flicker is only noticeable around intra frames

When using periodically-inserted intra frames, averaging over the entire video will mask the jump

For filtered videos, flicker was averaged only for first 6 frames of each GOP

58

No Filter vs. Filter

59

Temporal Post Processing Methods Mosquito Noise Reducer: 2002, [1]

Identifies moving blocks (LPF on frame difference)

DC median filter (temporal & spatial) on still blocks

No perceivable results on H.264 videos

Adaptive Anisotropic Filter: 2002, [2] Identifies the types of blocks

Applies a set of spatial filters on different block types

Deals with ringing-related noise, not relevant to H.264

Others (e.g. [3]) - Similar drawbacks

60

Intermediate Conclusions Most literature deals with MPEG2 temporal

discontinuities

Temporal discontinuities characterization not relevant to H.264

Practically no ringing in H.264

“Temporal discontinuities” is too wide a term

61

PSNR & Flicker vs. Bitrate

62

0

5

10

15

20

25

30

35

0

2

4

6

8

10

12

14

0 200 400 600 800 1000 1200

mobile PSNR & Flicker vs. bitrate

Flicker fi Flicker un PSNR fi PSNR un

33

34

35

36

37

38

39

40

0

1

2

3

4

5

0 100 200 300 400 500 600

container PSNR & Flicker vs. bitrate

Flicker fi Flicker un PSNR fi PSNR un

0

10

20

30

40

50

0

2

4

6

8

10

12

0 200 400 600 800 1000 1200

paris PSNR & Flicker vs. Bitrate

Flicker fi Flicker un PSNR fi PSNR un