TRUE-MOTION ESTIMATION WITH 3-D RECURSIVE SEARCHBLOCK MATCHING Gerard de Haan,

Paul W. A. C. Biezen

Henk Huijgen

Olukayode A. Ojo

(Philips Research Laboratories, 5600 JA Eindhoven, the Netherlands.)

This paper appears in:

Circuits and Systems for Video Technology, IEEE Transactions on

Page 368–379.388 ,Oct 1993

1

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion2

INTRODUCTION What is true motion?

Why do we find the true motion? Consumer display scan rate conversion[1]-[8]. Common drawback is decreased dynamic

resolution. Motion compensation techniques[9]-[12] are too

expensive for consumer television applications.3

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion4

1-D Recursive Search: similar to 2-D logarithmic search[22]

The candidate set (CSi) & prediction vector (Di-1):

Indicate with S rather than Di-1

as the spatial prediction vector

(pel-recursive algo. [23][24] ):

RECURSIVE SEARCH METHOD FOR TRUE ME(1/5)

5

2-D Recursive Search: two spatial prediction vectors A 1-D recursive algorithm cannot cope with

discontinuities in the velocity plane. Assumption (1):

The discontinuities in the velocity plane are spaced at a distance that enables convergence of the recursive block matcher in between two discontinuities.

Two estimators and the selection criterion:

As described in 1-DRS, updating, respectively, prediction vectors:

RECURSIVE SEARCH METHOD FOR TRUE ME(2/5)

6

2-D Recursive Search solves the run-in problem at the boundaries of moving objects.

The best implementation of 2-DC results with predictions from blocks 1 and 3 for estimators a and b, respectively:

RECURSIVE SEARCH METHOD FOR TRUE ME(3/5)

where (X,Y) is the size of block.

7

3-D Recursive Search: temporal prediction vectors Assumption (2):

The displacements between two consecutive velocity planes, due to movements in the picture, are small compared to the block size.

Rather than choosing the additional estimators c and d, applying temporal prediction vectors as additional candidates:

These convergence accelerators (CA) are taken from a block shifted diagonally over “ r ” blocks.

RECURSIVE SEARCH METHOD FOR TRUE ME(4/5)

8

3-D RS candidate set CSa & CSb:

The CA's are particularly advantageous at the top of the screen, where the spatial process starts converging.

The CA's improve the temporal consistency.

RECURSIVE SEARCH METHOD FOR TRUE ME(5/5)

9

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion10

UPDATING STRATEGY The asynchronous cyclic search (ACS):

Nbl is the output of a block counter lut is a look-up table function

The pseudorandom look-up table (for p=9):

symmetrical distribution around 0 with p updates

11

0 improves the performance for small stationary image parts but disturbs the convergence.

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion12

FURTHER EMPHASIS ON SMOOTHNESS (1/2)

The risks which jeopardize the smoothness:1) An element of the update sets may equal a multiple

of the basic period of the structure.2) "The other" estimator may not be converged, or may

be converged to wrong value that does not correspond to the actual displacement.

3) Directly after a scene change, the convergence accelerators (CAs) yield the threatening candidate.

Improve the result for risks 1) & 3): Add penalties to the error function related to the length

of the difference vector between the candidates to be evaluated:

13

Respectively, 0.4%, 0.8%, and 1.6% of the maximum error value, for the cyclic update(Sn), the convergence accelerator (CA), and the fixed 0 candidate vector.

The last candidate(0) especially requires a large penalty. Improve the result for risk 2):

The situation occurs if a periodic part enters the picture from the blanking or appears from behind an other object.

Advantage of two independent estimators would be lost.

FURTHER EMPHASIS ON SMOOTHNESS (2/2)

14

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion15

BLOCK EROSION TO ELIMINATE BLOCKING EFFECTS

Improve the result for: Eliminating the visible block structures in the

picture. Eliminating fixed block boundaries from the

vector field without blurring contours.

Finally assigned to the pixels in the quadrant:

E

F

16

H-1-1

OVERVIEW Introduction Recursive Search Method for True ME

1-D Recursive Search 2-D Recursive Search 3-D Recursive Search

Updating Strategy Further Emphasis on Smoothness Block Erosion to Eliminate Blocking Effects Evaluation Results & Experiments

Modified Mean Square Prediction Error(M2SE) Smoothness

Conclusion17

EVALUATION RESULTS & EXPERIMENTS (1/4) Modified Mean Square Prediction Error(M2SE):↓,

quality↑

s identifies the test sequence 1~5 P . L is the number of pixels in the image excluding margin.

Smoothness Indicator: S(t)↑, smoothness↑

Nb is the number of

blocks in a field.18

Experiments:

EVALUATION RESULTS & EXPERIMENTS (2/4)

19

EVALUATION RESULTS & EXPERIMENTS (3/4)

Captured from: Frame Rate Up-Conversion,陳秉昱 ,January 8,2006

20

EVALUATION RESULTS & EXPERIMENTS (4/4)

Captured from: Frame Rate Up-Conversion,陳秉昱 ,January 8,2006

21

CONCLUSION

The newly designed motion estimation algorithm is emerging as the most attractive of the tested block-matching algorithms in the application of consumer field rate conversion.

The bidirectional convergence principle enabled combination of the conflicting demands for smoothness and yet steep edges in the velocity field.

Using new test criteria, the suitability of motion estimators for television with motion compensated field rate doubling was tested. 22