Roadmap

EE591f Digital Video Processing 1

Roadmap

Introduction to Block Matching Algorithm Fast BMA

Three classes of speed-up strategies

Generalized BMA From integer-pel to fractional-pel From fixed block size to variable block size Deformable BMA (DBMA) or mesh-based BMA

Experimental results How do block size and motion accuracy affect the MCP

efficiency?


An Intuitive Way of Understanding Block Matching Algorithm (BMA)

a

b

c

f

e?

template

database

d


inquiry blockin current frame

reference frame

a b

c

d

Block Matching in Motion Estimation

a: (-3,-2)b: (-3,-1)c: (0,0)d: (1,2)


Motion Estimation and Compensation

Bjidjdisjisjie kk ),(),,(),(),( 211

Motion Compensated Prediction (MCP) residues

Motion Compensation

With the estimated motion vector, the block in the reference frame isdisplaced to generate a prediction of the inquiry block in the current frame. Such procedure is called “motion compensation”.

(d1,d2) : estimated motion vector

current frame displaced reference frame


Two Key Elements in BMA

Matching criterion: How do I measure the similarity between two blocks? Mean Square Error (MSE): L2 norm

Mean Absolute Difference (MAD): L1 norm

Search strategy: How do I find the best match of the given block? Exhaustive search: global minimum Non-exhaustive search: close to global

minimum


Goal: Find the Best Tradeoff

computational cost

variance of MCP residues


Roadmap

Introduction to Block Matching AlgorithmFast BMA

Three classes of speed-up strategiesGeneralized BMA

From fixed block size to variable block size From integer-pel to fractional-pel

Experimental results How do block size and motion accuracy affect

the MCP efficiency?


Benchmark: Exhaustive Search

It searches (2T+1)2=225 points in total

An example of window size T=7


Fast Block Matching Algorithms

Class-A (I-IV): ad-hoc speed-up strategiesClass-B (V-VII): advanced speed-up

strategies (wise use of computational resource to account for probabilities)

Class-C (VIII): hierarchical strategy

General Principle – trade complexity with performance


Fast BMA (I): 3-Step-Search

search 9+8+8=25 points


Fast BMA (II): Logarithmic Search

search at most5+4+2+3+2+8=

24 points


Fast BMA (III): Orthogonal Search

search at most2(3+2+2+2+2+2)=

26 points


Fast BMA (IV): Cross Search

search at most5+4+4+4=17 points


Why does probabilistic modeling of MV help?

Empirical pdf of motion vectors


Fast BMA (V): New 3-Step Search


New 3-Step Search: Examples


Search the 9 checking points located at a 5-by-5 window to see if the point reaching the minimum distortion is found at the center?

N

Search 5 additionalChecking points

Search 3 additionalChecking points

Is it at the corner or not?Y N

Final 3-by-3 search

Repeat the procedure in the dashed box

Y

Fast BMA (VI): 4-Step Search


4-Step Search: Examples


The Idea of Successive Refinement

Note that in all previous approaches to fast BMA, we only consider the possibility of reducing the number of search points

For each search point, we still need to calculate the matching criterion for a B-times-B block

To further reduce the complexity, we might consider reducing the cost of each matching as well


M

N

M/2

N/2

M/4N/4

Multi-resolution representation by pyramid

Multi-resolution Representation of Images


Why does Hierarchical Strategy Help?

Level-0

Level-1

Level-2ME result

ME result


Hierarchical Block Matching Algorithm (HBMA)


Example: Three-level HBMA


Fast BMA (VIII): Hierarchical Search


Summary

Why do we care fast BMA? Driven by the application demands of video

coding

Can we go beyond BMA? The block-based constraint is simple but not

appropriate for accounting for arbitrary shape of moving objects

The integer-pel accuracy is not sufficient to account for continuous nature of motion


Roadmap



From integer-pel to fractional-pel From fixed block size to variable block size Deformable BMA (DBMA) or mesh-based BMA*

Experimental results How do block size and motion accuracy affect the

MCP efficiency?


Why Do We Need Fraction-pel?


Fractional-pel BMA

original reference frame

linearinterpolation

interpolated reference frame

M

N

2M

2N


Half-pel BMA

current frame

reference frame

1

1

1

1

digits indicate physical distances


Bilinear Interpolation

(x+1,y)(x,y)

(x+1,y+1)(x,y+!)

(2x,2y) (2x+1,2y)

(2x,2y+1) (2x+1,2y+1)

O[2x,2y]=I[x,y]O[2x+1,2y]=(I[x,y]+I[x+1,y])/2O[2x,2y+1]=(I[x,y]+I[x,y+1])/2O[2x+1,2y+1]=(I[x,y]+I[x+1,y]+I[x,y+1]+I[x+1,y+1])/4

Generalize to 1/K pixel where K >2


Hierarchical Strategy for Half-pel BMA

Integer-pel

Half-pel


Beyond Half-pel Accuracy

There exist results supporting the further prediction efficiency gain from half-pel to quarter-pel; sometimes it is even worthwhile to reach 1/8-pel accuracy

The improved prediction efficiency is comprised by modestly increased computational complexity and overhead

Question: for what kind of video, finer-accuracy improves the MCP efficiency most?


Generalizations of BMA

Variable block-size matching algorithms Widely used by various video coding standards H.264 includes three variable block sizes: 4-by-

4, 8-by-8 and 16-by-16

Fractional-pel accuracy BMA Half-pel : MPEG-1/2/4, H.263/H.263+ Quarter-pel: H.264 (even 1/8-pel)

Tradeoff between overhead on motion and MCP efficiency


Variable Block-size BMA

16-by-16 8-by-8 4-by-4


BMA Strategy Adopted by H.263

16-by-16 8-by-8

Macroblock level Block level


BMA Strategy Adopted by H.264

16-by-16 8-by-16 16-by-8 8-by-8

8-by-8 4-by-8 8-by-4 4-by-4

Note: require overhead to signal which partition is adopted by the encoder


Deformable Block Matching Algorithm


Overview of DBMAThree steps:

Partition the anchor frame into regular blocks

Model the motion in each block by a more complex motion

The 2-D motion caused by a flat surface patch undergoing rigid 3-D motion can be approximated well by projective mapping

Projective Mapping can be approximated by affine mapping and bilinear mapping

Estimate the motion parameters block by block independently

Discontinuity problem cross block boundaries still remain


Affine and Bilinear Model

Affine (6 parameters): Good for mapping triangles to triangles

Bilinear (8 parameters): Good for mapping blocks to quadrangles

ybxbb

yaxaa

yxd

yxd

y

x

210

210

),(

),(

xybybxbb

xyayaxaa

yxd

yxd

y

x

3210

3210

),(

),(


Mesh-Based Motion Estimation

(a) Using a triangular mesh

(b) Using a quadrilateral mesh

A control grid is used to partition a frame into non-overlapping polygon elements. The nodal motion is constrained so that a feasible mesh is still formed with the motion.


Mesh-based vs Block-based

(a) block-based ME

(b) mesh-based ME

(c) mesh-based motion tracking

EE591f Digital Video Processing 43Mesh-based method (29.72dB)

EBMA (half-pel) (29.86dB)

Example: BMA vs Mesh-based

Target

Anchor

Predicted


Roadmap



From fixed block size to variable block size From integer-pel to fractional-pel

Experimental results How do block size and motion accuracy affect

the MCP efficiency?


Experiment Results

Frame #1 Frame #2


Motion-Compensated Prediction Residues

16-by-16 block, integer-pel, var(e)=271.8



8-by-8 block, integer-pel, var(e)=220.8



16-by-16 block, half-pel, var(e)=164.2



8-by-8 block, half-pel, var(e)=123.8

Roadmap

Documents

Transcript of Roadmap