Motion Detail Preserving Optical Flow Estimation
description
Transcript of Motion Detail Preserving Optical Flow Estimation
![Page 1: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/1.jpg)
Motion Detail Preserving Optical Flow Estimation
Li Xu1, Jiaya Jia1, Yasuyuki Matsushita2
1 The Chinese University of Hong Kong 2 Microsoft Research Asia
![Page 2: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/2.jpg)
Conventional Optical Flow
• Middlebury Benchmark [Baker et al. 07]• Dominant Scheme: Coarse-to-Fine Warping
![Page 3: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/3.jpg)
Large Displacement Optical Flow
• Region Matching [Brox et al. 09, 10]• Discrete Local Search [Steinbrucker et al. 09]
![Page 4: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/4.jpg)
Both Large and Small Motion Exist
• Capture large motion• Preserve sub-pixel accuracy
![Page 5: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/5.jpg)
Our Work
• Framework– Extended coarse-to-fine motion estimation for both
large and small displacement optical flow• Model– A new data term to selectively combine constraints
• Solver– Efficient numerical solver for discrete-continuous
optimization
![Page 6: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/6.jpg)
Outline
• Framework– Extended coarse-to-fine motion estimation for both
large and small displacement optical flow• Model– A new data term to selectively combine constraints
• Solver– Efficient numerical solver for discrete-continuous
optimization
![Page 7: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/7.jpg)
The Multi-scale Problem
![Page 8: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/8.jpg)
The Multi-scale Problem
Ground truthGround truthGround truth
![Page 9: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/9.jpg)
The Multi-scale Problem
Ground truthGround truthGround truth
![Page 10: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/10.jpg)
Ground truth
…Estimate EstimateEstimate
Ground truthGround truth
![Page 11: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/11.jpg)
The Multi-scale Problem
• Large discrepancy between initial values and optimal motion vectors
• Our solution – Improve flow initialization to reduce the reliance
on the initialization from coarser levels
![Page 12: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/12.jpg)
Extended Flow Initialization
• Sparse feature matching for each level
![Page 13: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/13.jpg)
Extended Flow Initialization
• Identify missing motion vectors
![Page 14: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/14.jpg)
Extended Flow Initialization
• Identify missing motion vectors
![Page 15: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/15.jpg)
Extended Flow Initialization
…
![Page 16: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/16.jpg)
…
Extended Flow Initialization
Fuse
![Page 17: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/17.jpg)
Outline
• Framework: extended initialization for coarse-to-fine motion estimation
• Model: selective data term • Efficient numerical solver
![Page 18: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/18.jpg)
Data Constraints
• AverageI
xI
1 1(u, x) (u, x) (u, x)2 2DE D D
I 2 1(u, x) (x u) (x)I ID
I 2 1(u, x) (x u) (I I x)D • Gradient constancy
• Color constancy
I 2 1(u, x) (x u) (I I x)D
![Page 19: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/19.jpg)
• Pixels moving out of shadow
Problems
pI 1 1p(u , ) 6.63D
• Color constancy is violated
I Ip1 1 p1 11 (u , ) (u , ) = 3.482
p pD D
• Average:
p1u : ground truth motion of p1
• Gradient constancy holdsp 1I 1 p(u , ) 0.32D
![Page 20: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/20.jpg)
• Pixels undergoing rotational motion
Problems
• Color constancy holds
• Gradient constancy is violatedp2u : ground truth motion of p2
p 2I 2 p(u , ) 4.20D
• Average:
I Ip2 2 p2 21 (u , ) (u , ) = 2.242
p pD D
pI 2 2p(u , ) 0.29D
![Page 21: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/21.jpg)
Our Proposal
• Selectively combine the constraints
where
I Ix
(u, ) (x) (u,x) (1 (x)) (u,x)DE D D 2(x) : {0,1}
I Ix
(u, ) (u,x(x) 1) ( ) (u,x)(x)DE D D
![Page 22: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/22.jpg)
Comparisons
RubberWhale Urban22
2.5
3
3.5
4
4.5
5
colorgradientaverageours
AAE
![Page 23: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/23.jpg)
Outline
• Framework: extended initialization for coarse to fine motion estimation
• Model: selective data term
• Efficient numerical solver
I Ix
(x) (u, x) (1 (x)) (u, x)D D
![Page 24: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/24.jpg)
Energy Functions and Solver
• Total energy
• Probability of a particular state of the system
(u, )1(u, ) EP eZ
I Ix
(u, ) (x) (u, x) (1 (x)) (u, x) ( u, x)E D D S
(u, )1(u, ) EP eZ
(u, )1(u, ) EP eZ
Ix
I(x) (u, x) (1 (x)) (u, x)(u, ) ( u, x)E SD D
![Page 25: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/25.jpg)
Mean Field Approximation
• Partition function
• Sum over all possible values of α
(u, )
{u} { 0,1}
EZ e
I Ix
( u,x) (u, x) ((x) (x)
{ 0,
1 ) (u, x)
{u 1}}
x
S D D
e e
(u, x)(u, x) II
x
1{ ( u,x) ln( )}
{u}
DDS e e
e
. . .
The effective potential Eeff (u) [Geiger & Girosi, 1989]
![Page 26: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/26.jpg)
• Optimal condition (Euler-Lagrange equations)
• It decomposes to
II (u, x)(u, x)
x
1(u) ( u,x) ln( )DDeffE S e e
I I
I II I
(u,x) (u,x)
u I u I(u,x) (u,x)(u,x) (u,x)
u
(u, x) (u, x)
div( ( u,x)) 0
D D
D DD D
e eD De e e e
S
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S
I I
I II I
(u,x) (u,x)
(u,x) (u,u I u I
u
x)(u,x) (u,x)(u, x) (u, x)
div( ( u,x)) 0
D D
D DD D
e ee e e e
D D
S
( )x 1 ( )x
{
![Page 27: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/27.jpg)
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {
![Page 28: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/28.jpg)
Algorithm Skeleton
• For each level
• Extended Flow Initialization (QPBO)• Continuous Minimization (Iterative reweight)– Update– Compute flow field (Variable Splitting)
I I( (u,x) (u,x))
1(x)1 D De
u I u I u(x) (u, x) (1 (x)) (u, x) div( ( u,x)) 0D D S {
![Page 29: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/29.jpg)
Results
Averaging constraints Ours
Difference
![Page 30: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/30.jpg)
Middlebury Dataset
EPE=0.74
![Page 31: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/31.jpg)
Results from Different Steps
Coarse-to-fine
Extended coarse-to-fine
![Page 32: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/32.jpg)
EPE=0.15 rank =1
EPE=0.24 rank =1
![Page 33: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/33.jpg)
Large Displacement
Overlaid Input
![Page 34: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/34.jpg)
Large Displacement
• Motion Estimates
Coarse-to-fine Our Result Warping Result
![Page 35: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/35.jpg)
Comparison
• Motion Magnitude Maps
LDOP [Brox et al. 09 ] [Steinbrucker et al. 09] Ours
![Page 36: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/36.jpg)
More Results
Overlaid Input
![Page 37: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/37.jpg)
Conventional Coarse-to-fine Our Result
![Page 38: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/38.jpg)
More Results
Overlaid Input
![Page 39: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/39.jpg)
Coarse-to-fine Our Result
![Page 40: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/40.jpg)
Conclusion
• Extended initialization (Framework)• Selective data term (Model)• Efficient numerical scheme (Solver)
• Limitations– Featureless motion details – Large occlusions
![Page 41: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/41.jpg)
Thank you!
![Page 42: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/42.jpg)
More Results
Overlaid Input
![Page 43: Motion Detail Preserving Optical Flow Estimation](https://reader035.fdocuments.us/reader035/viewer/2022062410/568165d2550346895dd8e09c/html5/thumbnails/43.jpg)
Coarse-to-fine Our Results