Optical Flow Methods
description
Transcript of Optical Flow Methods
Optical Flow Methods
CISC 489/689Spring 2009
University of Delaware
Outline
• Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods
• Lucas-Kanade Meets Horn and Schunck• 3D Regularization• Techniques for solving optical flow• Confidence Measures in Optical Flow
Optical Flow Constraint
0
0
),,(),,(
),,(),,(
tyx fvfuftf
dtdy
yf
dtdx
xf
tyxftft
yfy
xfxtyxf
tyxftttvytuxf
0limit takingand by Dividing tt
),( tvytux
),( yx
),,( tyxf
),,( ttyxf
Interpretation
Constraint Line
tT
tyx
fvu
f
fvfuf
0
yx ff ,
v
u 0
1
vu
fff tyx
Lucas-Kanade Method
* = JK J
1*),( v
uJKvuELK
2
2
2
2
2
2
01
ttytx
tyyyx
txyxx
ttytx
tyyyx
txyxx
fffffffffffffff
J
vu
fffffffffffffff
Lucas-Kanade Method
• Local Method, window based• Cannot solve for optical flow everywhere• Robust to noise
5.7 15
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
Dense optical Flow
?)(),( Minimize 2 tyx fvfufvuE
Lacks SmoothnessFigures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
Horn and Schunck MethoddxdyvufvfufvuE tyxHS )()(),( 222
Euler-Lagrange Equations
Horn and Schunck Method
510 610Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
• Global Method • Estimates flow everywhere• Sensitive to noise• Oversmooths the edges
Why combine them?
• Need dense flow estimate• Robust to noise• Preserve discontinuities
Combining the two…
1*),( v
uJKvuELK
dxdyvufvfufvuE tyxHS )()(),( 222
dxdyvuvu
JKvuECLG )(1
*),( 22
Combined Local Global Method
dxdyvuvu
JKvuECLG )(1
*),( 22
Euler-Lagrange Equations
AverageError
Standard Deviation
Lucas&Kanade 4.3(density 35%)
Horn&Schunk 9.8 16.2Combining local
and global 4.2 7.7
Table: Courtesy - Darya Frolova, Recent progress in optical flow
Preserving discontinuities• Gaussian Window does not preserve
discontinuities• Solutions
– Use bilateral filtering
– Add gradient constancy
dxdyvuvu
JKvuE bilbil )(1
*),( 22
dxdyffvuvu
JKvuE tgrad2
122 )()(
1*),(
Bilateral support window
Images: Courtesy, Darya Frolova, Recent progress in optical flow
Robust statistics – simple exampleFind “best” representative for the set of numbers
min2 i
ixxEL2: mini
ixxEL1:
xix
Influence of xi on E: xi → xi + ∆
)mean( ixx
Outliers influence the most
ixx proportional to
xxEE ioldnew 2
)median( ixx
Majority decides
equal for all xi
oldnew EE
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
Robust statistics
many ordinary people a very rich man
Oligarchy
Votes proportional to the wealth
Democracy
One vote per person
wealth
like in L2 norm minimization like in L1 norm minimization
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
Combination of two flow constraints
robust: L1
x
robust in presence of outliers– non-smooth: hard to analyze
usual: L2
easy to analyze and minimize– sensitive to outliers
2x
robust regularized
smooth: easy to analyze robust in presence of outliers
22 x
ε
video
warpedwarped IIII min
),,( ; )1,,( tyxIItvyuxIIwarped
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performanceSlide: Courtesy - Darya Frolova, Recent progress in optical flow
Robust statistics
3D Regularization
• accounted for spatial regularization
• If velocities do not change suddenly with time, can we regularize in time as well?
)( 22 vu
3D Regularization
tv
yv
xvv
tu
yu
xuu
3
3
dxdydtvuvu
JKvuETXCLG )(
1*),( 2
32
3],0[3
Multiresolution estimation
21
image Iimage J
Gaussian pyramid of image 1 Gaussian pyramid of image 2
Image 2image 1
run iterative estimation
run iterative estimation
warp & upsample
.
.
.
Multi-resolution Lucas Kanade Algorithm
Compute Iterative LK at highest level•For Each Level i• Take flow u(i-1), v(i-1) from level i-1
• Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.
• Multiply u*(i), v*(i) by 2
• Compute It from a block displaced by u*(i), v*(i)
• Apply LK to get u’(i), v’(i) (the correction in flow)
• Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)
Comparison of errors
For Yosemite sequence with cloudsTable: Courtesy - Darya Frolova, Recent progress in optical flow
Solving the system
fAu
How to solve?
2 components of success: fast convergence
good initial guess
Start with some initial guess
and apply some iterative method
initialu
Relaxation schemes have smoothing property:
Only neighboring pixels are coupled in relaxation
scheme
It may take thousands of iterations to propagate
information to large distance
. . . . . . . . . . . .
Relaxation smoothes the error
Relaxation smoothes the error Examples
2D case:
1D case:
Error of initial guess Error after 5 relaxation Error after 15 relaxations
Idea: coarser grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
initial grid – fine grid
coarse grid – we take every second point
Multigrid 2-Level V-Cycle
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
3. Solve for the error on the coarse level ⇒ good error estimation
4. Transfer error to the fine level
5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
Coarse grid - advantages
initialu
fAu
Coarsening allows:
go to the coarsest grid
solve here the equation
to find initialu
interpolate to the
finer grid
initialu
Multigrid approach – Full scheme
Confidence Metric
• Intrinsic in Local Methods• How to evaluate for global methods?
– Edge strength?• Doesn’t work (Barron et al.,1994)
Confidence Metric
• Histogram of error contribution
Error
Number of pixels
Confidence Metric
More Results
More Results
Further Reading• Combining the advantages of local and global optic flow methods
(“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005
• High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005• Towards ultimate motion estimation: Combining highest accuracy with
real-time performance. A. Bruhn, J. Weickert, 2005• Bilateral filtering-based optical flow estimation with occlusion
detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006