DR. J. Shanbehzadeh Shanbehzadeh@gmail Sara Omid
description
Transcript of DR. J. Shanbehzadeh Shanbehzadeh@gmail Sara Omid
DR. J. [email protected]
Sara Omid
Science and Research Branch of Azad University
Lecture 06Computing Optical Flow
Machine Vision
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Table of Contents2
Optical Flow ExamplesOptical FlowHorn & Schunck Optical Flow
β’ Interpretation Of Optical Flow eqβ’ Horn & Schunckβ’ Derivative Masksβ’ Laplacianβ’ Algorithm-1β’ Synthetic Imagesβ’ Horn & Schunck Results
Lucas & KanadeComments
3
Optical Flow Examples
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples4
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples5
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples6
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples7
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples8
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples9
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples10
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples11
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples12
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow Examples13
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
14
Optical Flow
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Optical Flow15
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Applications
β’ Motion based segmentation
β’ Structure from Motion(3D shape and Motion)
β’ Alignment (Global motion compensation) Camcorder video stabilization UAV Video Analysis
β’ Video Compression
16
Horn & Schunck
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck Optical Flow17
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Brightness constancy assumptionπ (π₯ , π¦ , π‘ )= π (π₯+ππ₯ , π¦+ππ¦ , π‘+ππ‘ )
π (π₯ , π¦ , π‘ )= π (π₯ , π¦ ,π‘ )+π πππ₯ ππ₯+π ππ π¦ ππ¦+
π πππ‘ ππ‘
π π₯ππ₯+ π π¦ππ¦+ π π‘ ππ‘=0
π π₯π’+ π π¦π£+ π π‘=0
Taylor Series
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Interpretation Of Optical Flow eq18
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
π π₯π’+ π π¦π£+ π π‘=0
d = normal flowP = parallel flowπ=
π π‘β π π₯
2+ π π¦2
π£=βπ π₯
π π¦π’β
π π‘π π¦
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck19
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
β¬ { ( π π₯π’+ π π¦π£+ π π‘ )2+Ξ» (π’π₯2+π’π¦
2 +π£π₯2+π£π¦
2 ) }ππ₯ππ¦Brightness Constancy
Smoothness Constraint
min
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck20
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Laplacian
=
=
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Derivative Masks21
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
fx fy
ft
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Derivative Masks (Roberts)22
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Apply first mask to 1st image
Second mask to 2nd image
Add the responses to get fx, fy, ft
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian23
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian24
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
Discrete version
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Laplacian25
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
variational calculus
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Algorithm-126
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
β’ k=0
β’ Initialize uk and vk
β’ Repeat until some error measure is satisfied(converges)
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Synthetic Images27
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Horn & Schunck Results28
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
29
Lucas & Kanade
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade30
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
β’ Optical Flow eq
A u = ft
π π₯π’+ π π¦π£=β π π‘
β’ Consider 3 by 3 window
. . .
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade31
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
A U = ft
AT A U = AT ft
U = (ATA)-1 AT ft
Pseudo Inverse
Least Squares Fit
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade32
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade33
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
[ β π π₯π2 β π π₯π π π¦π
β π π₯π π π¦π β π π¦π2 ][π’π£ ]=[ββ π π₯π π π‘π
ββ π π¦π π π‘π]
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade34
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade35
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade36
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Lucas & Kanade37
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
38
Comments
R. C
. Gon
zale
z, a
nd R
. E. W
oods
, Dig
ital I
mag
e P
roce
ssin
g, N
ew J
erse
y: P
rent
ice
Hal
l, 3rd
edi
tion,
200
8.
Comments39
Optical Flow Examples
Optical Flow
Horn & Schunck Optical Flow
Interpretation Of Optical Flow eq
Horn & Schunck
Derivative Masks
Laplacian
Algorithm-1
Synthetic Images
Horn & Schunck Results
Lucas & Kanade
Comments
β’ HornβSchunck and LucasβKanade optical methods work only for small motion.
β’ If object moves faster, the brightness changes rapidly,
β’ 2x2 or 3x3 masks fail to estimate spatiotemporal derivatives.
β’ Pyramids can be used to compute large optical flow vectors.