A dvanced C omputer V ision Feature-based Alignment
description
Transcript of A dvanced C omputer V ision Feature-based Alignment
Advanced Computer Vision
Feature-based AlignmentLecturer: Lu Yi & Prof. Fuh
Content• Pre-requisites: mainly about camera model• Alignment Algorithm• RANdom Sample Consensus (RANSAC)• Pose Estimation
Sources• EGGN 512 – Computer Vision course by Prof. Willliam Hoff• http://inside.mines.edu/~whoff/courses/EENG512/• Playlist@Youtube:• http://www.youtube.com/playlist?list=PL4B3F8D4A5CAD8DA3
• Digital Visual Effects course by Prof. Yung-Yu Chuang• http://www.csie.ntu.edu.tw/~cyy/courses/vfx/14spring/news/
• Both are highly recommended
Taught Way• Cover the main parts of textbook chapter 6• Lecture mostly depended on EGGN 512 course videos
Frames of Reference• Image frames are 2D; other are 3D
• The “pose” (position and orientation) of a 3D rigid body has 6 degrees of freedom
Pre-requisites• Image Formation• Geometric Transformations (2D-2D, 3D-3D, 3D-2D)• Camera Model (Camera Matrix = Intrinsic + Extrinsic camera matrix)
• All related contents are in textbook chapter 2: Image formation
Pre-requisites• Review1. Projective Transformation2. Camera Matrix• EGGN 512 – Lecture 5-1 3D-2D Transforms• http://
www.youtube.com/watch?v=DFNjOUMuecU&index=11&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 5-2 3D-2D Transforms• http://
www.youtube.com/watch?v=5gesrLgNuQo&list=PL4B3F8D4A5CAD8DA3
Alignment Algorithm• EGGN 512 – Lecture 14-1 Alignment• http://www.youtube.com/watch?v=UcU4814hvR8&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-2 Alignment• http://
www.youtube.com/watch?v=XxEKMecNZk0&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 14-3 Alignment• http://www.youtube.com/watch?v=lSXfv4baMwk&list=PL4B3F8D4A5CAD8DA3• Slides • http://inside.mines.edu/~
whoff/courses/EENG512/lectures/15-AlignmentNonlinear.pdf
Alignment Algorithm• Feature-based methods: only use feature points to estimate
parameters• Features: SIFT, SURF, Harris .etc.
Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature
matches• It is possible to use more complicated models such as affine or
perspective• For example, assume M is a 2x2 matrix
• Find M with the least square error
yx
mmmm
yx
2221
1211
''
n
i
pMp1
2'
Determine pairwise alignment
• Overdetermined system
yx
mmmm
yx
2221
1211
''
'1221211
'1121111
ymymx
xmymx
'
'
'2
'1
'1
22
21
12
11
22
11
11
0000
0000
00
n
n
nn
nn
yx
xyx
mmmm
yxyx
yxyx
yx
Normal Equation• Given an overdetermined system
• the normal equation is that which minimizes the sum of the square differences between left and right sides
• Why?
bAx
bAAxA TT
Normal Equation 2)( bAxx E
n
m
nmn
m
b
b
x
x
aa
aa
:::
:
...::::::
... 1
1
1
111
nxm, n equations, m variables
Normal Equation
n
m
jjnj
i
m
jjij
m
jjj
n
i
m
jjnj
m
jjij
m
jjj
bxa
bxa
bxa
b
b
b
xa
xa
xa
1
1
11
11
1
1
11
:
:
:
:
:
:
bAx
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
Normal Equation
n
ii
m
jjij bxaE
1
2
1
2)( bAxx
1
0xE
n
iii
n
ij
m
jiji baxaa
11
1 11 22
11 1
2 i
n
ii
m
jjij abxa
)(20 bAAxAx
TT
E bAAxA TT →
Normal Equation 2bAx
Normal Equation
bbxbAxbAAxAx
bbbAxAxbAxAxbAxbAxbAxbAx
bAxbAx
bAx
TTTTTTT
TTTTTT
TTT
TT
T
2
bAAxAx
TT 22 E
Determine pairwise alignment• p’=Mp, where M is a transformation matrix, p and p’ are feature
matches• For translation model, it is easier.
• What if the match is false? Avoid impact of outliers. (Ans: RANSAC)
n
iiiii yymxxmE
1
2'2
2'1
1
0mE
Application: Image Stitching• Stitching = alignment + blending
geometricalregistration
photometricregistration
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°
Application: Image Stitching• Panorama – the whole picture
• Compact Camera FOV = 50 x 35°• Human FOV = 200 x 135°• Panoramic Mosaic = 360 x 180°
Application: Image Stitching• Example:• http://graphics.stanford.edu/courses/cs178/applets/projection.html
Application: Face Alignment• Face alignment is the key preprocessing step for face recognition
Application: Face Alignment• Examples from LFW dataset:
RANSAC Algorithm• RANSAC = RANdom Sample Consensus• An iterative method to estimate parameters of a mathematical model
from a set of observed data which contains outliers [Wiki]• Compare to robust statistics• Given N data points xi, assume that majority of them are generated from a
model with parameters , try to recover .
• Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381-395.
RANSAC AlgorithmRun k times: (1) draw n samples randomly (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, cOutput with the largest c
How many times?How big? Smaller is better
How to define?Depends on the problem.
RANSAC Algorithm (How to determine K)
p: probability of real inliersP: probability of success after k trials
knpP )1(1 n samples are all inliers
a failurefailure after k trials
)1log()1log(
npPk
n p k
3 0.5 356 0.6 976 0.5 293
for P=0.99
RANSAC Algorithm (Example: line fitting)
RANSAC Algorithm (Example: line fitting)
n=2
RANSAC Algorithm (Example: line fitting)
RANSAC Algorithm (Example: line fitting)
RANSAC Algorithm (Example: line fitting)
c=3
RANSAC Algorithm (Example: line fitting)
c=3
RANSAC Algorithm (Example: line fitting)
c=15
RANSAC Algorithm• EGGN 512 – Lecture 27-1 RANSAC• http://www.youtube.com/watch?v=NKxXGsZdDp8&list=PL4B3F8D4A5
CAD8DA3&index=54
RANSAC Algorithm (Example: Homography)
RANSAC Algorithm (Example: Homography)
RANSAC Algorithm (Example: Homography)
Pose Estimation• EGGN 512 – Lecture 16-1 Pose Estimation• http://
www.youtube.com/watch?v=kq3c6QpcAGc&list=PL4B3F8D4A5CAD8DA3
• EGGN 512 – Lecture 17-1 Pose from Lines• http://
www.youtube.com/watch?v=D_4eUoqgWdc&list=PL4B3F8D4A5CAD8DA3
Pose Estimation• EGGN 512 – Lecture 18-1 SVD• http://
www.youtube.com/watch?v=C852P1JrHXI&list=PL4B3F8D4A5CAD8DA3
• EGGN 512 – Lecture 18-2 SVD• http://
www.youtube.com/watch?v=aIkzK4CdYes&list=PL4B3F8D4A5CAD8DA3
Pose Estimation• EGGN 512 – Lecture 19-1 Linear Pose Estimation• http://
www.youtube.com/watch?v=HojSSrxsB4Q&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-2 Linear Pose Estimation• http://
www.youtube.com/watch?v=ik8dFybnyPY&list=PL4B3F8D4A5CAD8DA3• EGGN 512 – Lecture 19-3 Linear Pose Estimation• http://
www.youtube.com/watch?v=OS5b-3Xfn1M&list=PL4B3F8D4A5CAD8DA3
END