CS231A Section 6: Problem Set 3 - Artificial...
Transcript of CS231A Section 6: Problem Set 3 - Artificial...
![Page 1: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/1.jpg)
Review 6 - !Kevin Wong!
CS231A Section 6: Problem Set 3
Kevin Wong
11/09/2012 1
![Page 2: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/2.jpg)
Review 6 - !Kevin Wong!
Announcements
• PS3 Due 2:15pm Tuesday, Nov 13
• Extra Office Hours: – Friday 6 – 8pm Huang Common Area,
Basement Level.
11/11/2011 2
![Page 3: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/3.jpg)
Review 6 - !Kevin Wong!
Topics
• Review – Optical Flow
• PS3 – Keypoints and features – SIFT Descriptors – SIFT Matching
• Projects – OpenCV with Matlab
11/09/2012 3
![Page 4: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/4.jpg)
Review 6 - !Kevin Wong!
Review: Optical Flow
• Overview • Review of Theory • Applications
11/09/2012 4
![Page 5: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/5.jpg)
Review 6 - !Kevin Wong!
Optical Flow
11/09/2012 5 Learning OpenCV, page 322
![Page 6: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/6.jpg)
Review 6 - !Kevin Wong!
Optical Flow
• A low level motion tracking algorithm that tracks apparent motion from frame to frame of a video.
• Uses brightness patterns as features, not corners.
• Key Assumptions – Brightness consistency:
11/09/2012 6
I(x, y, t� 1) = I (x+ u(x, y), y + v(x, y), t)
![Page 7: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/7.jpg)
Review 6 - !Kevin Wong!
Optical Flow
• Key Assumptions Cont. – Spatial Coherence
• Neighboring points are from the same surface and will have similar motion.
– Small Motion • The points do not move
very far between frames
11/09/2012 7 * Image from Michael Black, CS143 2003
![Page 8: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/8.jpg)
Review 6 - !Kevin Wong!
Optical Flow
• Taylor Expansion -> Brightness consistency
• We have one equation and two unknowns, motion perpendicular to the gradient, parallel to an edge is ambiguously, Aperture problem/Barber pole illusion.
11/09/2012 8
I (x+ u, y + v, t) ⇡ I(x, y, t� 1) + I
x
· u+ I
y
· v + I
t
I (x+ u, y + v, t)� I(x, y, t� 1) = I
x
·+I
y
· v + I
t
I
x
· u+ I
y
· v + I
t
⇡ 0
5I [u v]T + I
t
= 0
![Page 9: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/9.jpg)
Review 6 - !Kevin Wong! 11/09/2012 9
![Page 10: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/10.jpg)
Review 6 - !Kevin Wong!
Lucas-Kanade Flow
• Solvable when ATA is invertible – Eigenvectors of ATA describe the patch
• Good patches turn out to be Harris corners.
• Improvements? – Spatial consistency doesn’t really hold across
the scene. • Solution? Motion segmentation: Divide the region
into blocks, cluster pixel movement, reassign pixels to closets clusters and repeat.
11/09/2012 10
![Page 11: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/11.jpg)
Review 6 - !Kevin Wong!
Optical flow applications
• Object tracking via motion segmentation • Traffic analysis • Crowd flow/behavior analysis.
11/09/2012 11
![Page 12: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/12.jpg)
Review 6 - !Kevin Wong!
SIFT
• Motivations • What makes a point “interesting” • What to do with Keypoints • SIFT detector • SIFT descriptor • Object Recognition with SIFT
11/11/2011 12
![Page 13: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/13.jpg)
Review 6 - !Kevin Wong!
Where are we?
11/11/2011
Scene Object Local Patches Pixels Camera Models
13
CV
Tim
elin
e
![Page 14: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/14.jpg)
Review 6 - !Kevin Wong!
What to identify? (Keypoints)
• Salient points that would be present across a set of (reasonably) related images – Repeatable – Distinctive
11/11/2011 14
![Page 15: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/15.jpg)
Review 6 - !Kevin Wong!
What information? (Descriptors)
11/11/2011 15
![Page 16: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/16.jpg)
Review 6 - !Kevin Wong!
SIFT
• Ubiquitous: 10,000+ Citations • David Lowe, 2001 and 2004, from UBC • Both a detector and a descriptor • Invariant to:
– Illumination – Scale – Rotation – Affine – Perspective
11/11/2011 16
![Page 17: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/17.jpg)
Review 6 - !Kevin Wong!
SIFT: Keypoint Detector • Difference of Gaussians using Scale Space
Pyramid • Section 3 and 4 of Lowe, 2004
11/11/2011 17
−
=
![Page 18: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/18.jpg)
Review 6 - !Kevin Wong!
Extract and Prune max(DoG)
• A point is extreme if it larger/smaller than its 26 neighboring points
• Prune for: – Low Contrast – Edge Points
11/11/2011 18
![Page 19: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/19.jpg)
Review 6 - !Kevin Wong!
What can we do with keypoints?
11/11/2011 19
![Page 20: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/20.jpg)
Review 6 - !Kevin Wong!
SIFT Descriptor • Sections 5 and 6 in Lowe, 2004. • Inspired by neurological research and models • Keypoint is center of square patch of pixels, blurred at the scale of the
keypoint • Construction of Orientation Histogram for each 4x4 set of pixels • All pixels are rotated by the orientation of the keypoint
11/11/2011 20
![Page 21: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/21.jpg)
Review 6 - !Kevin Wong!
Steps of finding descriptor (for PS3) Given a keypoint P 1. Find gradient:
1. Apply Gaussian filter to the image using the scale of P as sigma (standard deviation)
2. Find gradient in x and y directions (down - up, right - left) 3. Magnitude: sqrt(Ix2+Iy2); Theta: atan2(Iy, Ix) 4. Account for keypoint orientation.
2. Find the patch associated with P 3. For each bin b of the 4 x 4 bins in the patch
1. Initialize the histogram of the 8 angle bins 2. For each pixel p in b
1. Find the angle bin it falls into 2. In the histogram, increase the value of this angle bin by the gradient
magnitude of p 3. Append this histogram to the end of the descriptor
11/11/2011 21
![Page 22: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/22.jpg)
Review 6 - !Kevin Wong!
Many little details • Many, many experimentally determined
parameters for descriptor. • The standard patch size is 16
– Feature Vector = 16 patches * 8 bins = 128 elements • Normalize each feature vector • Avoid large (and potentially erroneous) gradients
by capping each element at 0.2 • Add extra blur for camera sensor • [More steps…] • We try to make the process as linear as
possible…
11/11/2011 22
![Page 23: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/23.jpg)
Review 6 - !Kevin Wong!
The Matching Problem
• Locate arbitrary objects despite environmental difficulties
11/11/2011 23
![Page 24: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/24.jpg)
Review 6 - !Kevin Wong!
Object Recognition with SIFT (Sec 7, Lowe 2004)
• Step 1: Feature matching • Step 2: Hough transform in pose space • Step 3: Geometric verification via affine
transformation
11/11/2011 24
![Page 25: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/25.jpg)
Review 6 - !Kevin Wong!
Step 1: Feature matching
• A match is determined by distance between closest neighbor and second closest neighbor – Euclidean distance between descriptor
vectors • Nearest Neighbor problem
– k-d tree is inefficient – Best-Bin-First (BBF) (Beis and Lowe, 1997),
modified from k-d tree, giving approximated result
11/11/2011 25
![Page 26: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/26.jpg)
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
• Goal: given test image and training image, find object pose
• Input: Descriptor matches (pi -> pi’) of an object, many are false matches
• Output: estimated object pose
• A match is specified by 4 parameters <x, y, scale, orientation>
11/11/2011 26
![Page 27: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/27.jpg)
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
1. Discrete bins in 4D space; 2. Assign each match to the bin whose
object pose is consistent with it; 3. Find bins with > 3 votes.
11/11/2011 27
![Page 28: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/28.jpg)
Review 6 - !Kevin Wong!
Step 2: Hough transform in pose space
• Have to use broad bins since there are only 4 parameters but 6 dof – bin size of 30 degrees for orientation, a factor
of 2 for scale, etc. For the problem, you can use a uniform bins, which isn’t optimal, good enough for the problem.
11/11/2011 28
![Page 29: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/29.jpg)
Review 6 - !Kevin Wong!
Step 3: Geometric verification via affine transformation
• Verify each bin with at least 3 entries • 3 matches determine an affine transformation
(An approximation of finding the fundamental matrix which requires more matches)
11/11/2011 29
![Page 30: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/30.jpg)
Review 6 - !Kevin Wong!
Parameters of Affine Transformation
• Affine transformation:
• Least-squares solution for the best affine projection parameters
11/11/2011 30
![Page 31: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/31.jpg)
Review 6 - !Kevin Wong!
Results
11/11/2011 31
![Page 32: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/32.jpg)
Review 6 - !Kevin Wong!
Object Matching in PS3
• PS3 uses a simplified version of object matching – Match on keypoints
descriptors – Use Hough to determine
bounding box parameters.
11/11/2011 32
http
://w
ww
.cs.
was
hing
ton.
edu/
hom
es/a
nkit/
rese
arch
_pro
ject
s/su
ra/s
ura-
web
site
/sift
.htm
![Page 33: CS231A Section 6: Problem Set 3 - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/ta... · 2014-01-06 · 1 11/09/2012 . Kevin Wong! Review 6 - ! Announcements](https://reader034.fdocuments.us/reader034/viewer/2022050305/5f6e24353f48ad4d6215451e/html5/thumbnails/33.jpg)
Review 6 - !Kevin Wong!
Projects • OpenCV with Matlab • http://xanthippi.ceid.upatras.gr/people/evangelidis/
matlab_opencv/ – Guide to building OpenCV Mex files for windows that
can be called in Matlab. – Note that the Guide is for OpenCV 2.1. – Macs: Tested using OSX 10.8, Xcode 4.5, OpenCV
2.4.2. • Other options:
– Use files for I/O to standalone OpenCV applications – Use OpenCV exclusively, with CGAL for potential
matrix operations.
11/09/2012 33