Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3....
Transcript of Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3....
![Page 1: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/1.jpg)
Keypoint‐based Recognition
Computer Vision
CS 543 / ECE 549
University of Illinois
Derek Hoiem
03/04/10
![Page 2: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/2.jpg)
General Process of Object Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
![Page 3: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/3.jpg)
General Process of Object Recognition
Example: Keypoint-based Instance Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
A1
A2
A3
Last Class
![Page 4: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/4.jpg)
Overview of Keypoint Matching
K. Grauman, B. Leibe
Npi
xels
N pixels
Af
e.g. color
Bf
e.g. color
A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key-points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 5: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/5.jpg)
General Process of Object Recognition
Example: Keypoint-based Instance Recognition
Specify Object Model
Generate Hypotheses
Score Hypotheses
Resolution
A1
A2
A3
Affine Parameters
Choose hypothesis with max score above threshold
# Inliers
Affine-variant point locations
This Class
![Page 6: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/6.jpg)
Matching Keypoints
• Want to match keypoints between:1. Training images containing the object2. Database images
• Given descriptor x0, find two nearest neighbors x1, x2 with distances d1, d2
• x1 matches x0 if d1/d2 < 0.8– This gets rid of 90% false matches, 5% of true matches in Lowe’s study
![Page 7: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/7.jpg)
Affine Object Model
• Accounts for 3D rotation of a surface under orthographic projection
![Page 8: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/8.jpg)
Affine Object Model
• Accounts for 3D rotation of a surface under orthographic projection
Scaling/skew Translation
How many matched points do we need?
![Page 9: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/9.jpg)
Finding the objects1. Get matched points in database image
2. Get location/scale/orientation using Hough voting– In training, each point has known position/scale/orientation
wrt whole object– Bins for x, y, scale, orientation
• Loose bins (0.25 object length in position, 2x scale, 30 degrees orientation)
• Vote for neighboring bins also (so 16 votes per point)
3. Geometric verification– For each bin with at least 3 keypoints– Iterate between least squares fit and checking for inliers and
outliers4. Report object if > T inliers (T is typically 3, can be
computed by probabilistic method)
![Page 10: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/10.jpg)
Matched objects
![Page 11: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/11.jpg)
View interpolation• Training
– Given images of different viewpoints
– Cluster similar viewpoints using feature matches
– Link features in adjacent views
• Recognition– Feature matches may be
spread over several training viewpoints
⇒ Use the known links to “transfer votes” to other viewpoints
Slide credit: David Lowe
[Lowe01]
![Page 12: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/12.jpg)
Applications• Sony Aibo(Evolution Robotics)
• SIFT usage– Recognize docking station
– Communicate with visual cards
• Other uses– Place recognition– Loop closure in SLAM
K. Grauman, B. Leibe 13Slide credit: David Lowe
![Page 13: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/13.jpg)
Location Recognition
Slide credit: David Lowe
Training
[Lowe04]
![Page 14: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/14.jpg)
Fast visual search
“Scalable Recognition with a Vocabulary Tree”, Nister and Stewenius, CVPR 2006.
“Video Google”, Sivic and Zisserman, ICCV 2003
![Page 15: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/15.jpg)
Slide
110,000,000Images in
5.8 Seconds
Slide Credit: Nister
![Page 16: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/16.jpg)
Slide Slide Credit: Nister
![Page 17: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/17.jpg)
Slide Slide Credit: Nister
![Page 18: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/18.jpg)
Slide Credit: NisterSlide
![Page 19: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/19.jpg)
Key Ideas
• Visual Words– Cluster descriptors (e.g., K‐means)
• Inverse document file– Quick lookup of files given keypoints
tf-idf: Term Frequency – Inverse Document Frequency
# words in document
# times word appears in document
# documents
# documents that contain the word
![Page 20: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/20.jpg)
Recognition with K‐tree
Following slides by David Nister (CVPR 2006)
![Page 21: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/21.jpg)
![Page 22: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/22.jpg)
![Page 23: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/23.jpg)
![Page 24: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/24.jpg)
![Page 25: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/25.jpg)
![Page 26: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/26.jpg)
![Page 27: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/27.jpg)
![Page 28: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/28.jpg)
![Page 29: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/29.jpg)
![Page 30: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/30.jpg)
![Page 31: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/31.jpg)
![Page 32: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/32.jpg)
![Page 33: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/33.jpg)
![Page 34: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/34.jpg)
![Page 35: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/35.jpg)
![Page 36: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/36.jpg)
![Page 37: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/37.jpg)
![Page 38: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/38.jpg)
![Page 39: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/39.jpg)
![Page 40: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/40.jpg)
![Page 41: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/41.jpg)
![Page 42: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/42.jpg)
![Page 43: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/43.jpg)
![Page 44: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/44.jpg)
Performance
![Page 45: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/45.jpg)
More words is better ImprovesRetrieval
ImprovesSpeed
Branch factor
![Page 46: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/46.jpg)
Higher branch factor works better (but slower)
![Page 47: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/47.jpg)
Video Google System
1. Collect all words within query region
2. Inverted file index to find relevant frames
3. Compare word counts4. Spatial verification
Sivic & Zisserman, ICCV 2003
• Demo online at : http://www.robots.ox.ac.uk/~vgg/research/vgoogle/index.html
48 K. Grauman, B. Leibe
Query region
Retrieved fram
es
![Page 48: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/48.jpg)
Application: Large‐Scale Retrieval
K. Grauman, B. Leibe 50[Philbin CVPR’07]
Query Results on 5K (demo available for 100K)
![Page 49: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/49.jpg)
Example Applications
B. Leibe 51
Mobile tourist guide• Self-localization• Object/building recognition• Photo/video augmentation
[Quack, Leibe, Van Gool, CIVR’08]
![Page 50: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/50.jpg)
Application: Image Auto‐Annotation
K. Grauman, B. Leibe 52
Left: Wikipedia imageRight: closest match from Flickr
[Quack CIVR’08]
Moulin Rouge
Tour Montparnasse Colosseum
ViktualienmarktMaypole
Old Town Square (Prague)
![Page 51: Keypoint based Recognitiondhoiem.cs.illinois.edu/courses/vision_spring10/lectures... · 2010. 3. 4. · Keypoint‐based Recognition Computer Vision CS 543 / ECE 549 University of](https://reader035.fdocuments.us/reader035/viewer/2022081516/60f9703d578b120f2025064d/html5/thumbnails/51.jpg)
Things to remember
• Object instance recognition– Find keypoints, compute descriptors
– Match descriptors
– Vote for / fit affine parameters
– Return object if # inliers > T
• Keys to efficiency– Visual words
• Used for many applications
– Inverse document file• Used for web‐scale search