Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot...
Transcript of Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot...
![Page 1: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/1.jpg)
Towards Hierarchical Place Recognition for
Long-Term Autonomy
Kirk MacTavish and Timothy D. Barfoot
ICRA Workshop on Visual Place Recognition in Changing Environments
June 2014
![Page 2: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/2.jpg)
•Why use Place Recognition?
•How can we deal with variable lighting?
•What about Long-term Autonomy?
Motivation | Overview
2
![Page 3: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/3.jpg)
Motivation | GPS Denied Environments
3
Mines
Planetary ExplorationUrban Canyons
![Page 4: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/4.jpg)
Motivation | What is place recognition?
Relative localization is sufficient for many
tasks.
Have we beenhere before?
Cummins, M. and Newman, P., “FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance,” The International Journal of Robotics Research, 27(6):647–665, 2008.
Perceptual Aliasing Scene & Perspective Change
4
![Page 5: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/5.jpg)
•Why use Place Recognition?
•How can we deal with variable lighting?
•What about Long-term Autonomy?
Motivation | Overview
5
![Page 6: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/6.jpg)
11 am 6 pm
Motivation | Images Change over Time
6
![Page 7: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/7.jpg)
• We can get some degree of lighting invariance from color-space manipulation.
• Dealing with Shadows: Capturing Intrinsic Scene Appearance for
Image-based Outdoor Localisation (Corke et al. 2013)
• Shady Dealings: Robust, Long- Term Visual Localisation using
Illumination Invariance (McManus et al. 2014)
• Lidar intensity images are unaffected by lighting conditions over the full day-night period.
Motivation | Lighting Invariance
7
![Page 8: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/8.jpg)
Motivation | 24 Hours of Images
Place recognition compares places after significant time has
passed. This makes lighting invariance extremely important.
8
![Page 9: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/9.jpg)
Motivation | Discretization
Camera
Lidar
9
The discretization is less obvious
![Page 10: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/10.jpg)
•Why use Place Recognition?
•How can we deal with variable lighting?
•What about Long-Term Autonomy?
Motivation | Overview
10
![Page 11: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/11.jpg)
It would be interesting to design the algorithm for operation over a 10 year period.
• Algorithms should be able to run in a constant computational budget for 10 years.
• Algorithms should be able to learn and adapt to the changing environment over the 10 year period.
Motivation | Long-Term Autonomy
11
![Page 12: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/12.jpg)
• “Highly Scalable Appearance-Only SLAM – FAB-MAP 2.0” (Cummins and Newman, 2009)
• Linear complexity but very fast
• Are We There Yet? Challenging SeqSLAM on a 3000 km Journey Across All Four Seasons (Sunderhauf, Neubert and Protzel, 2013)
• Performs well under seasonal change, but still linear complexity and is sensitive to
camera alignment
• “Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation” (Labbé and Michaud, 2013)
• Constant-time, but does not consider the whole map
Motivation | Long-Term Autonomy
12
![Page 13: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/13.jpg)
Methodology | Roadmap
Place Recognition Lighting Change
Computational
Complexity
Hierarchy
Place
Discretization
LIDAR
Two Problems
One Solution
13
![Page 14: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/14.jpg)
•Place Hierarchy
•FAB-MAP with groups
Methodology | Overview
14
![Page 15: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/15.jpg)
Methodology | Computational Complexity
15
![Page 16: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/16.jpg)
•Place Hierarchy
•FAB-MAP with groups
Methodology | Overview
16
![Page 17: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/17.jpg)
Methodology | FAB-MAP with groups
City Centre Dataset Groups of 128 images
• Used the OpenFABMAPimplementation
• Adapted Bag-of-Words (BoW) descriptor to use features from groups of images, rather than single images
• Have not yet done hierarchical expansion
17
![Page 18: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/18.jpg)
Methodology | FAB-MAP with groups
City Centre Dataset
Groups of 128 images
18
Bag-of-Words is a histogram of discretized
visual features
![Page 19: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/19.jpg)
Methodology | FAB-MAP with groups
City Centre Dataset
Groups of 128 images
Single Image
Larger Group
BoW descriptors became less sparse with larger groups. This invalidates training.
19
![Page 20: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/20.jpg)
Methodology | FAB-MAP with groups
The number of times a word must be seen in a group to be counted
as present
20
Single Image
Larger Group
BoW descriptors became less sparse with larger groups. This invalidates training.
![Page 21: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/21.jpg)
Methodology | FAB-MAP with groups
Single Image
Larger Group
Group BoW descriptors are now as sparse as single images.
Training is useful again.
The number of times a word must be seen in a group to be counted
as present
21
![Page 22: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/22.jpg)
•Datasets
•Results on camera images
•Results on LIDAR intensity
Results | Overview
22
![Page 23: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/23.jpg)
Results | Datasets
City Centre New College
Oxford Mobile Robotics Group
23
![Page 24: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/24.jpg)
Results | Datasets
KITTI Vision Benchmark SuiteOdometry Dataset
24
![Page 25: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/25.jpg)
Results | Datasets
ASRL Sudbury LIDAR Dataset
25
Available: http://asrl.utias.utoronto.ca/datasets/abl-sudbury/
Anderson A, McManus C, Dong H, Beerepoot E, and Barfoot T D.
“The Gravel Pit Lidar-Intensity Imagery Dataset”.
University of Toronto Technical Report ASRL-2012-ABL001
![Page 26: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/26.jpg)
•Datasets
•Results on camera images
•Results on LIDAR intensity
Results | Overview
26
![Page 27: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/27.jpg)
Results | Camera | Confusion Matrices
KITTI-06 New College City Centre
27
La
rger
Gro
up
s o
f Im
ag
es
![Page 28: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/28.jpg)
Results | Camera | Performance
28
Single-image FABMAP
Single-image FABMAP
![Page 29: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/29.jpg)
Results | Camera | City Centre with groups
29
![Page 30: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/30.jpg)
•Datasets
•Results on camera images
•Results on LIDAR intensity
Results | Overview
30
![Page 31: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/31.jpg)
Results | LIDAR | Sudbury Dataset
31
Single-image FABMAP
![Page 32: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/32.jpg)
• Develop the hierarchical inferencing algorithm.
• Develop a deeper high-level descriptor that can describe large places better than Bag-of-Words.
• Adapt the algorithm to use an unstructured LIDAR descriptor for continuous scans.
Future Work
32
![Page 33: Towards Hierarchical Place Recognition for Long-Term …...Kirk MacTavish and Timothy D. Barfoot ICRA Workshop on Visual Place Recognition in Changing Environments June 2014 •Why](https://reader033.fdocuments.us/reader033/viewer/2022053118/609e789653b69012450ce823/html5/thumbnails/33.jpg)
http://asrl.utias.utoronto.ca
Email:
Web:
33
Towards Hierarchical Place Recognition for Long-Term Autonomy