Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2...
Transcript of Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2...
![Page 1: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/1.jpg)
Semantic Localization of
Indoor PlacesLukas Kuster
![Page 2: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/2.jpg)
GPS for localization
2
Motivation
[7]
![Page 3: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/3.jpg)
Indoor navigation
3
Motivation
[8]
![Page 4: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/4.jpg)
Crowd sensing
4
Motivation
[9]
![Page 5: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/5.jpg)
Targeted Advertisement
5
Motivation
[10]
![Page 6: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/6.jpg)
Tourist guidance
6
Motivation
[12]
![Page 7: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/7.jpg)
GPS
WiFi
Images
Sound
Mobility
7
Semantic Localization
![Page 8: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/8.jpg)
GPS
WiFi
Images
Sound
Mobility
8
Semantic Localization
Works for unseen places
Outdoor and indoor
Rich in information
User’s point of view
No special hardware
![Page 9: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/9.jpg)
Motivation
Image Indoor Scene Recognition
Recognizing Indoor Scenes – 2009
Unsupervised Discovery of Mid-Level Discriminative Patches – 2012
Blocks that Shout – 2013
Semantic Localization in full Systems
Conclusions
9
Overview
![Page 10: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/10.jpg)
Goals:
Assign a scene category to an input image
10
Scene classification in computer vision
Scene
classifier
Library
Classroom
![Page 11: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/11.jpg)
Outdoor scenes
Global properties
Geometric
Indoor scenes
Local properties
Semantic meaningful objects
Arrangement of Objects
11
Challenges in scene recognition
![Page 12: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/12.jpg)
12
Scene Classification
2009 2012 2013
1 2 3
Recognizing Indoor
Scenes
Quattoni et al.
Unsupervised Discovery
of Mid-Level
Discriminative Patches
Singh et al.
Blocks that Shout:
Distinctive Parts for
Scene Classification
Juneja et al.
...
![Page 13: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/13.jpg)
Two different Image feature descriptors
Global information – Gist descriptors
Local informations – Sift descrptors
MIT Scene 67 dataset
13
Recognizing Indoor Scenes - Quattoni et al. (2009)
![Page 14: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/14.jpg)
14
Recognizing Indoor Scenes - Quattoni et al. (2009)
Random
Prototypes
![Page 15: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/15.jpg)
Manual and automatic segmentation into ROI
15
Recognizing Indoor Scenes - Quattoni et al. (2009)
Random
PrototypesSegmentation
![Page 16: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/16.jpg)
Manual and automatic segmentation into ROI
2x2 Histogram of Visual Words
16
Recognizing Indoor Scenes - Quattoni et al. (2009)
Random
PrototypesSegmentation ROI descriptors
![Page 17: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/17.jpg)
Manual and automatic segmentation into ROI
2x2 Histogram of Visual Words
Optimize parameters on test set
17
Recognizing Indoor Scenes - Quattoni et al. (2009)
Random
PrototypesSegmentation
Learning
p
k
xgxf
k
km
j kkGkjkj
xh1
)()(1exp)(
ROI descriptors
![Page 18: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/18.jpg)
Manual and automatic segmentation into ROI
2x2 Histogram of Visual Words
Optimize parameters on test set
18
Recognizing Indoor Scenes - Quattoni et al. (2009)
Random
PrototypesSegmentation
Learning
p
k
xgxf
k
km
j kkGkjkj
xh1
)()(1exp)(
ROI descriptors
Prototype weight
Local features Global feature
![Page 19: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/19.jpg)
MIT Scene 67 dataset
19
15620 labeled images
67 indoor scenes categories
![Page 20: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/20.jpg)
Category 1 (Predicted)
Category 2 (Predicted)
Category 3 (Predicted)
Category 4 (Predicted)
Category 5 (Predicted)
Category 1 (Actual)
90.12% 0.00% 9.88% 0.00% 0.00%
Category 2 (Actual)
0.00% 100.00% 0.00% 0.00% 0.00%
Category 3 (Actual)
0.00% 0.00% 92.66% 0.00% 7.34%
Category 4 (Actual)
37.20% 0.00% 10.34% 52.46% 0.00%
Category 5 (Actual)
0.00% 0.00% 12.69% 0.00% 87.31%
20
Test Setup – Quattoni et al. (2009)
67 * 80 images for training
67 * 20 images for testing
Performance metric: Standard average multiclass
prediction accuracy
![Page 21: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/21.jpg)
21
Results – Quattoni et al. (2009)
![Page 22: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/22.jpg)
Evaluation – Quattoni et al. (2009)
22
Segmentation Methods:
Segmentation: automatic
Annotation: manual
Features:
Only ROI
ROI + Gist
![Page 23: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/23.jpg)
Indoor Scene classification
Local and global features
Low accuracy (26%)
Manual annotation
23
Conclusion – Quattoni et al. (2009)
![Page 24: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/24.jpg)
24
Scene Classification
2009 2012 2013
1 2 3
Recognizing Indoor
Scenes
Quattoni et al.
Unsupervised Discovery
of Mid-Level
Discriminative Patches
Singh et al.
Blocks that Shout:
Distinctive Parts for
Scene Classification
Juneja et al.
...
![Page 25: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/25.jpg)
Unsupervised Discovery of Mid-Level
Discriminative Patches – Singh et al. (2012)
25
Mid-Level patches
Representative: frequent occurence in world
Discriminative: diffrent enough from rest of the world
![Page 26: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/26.jpg)
26
Singh et al. (2012)
Random
discovery set
![Page 27: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/27.jpg)
27
Singh et al. (2012)
Random
patches
Random
discovery set
![Page 28: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/28.jpg)
Cluster patches in HOG space
28
Singh et al. (2012)
Kmeans
clustering
Random
patches
Random
discovery set
![Page 29: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/29.jpg)
Cluster patches in HOG space
Train detector for each cluster
29
Singh et al. (2012)
SVM trainKmeans
clustering
Random
patches
Random
discovery set
![Page 30: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/30.jpg)
Cluster patches in HOG space
Train detector for each cluster
Use detector on validation set
Get top 5 matches for new cluster
Kill clusters that have less than 2 matches
30
Singh et al. (2012)
Detect new
patches
SVM trainKmeans
clustering
Random
patches
Random
discovery set
![Page 31: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/31.jpg)
Purity
Same visual concept
Sum of top r detection scores
Discriminativeness
Detected rarely in natural world
31
Ranking Detectors – Singh et al. (2012)
world)natural set (trainingin detections#
set gin trainin detections#
![Page 32: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/32.jpg)
Detect Patches on diffrent scales and
diffrent spatial pyramid levels
Train classifier with SVM
32
Image descriptor – Singh et al. (2012)
Object Bank Image representation – Li, L-J et al. (2010)
![Page 33: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/33.jpg)
Detect Patches on diffrent scales and
diffrent spatial pyramid levels
Train classifier with SVM
33
Image descriptor – Singh et al. (2012)
SVM
Object Bank Image representation – Li, L-J et al. (2010)
![Page 34: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/34.jpg)
34
Top Ranked patches – Singh et al. (2012)
MIT 67 Benchmark
![Page 35: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/35.jpg)
Evaluation – Singh et al. (2012)
35
Spatial Pyramid HOG 29,8
Spatial Pyramid SIFT (SP) 34,4
ROI-GIST (Quattoni et al.) 26,5
Object Bank 37,6
Patches 38,1
Accuracy:
![Page 36: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/36.jpg)
Evaluation – Singh et al. (2012)
36
Spatial Pyramid HOG 29,8
Spatial Pyramid SIFT (SP) 34,4
ROI-GIST (Quattoni et al.) 26,5
Object Bank 37,6
Patches 38,1
GIST+SP+DPM 43,1
Patches+GIST+SP+DPM 49,4
Combination approaches:
Accuracy:
![Page 37: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/37.jpg)
Indoor Scene classification
Local and global features
Low accuracy (26%)
Manual annotation
37
Conclusion
Quattoni et al. (2009) Singh et al. (2012)
Low supervision
Better accuracy
Low accuracy (49%)
Inefficient
![Page 38: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/38.jpg)
38
Scene Classification
2009 2012 2013
1 2 3
Recognizing Indoor
Scenes
Quattoni et al.
Unsupervised Discovery
of Mid-Level
Discriminative Patches
Singh et al.
Blocks that Shout:
Distinctive Parts for
Scene Classification
Juneja et al.
...
![Page 39: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/39.jpg)
More efficient
Distinctive patches
39
Blocks that Shout: Distinctive Parts for Scene
Classification – Juneja et al. (2013)
![Page 40: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/40.jpg)
40
Blocks that Shout – Juneja et al. (2013)
Initial training set
Seeding
![Page 41: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/41.jpg)
Automatic segmentation into superpixels
41
Blocks that Shout – Juneja et al. (2013)
SuperpixelsInitial training set
Seeding
![Page 42: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/42.jpg)
Automatic segmentation into superpixels
Seedblocks:
Intermediate sized superpixels
Image variation
42
Blocks that Shout – Juneja et al. (2013)
Initial training set Superpixels Seed Blocks
Seeding
![Page 43: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/43.jpg)
Seeding Expansion
43
Blocks that Shout – Juneja et al. (2013)
Seed Block HOG descriptor
8x8 HOG cells of 8x8 pixels
![Page 44: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/44.jpg)
44
Seed Block HOG descriptor
8x8 HOG cells of 8x8 pixels
Detect similiar blocks
Exemplar SVM
Blocks that Shout – Juneja et al. (2013)
Seeding Expansion
![Page 45: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/45.jpg)
45
Seed Block HOG descriptor
8x8 HOG cells of 8x8 pixels
Detect similiar blocks
5 iterations for final part detector
Exemplar SVMseed round1 round2 round3 round4 round5
Blocks that Shout – Juneja et al. (2013)
Seeding Expansion
![Page 46: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/46.jpg)
46
Select most distincitve part detectors
Entropy:
N
y
rypryprYH1
2 ),(log),(),(
Blocks that Shout – Juneja et al. (2013)
Seeding Expansion Selection
![Page 47: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/47.jpg)
Detect Patches on diffrent scales and
diffrent spatial pyramid levels
Train classifier with SVM
47
Image descriptor – Blocks that Shout (2013)
SVM
Object Bank Image representation – Li, L-J et al. (2010)
![Page 48: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/48.jpg)
48
Blocks that Shout – Juneja et al. (2013)
Results
![Page 49: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/49.jpg)
49
Blocks that Shout – Juneja et al. (2013)
Accuracy:
ROI-GIST (Quattoni et al.) 26,5
Object Bank 37,6
Patches (Singh et al.) 38,1
BoP 46,1
Evaluation
![Page 50: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/50.jpg)
50
Blocks that Shout – Juneja et al. (2013)
Patches+GIST+SP+DPM (Singh et al.) 49,4
IFV + BoP 63,1
Combination approaches:
Accuracy:
ROI-GIST (Quattoni et al.) 26,5
Object Bank 37,6
Patches (Singh et al.) 38,1
BoP 46,1
Evaluation
![Page 51: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/51.jpg)
Indoor Scene
classification
Local and global
features
Low accuracy (26%)
Manual annotation
51
Conclusion
Quattoni et al. (2009) Singh et al. (2012)
Low supervision
Better accuracy
Low accuracy (49%)
Inefficient
Juneja et al. (2013)
Low supervision
More efficient
Distinctive Parts
Even better
accuracy
Low accuracy (63%)
![Page 52: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/52.jpg)
Motivation
Image Indoor Scene Recognition
Recognizing Indoor Scenes – 2009
Unsupervised Discovery of Mid-Level Discriminative Patches – 2012
Blocks that Shout – 2013
Semantic Localization in full Systems
Conclusions
52
Overview
![Page 53: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/53.jpg)
53
Systems Overview
CrowdSense@Place
- 2012
- Crowd sensing
- Link visits with
place categories
- Share output with
location sensitive
applications
![Page 54: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/54.jpg)
54
Systems Overview
CrowdSense@Place
- 2012
- Crowd sensing
- Link visits with
place categories
- Share output with
location sensitive
applications
Place Naming System
- 2013
- Crowd sensing
Output:
- Functional name
(eg. Food place)
- Business name
(eg. Starbucks)
- Personal name
(eg. My home)
![Page 55: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/55.jpg)
CrowdSense@Place
- 2012
- Crowd sensing
- Link visits with
place categories
- Share output with
location sensitive
applications
55
Systems Overview
Place Naming System
- 2013
- Crowd sensing
Output:
- Functional name
(eg. Food place)
- Business name
(eg. Starbucks)
- Personal name
(eg. My home)
CheckInside
- 2014
- Location-based
Social Network
- Improved venues
list in Check-ins
![Page 56: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/56.jpg)
Mobility:
GPS
WiFi
Trajectory
56
Sensor Data
![Page 57: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/57.jpg)
Mobility:
GPS
WiFi
Trajectory
Visual Classifiers:
Text Recognition
Indoor Scene Classification
Object Recognition
57
Sensor Data
![Page 58: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/58.jpg)
Mobility:
GPS
WiFi
Trajectory
Visual Classifiers:
Text Recognition
Indoor Scene Classification
Object Recognition
Sound Classifiers:
Speech Recognition
Sound Classification58
Sensor Data
![Page 59: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/59.jpg)
CrowdSense@Place
- 1241 places
- 6 categories
- Accuracy:
~ 40% - 95%
- Overall : ~ 69%
59
Evaluation
![Page 60: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/60.jpg)
CrowdSense@Place
- 1241 places
- 6 categories
- Accuracy:
~ 40% - 95%
- Overall : ~ 69%
60
Evaluation
Place Naming System
- 3800 places
- 9 categories
- Functional name:
~ 20% - 90%
Business Name:
![Page 61: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/61.jpg)
CrowdSense@Place
- 1241 places
- 6 categories
- Accuracy:
~ 40% - 95%
- Overall : ~ 69%
61
Evaluation
Place Naming System
- 3800 places
- 9 categories
- Functional name:
~ 20% - 90%
CheckInside
- 711 stores
- 99% in top 5
Business Name:
![Page 62: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/62.jpg)
Good for functional
naming
62
Visual Scene Recognition Evaluation
CrowdSense@Place accuracy:
![Page 63: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/63.jpg)
Good for functional
naming
Intermediate
performance gain for
business naming
63
Visual Scene Recognition Evaluation
CrowdSense@Place accuracy:
Business Naming accuracy:
![Page 64: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/64.jpg)
Crowd sensing improves semantic localization
Relatively low accuracy
User interaction still needed
Visual scene recognition:
Fast progress
State of the art could improve the systems
64
Conclusion
![Page 65: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/65.jpg)
(1) Quattoni, A.; Torralba, A., "Recognizing indoor scenes," IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), 2009.
(2) Singh, S.; Gupta, A; Efros, A. A., “Unsupervised discovery of mid-level discriminative patches,”
European conference on Computer Vision (ECCV), 2012.
(3) Juneja, M.; Vedaldi, A.; Jawahar, C.V.; Zisserman, A., "Blocks That Shout: Distinctive Parts for
Scene Classification," IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2013.
(4) Chon, Y.; Lane, N. D.; Li, F.; Cha, H.; Zhao, F., “Automatically characterizing places with
opportunistic crowdsensing using smartphones,” ACM Conference on Ubiquitous Computing
(UbiComp), 2012.
(5) Chon, Y.; Kim, Y.; Cha, H., “Autonomous place naming system using opportunistic crowdsensing
and knowledge from crowdsourcing,” International conference on Information processing in sensor
networks (IPSN), 2013.
(6) Elhamshary, M; Youssef, M., “CheckInside: a fine-grained indoor location-based social network,”
ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), 2014.
65
References
![Page 66: Semantic Localization of Indoor Places - ETH Z...Manual and automatic segmentation into ROI 2x2 Histogram of Visual Words Optimize parameters on test set 18 Recognizing Indoor Scenes](https://reader035.fdocuments.us/reader035/viewer/2022081607/5ec714b7f17d3575a27ac033/html5/thumbnails/66.jpg)
(7) http://www.extremetech.com/extreme/126843-think-gps-is-cool-ips-will-blow-your-mind/2
(8) http://free-wifi-service.com/
(9) http://denimandsteel.com/talks/polyglot/
(10) http://www.elatewiki.org/images/Special.jpeg
(11) Li, L.J., Su, H., Xing, E., Fei-fei, L., “Object bank: A high-level image representation for scene
classication and semantic feature sparsication,” Conference on Neural Information Processing
Systems (NIPS), 2010.
(12) https://www.flip4new.de/blog/nokia-lumia-920-review-was-kann-das-windows-phone/
66
References