Convolutional Features for Instance Search
-
Upload
universitat-politecnica-de-catalunya -
Category
Technology
-
view
6.880 -
download
0
Transcript of Convolutional Features for Instance Search
![Page 1: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/1.jpg)
Convolutional Features for Instance Search
Amaia Salvador
03/05/2016
![Page 2: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/2.jpg)
2
Related Publications
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance SearchAccepted at ICMR 2016
A. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance SearchAccepted at DeepVision CVPRW 2016
![Page 3: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/3.jpg)
Part IE. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance Search
![Page 4: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/4.jpg)
Visual Image Retrieval
4Image Database
Visual Query
“A dog”
Expected outcome:
![Page 5: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/5.jpg)
Visual Instance Retrieval
5Image Database
Visual Query
“This dog”
Expected outcome:
![Page 6: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/6.jpg)
Visual Instance Retrieval
6
Image RepresentationsQuery image
Image Database
Image Matching Ranking List
Similarity score Image
...
0.98
0.97
0.10
0.01
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(e.g. cosine similarity)
...
![Page 7: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/7.jpg)
7
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)...
INVERTED FILE
word Image ID1 1, 12, 2 1, 30, 1023 10, 124 2,3 6 10
...
Local hand-crafted features(e.g. SIFT)
Bag of Visual WordsN-Dimensional
feature space
Image Representations
High-dimensionalHighly sparse
![Page 8: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/8.jpg)
8
Image Representations
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097-1105).
Convolutional Neural Networks
![Page 9: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/9.jpg)
9
Image Representations
Babenko, A., Slesarev, A., Chigorin, A., & Lempitsky, V. (2014). Neural codes for image retrieval. In ECCV 2014Razavian, A., Azizpour, H., Sullivan, J., & Carlsson, S. (2014). CNN features off-the-shelf: an astounding baseline for recognition. In DeepVision CVPRW 2014
Convolutional Neural Networks FC layers as global feature representation
![Page 10: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/10.jpg)
10
Image Representations
Babenko, A., & Lempitsky, V. (2015). Aggregating local deep features for image retrieval. ICCV 2015Tolias, G., Sicre, R., & Jégou, H. (2015). Particular object retrieval with integral max-pooling of CNN activations. ICLR 2015Kalantidis, Y., Mellina, C., & Osindero, S. (2015). Cross-dimensional Weighting for Aggregated Deep Convolutional Features. arXiv preprint arXiv:1512.04065.
Convolutional Neural Networks
sum/max pooled conv features as global representation
![Page 11: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/11.jpg)
11
Image Representations
Ng, J., Yang, F., & Davis, L. (2015). Exploiting local features from deep networks for image retrieval. In DeepVision CVPRW 2015
Convolutional Neural Networks
conv features encoded with VLAD as global representation
![Page 12: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/12.jpg)
12
MotivationDataset Complexity
TRECVID Instance Search464 hours of video content
![Page 13: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/13.jpg)
13
Motivation: Image Representations
High-dimensional & Sparse
Bag of Visual Words
Compact & Dense(e.g. sum/max pooling conv feats, FC feats)
Capacity?
High-dimensional & Dense(e.g. VLAD encoding)
Scalability?
![Page 14: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/14.jpg)
Methodology
![Page 15: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/15.jpg)
15
Bag of Words Framework
![Page 16: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/16.jpg)
16
Bag of Words Framework
(336x256)Resolution
conv5_1 from VGG16[1]
(42x32)
[1]Simonyan K., Zisserman A., Very Deep Convolutional Networks for Large-Scale Image Recognition, arXiv 2014
25K centroids 25K-D vector
![Page 17: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/17.jpg)
17
Instance RetrievalQuery Representation
... ... ...
... ... ...
Global Search(GS)
Local Search(LS)
![Page 18: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/18.jpg)
18
Spatial RerankingImage RepresentationsQuery image
Image Database
Image Matching Ranking List
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(cosine similarity)
...
Top M imagesare locally analyzed
and reranked(M = 100)
![Page 19: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/19.jpg)
19
Spatial Reranking
All window combinations with:
Query Image Target image in top M ranking
...
...
![Page 20: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/20.jpg)
20
Query ExpansionImage RepresentationsQuery image
Image Database
Image Matching Ranking List
v = (v1, …, vn)
v1 = (v11, …, v1n)
vk = (vk1, …, vkn)
...
Similarity Metric
(cosine similarity)
...
Top N imagesare added to the query for a new
search(N = 5)
![Page 21: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/21.jpg)
Experiments
![Page 22: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/22.jpg)
22
DatasetsParis Buildings 6k Oxford Buildings 5k
TRECVID Instance Search 2013(subset of 23k frames)
Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Object retrieval with large vocabularies and fast spatial matching, CVPR 2007Philbin, J. , Chum, O. , Isard, M. , Sivic, J. and Zisserman, A. Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases. CVPR 2008Smeaton, A. F., Over, P., & Kraaij, W. Evaluation campaigns and TRECVid. ACM MM Multimedia information retrieval Workshop 2006
![Page 23: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/23.jpg)
23
Results I: SoA Comparison
![Page 24: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/24.jpg)
24
Results II: TRECVid INS
![Page 25: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/25.jpg)
25
Qualitative Results
![Page 26: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/26.jpg)
26
Conclusion
BoW encoding of convolutional features
• High-dimensional sparse representation suitable for fast retrieval
• Competitive results in two image retrieval benchmarks
• Well suited and more robust for scenarios where only small number of features are
in the target images are relevant to the query (INS).
![Page 27: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/27.jpg)
Part IIA. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance Search
![Page 28: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/28.jpg)
28
Reminder: Spatial Reranking
Query Image Target image in top M ranking
...
...
![Page 29: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/29.jpg)
29
Reminder: Spatial Reranking
Koen E. A. van de Sande, Jasper R. R. Uijlings, Theo Gevers, Arnold W. M. Smeulders. Segmentation as Selective Search for Object Recognition, ICCV 2011
Object Proposals
![Page 30: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/30.jpg)
30
Image & Region Representations
“dog”
CNN Architectures
plant, table, dog
CNN
CNN
Image Classification
Object Detection
![Page 31: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/31.jpg)
31
Image & Region RepresentationsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015
![Page 32: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/32.jpg)
32
Image & Region RepresentationsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Ren, S., He, K., Girshick, R., & Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. NIPS 2015
Image representation
Region Representation
![Page 33: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/33.jpg)
33
Image & Region Representations
Image representation Region Representation(for reranking)
RoI Pooling
Conv5_3 RoI Pooling
sum-pooling max-pooling
DD
![Page 34: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/34.jpg)
34
Fine tuning for query objectsFaster R-CNN
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Train object detector for query instances using query images as training data
![Page 35: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/35.jpg)
35
Fine tuning for query objectsFT #1: Train FC layers only
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
![Page 36: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/36.jpg)
36
Fine tuning for query objectsFT #2: Train all weights after conv2
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
![Page 37: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/37.jpg)
37
Spatial Reranking Strategies
Class-agnostic Spatial Reranking (CA-SR)
Query Image Database Image
FC6
Class probabilitiesFC7
FC8...
Class-specific Spatial Reranking (CS-SR)
![Page 38: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/38.jpg)
38
Results
![Page 39: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/39.jpg)
39Query image Top N retrieved images
![Page 40: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/40.jpg)
40
ConclusionFaster R-CNN for Instance Search
• Suitable to obtain image and region features in a single forward pass
• Fine tuning as an effective solution to boost retrieval performance (subject to
application time constraints)
Con
v la
yers
Region Proposal Network
FC6
Class probabilitiesFC7
FC8
RPN Proposals
RoI Pooling
Conv5_3
RPN Proposals
Image representation
Region Representation
![Page 41: Convolutional Features for Instance Search](https://reader034.fdocuments.us/reader034/viewer/2022050613/587354c61a28ab56378b711b/html5/thumbnails/41.jpg)
41
Thank you for your attention !
E. Mohedano, A. Salvador, K. McGuinness, F. Marques, N. E. O'Connor and X. Giro, Bags of Local Convolutional Features for Scalable Instance SearchAccepted at ICMR 2016
A. Salvador, X. Giro, F. Marques, S. Satoh,Faster R-CNN Features for Instance SearchAccepted at DeepVision CVPRW 2016