Semi-supervised concept detection by learning the structure of similarity graphs
-
Upload
symeon-papadopoulos -
Category
Technology
-
view
7.900 -
download
0
description
Transcript of Semi-supervised concept detection by learning the structure of similarity graphs
![Page 1: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/1.jpg)
19th International Conference on Multimedia ModelingHuangshan, China, Jan 7-9, 2012
Semi-supervised concept detection by learning the structure of similarity graphsSymeon Papadopoulos1, Christos Sagonas1, Ioannis Kompatsiaris1, Athena Vakali2
1 Centre for Research and Technology Hellas, Information Technologies Institute 2 Aristotle University of Thessaloniki, Informatics Department
![Page 2: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/2.jpg)
mklab.iti.gr #2
chocolatecakechocolateganachebuttercreamshamsd
IMAGE TAGS CONCEPTS
N/A
naturelandscapewaterreflectionmirrorflickreliteabigfave
food
femaleindoorpeopleportrait
cloudslakeskywater
SOURCE: MIR-Flickr
![Page 3: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/3.jpg)
mklab.iti.gr #3
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 4: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/4.jpg)
mklab.iti.gr #4
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 5: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/5.jpg)
mklab.iti.gr #5
Concept detection
ML perspective• Given an image, produce a set of relevant concepts
IR perspective• Given an image collection and a concept of interest,
rank all images in order of relevance.
![Page 6: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/6.jpg)
mklab.iti.gr
• Transductive learning setting
#6
Predict concepts associated with items of by processing together and .
Semi-supervised learning
target concepts
annotated set
D-dimensional feature vector from image i
concept indicator vector (labels) for image i
set of unknown items
![Page 7: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/7.jpg)
mklab.iti.gr #7
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 8: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/8.jpg)
mklab.iti.gr #8
Related work
• Neighborhood similarity (Wang et al., 2009)– Uses image similarity graphs in combination with graph-based SSL
(Zhu, 2005; Zhou et al., 2004) – Not incremental
• Sparse similarity graph by convex optim. (Tang et al., 2009)– Applicable to online settings - Computationally intensive training step
• Hashing-based graph construction (Chen et al., 2010)– Uses KL divergence multi-label propagation, but relies on iterative
computational scheme – Difficult to apply in incremental settings
• Social dimensions (Tang & Liu, 2011)– Uses LEs for networked classification problems (i.e. when network
between nodes is explicit) – Not incremental, not applied to multimedia
![Page 9: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/9.jpg)
mklab.iti.gr #9
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 10: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/10.jpg)
mklab.iti.gr #10
Graph Structure Features (GSF)
![Page 11: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/11.jpg)
mklab.iti.gr #11
Graph construction
Construction options• full weighted graph
• kNN graph (connect k most similar images)
• εNN graph (connect images < similarity threshold)
image similarity graph
set of nodes-images
cardinality of node set
![Page 12: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/12.jpg)
mklab.iti.gr #12
Eigenvector/value computation
Normalized graph Laplacian
degree matrix (diagonal)
adjacency matrix
(typical form of graph Laplacian: )
non-zero eigenvalues
graph structure features*
by solving
*aka Laplacian Eigenmaps
![Page 13: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/13.jpg)
mklab.iti.gr #13
Graph structure feature learning
• Each media item is represented by a vector
• At this point, any supervised learning method could be used.[note that the whole framework is still SSL since unlabeled items are
used during graph construction]
• SVM is selected– good performance in several problems– good implementations available (LibSVM, LIBLINEAR)– real-valued output (IR perspective rank images by concept)
![Page 14: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/14.jpg)
mklab.iti.gr #14
Intuition
2nd eigenvector of graph Laplacian
0.2748
0.24150.3077
0.24150.3144
-0.0893
-0.4552
-0.4552
-0.4663
coast, person
coast, person
coast, person
coast, person
coastcoast
coast
coastcoast
![Page 15: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/15.jpg)
mklab.iti.gr #15
Incremental learning setting (1)
• Transductive learning setting often impractical. For each new set of unlabeled items:
1. recompute image similarity matrix2. recompute graph structure features (LEs)3. use SVM to obtain prediction scores
• Step 2 is computationally expensive.• Devise two incremental schemes:
– Linear Projection (LP) :
– Submanifold Analysis (SA) [cf. next slide]
set of k most similar images
![Page 16: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/16.jpg)
mklab.iti.gr #16
Incremental learning setting (2)
• Submanifold Analysis [Jia et al., 2009]– Construct (k+1)x(k+1) similarity matrix WS between new
item and k most images from the annotated set– Construct sub-diagonal and sub-Laplacian matrices
– Compute eigenvalues and d eigenvectors corresponding to non-zero eigenvalues [computation is lightweight since k << n]
– Minimize reconstruction error:
– Reconstruct approximate eigenvectors:
![Page 17: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/17.jpg)
mklab.iti.gr #17
Fusion of multiple features
Feature fusion (F-FEAT)
Similarity graph fusion (F-SIM)
Graph struct. feature fusion (F-GSF)
Result fusion (F-RES)
![Page 18: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/18.jpg)
mklab.iti.gr #18
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 19: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/19.jpg)
mklab.iti.gr #19
Synthetic data - experiments
• Use of four 2D distributions with limited number of samples (thousands) to test many settings
• Performance aspects– Parameters of approach: number of features (CD), graph
construction technique (kNN, εNN) and parameters (k, ε)– Learning setting (training size, data noise, nr. of classes)– Inductive learning (LP vs SA)– Fusion method
TWO MOONS LINES CIRCLES GAUSSIANS
![Page 20: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/20.jpg)
mklab.iti.gr #20
Role of number of GSF (CD)TWO MOONS LINES
CIRCLES GAUSSIANS
noise levels
higher CD better mAPhigher noise higher CD
![Page 21: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/21.jpg)
mklab.iti.gr #21
Role of graph construction technique
kNN εNN
kNN better and less sensitive than εΝΝ
![Page 22: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/22.jpg)
mklab.iti.gr #22
Role of noise (σ)TWO MOONS LINES
CIRCLES GAUSSIANScompeting methods
In most cases GSF equal or better than the expensive SVM-RBF.
![Page 23: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/23.jpg)
mklab.iti.gr #23
Role of training samples (α%)TWO MOONS LINES
CIRCLES GAUSSIANS
In most cases few training samples (2-5%) are sufficient for high accuracy.
![Page 24: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/24.jpg)
mklab.iti.gr #24
Number of classes (K)
LINES CIRCLES
Sufficiently good accuracy wrt. number of classes(much better than linear SVM, a bit worse than SVM-RBF).
![Page 25: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/25.jpg)
mklab.iti.gr #25
Scalability wrt. number of features
Constant cost wrt. dimensionality
Linearly increasing cost wrt. dimensionality
![Page 26: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/26.jpg)
mklab.iti.gr #26
Comparison between fusion methods
LINES CIRCLES
Even when one feature goes bad, result and GSF fusion still do better than the best.
![Page 27: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/27.jpg)
mklab.iti.gr #27
Incremental schemesTWO MOONS LINES
CIRCLES GAUSSIANS
SA much better and less sensitive than LP.
![Page 28: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/28.jpg)
mklab.iti.gr #28
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 29: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/29.jpg)
mklab.iti.gr #29
Experimental setting
• MIR-Flickr– 25,000 images + tags– 38 concepts (24 + 14 with two interpretations [strict/rel])
• Benchmark methods– Semantic Spaces (SESPA) [Hare & Lewis, 2010]– Multiple Kernel Learning (MKL) [Guillaumin et al., 2010]
![Page 30: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/30.jpg)
mklab.iti.gr #30
GSF vs SESPA
GSF-F1, F2, F3: Single feature GSF
GSF-D1, D2: Result fusion using LIBLINEAR (1) and RBF (2)
GSF-C: Graph structure feature fusion
![Page 31: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/31.jpg)
mklab.iti.gr #31
GSF vs MKL
VISUAL
TAG
Possible thanks to scalable behavior wrt.
number of features.
GSF better in: baby, bird, car, dog, river, sea.
MKL better in: baby, bird, river, sea.
![Page 32: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/32.jpg)
mklab.iti.gr #32
Example results
![Page 33: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/33.jpg)
mklab.iti.gr #33
Evaluation: adding unlabeled samples (1)
GIST
~6% relative increase in mAP
![Page 34: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/34.jpg)
mklab.iti.gr #34
Evaluation: adding unlabeled samples (2)
DenseSiftV3H1
~12% relative increase in mAP
![Page 35: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/35.jpg)
mklab.iti.gr #35
Evaluation: adding unlabeled samples (3)
TagRaw50
~4% relative increase in mAP
![Page 36: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/36.jpg)
mklab.iti.gr #36
Overview
• Problem formulation• Related work• Graph Structure Features Approach• Evaluation
– Synthetic datasets– MIR-Flickr
• Conclusions
![Page 37: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/37.jpg)
mklab.iti.gr #37
Conclusions
• Concept detection approach based on the structure of image similarity graphs– Transductive learning setting– Two variants for online learning
• Thorough experimental analysis– Behavior under a variety of settings/parameters– Equivalent or better behavior compared to SoA approaches
• Fast: – SA with k=5 takes 38.4msec per image (not incl. feature extraction)– Future work: further analysis of computational characteristics +
application to larger scale datasets (NUS-Wide, ImageNet)
![Page 39: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/39.jpg)
mklab.iti.gr #39
References (1)
• Graph-based semi-supervised learningZhu, X.: Semi-supervised learning with graphs. PhD Thesis, Carnegie
Mellon University, 0-542-19059-1 (2005)Zhou, D., Bousquet, O., Navin Lal, T., Weston, J. Schoelkopf, B.: Learning
with Local and Global Consistency. Advances in NIPS 16, MIT Press (2004) 321-328
• Related approachesWang, M., Hua, X.-S. Tang, J., Hong, R.: Beyond distance measurement:
constructing neighborhood similarity for video annotation. TMM 11 (3) (2009), 465-476
Tang, J. et al.: Inferring semantic concepts from community contributed images and noisy tags. ACM Multimedia (2009) 223-232
Chen, X. et al.: Efficient large scale image annotation by probabilistic collaborative multi-label propagation. ACM Multimedia (2010), 35-44
Tang, L., Liu, H.: Leveraging social media networks for classification. Data Mining and Knowledge Discovery 23 (3) (2011), 447-478
![Page 40: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/40.jpg)
mklab.iti.gr #40
References (2)
• Relational classificationMacskassy, S.A., Provost, F.: Classification in Networked Data: A Toolkit
and a Univariate Case Study. Journal of Machine Learning Research 8, (2007), 935-983
• Laplacian EigenmapsMikhail, B., Partha, N.: Laplacian Eigenmaps for dimensionality reduction
and data representation. Neural Computing 15 (6), MIT Press (2003) 1373-1396
Jia, P., Yin, J., Huang, X., Hu, D.: Incremental Laplacian eigenmaps by preserving adjacent information between data points. PR Letters 30 (16) (2009), 1457–1463
![Page 41: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/41.jpg)
mklab.iti.gr #41
References (3)
• ToolsLeyffer, S., Mahajan, A.: Nonlinear Constrained Optimization: Methods and
Software. Preprint ANL/MCS-P1729-0310 (2010)Fan, R., Chang, K., Hsieh, C., Wang, X., Lin, C.: LIBLINEAR: A Library for Large Linear
Classification. Journal of ML Research 9 (2008), 1871-1874Chang, C.-C., Lin, C.-J.: LIBSVM: A library for support vector machines. ACM
Transactions on Intelligent Systems and Technology 2 (3) (2011), 27:1–27:27
• DatasetHuiskes, M.J., Michael S. Lew, M.S.: The MIR Flickr Retrieval Evaluation.
Proceedings of ACM Intern. Conf. on Multimedia Information Retrieval (2008)
• Competing methodsHare, J.S., Lewis, P.H.: Automatically annotating the MIR Flickr dataset. ACM ICMR
(2010), 547-556Guillaumin, M., Verbeek, J., Schmid, C.: Multimodal semi supervised learning for
image classification. Proceedings of IEEE CVPR Conference (2010), 902-909
![Page 42: Semi-supervised concept detection by learning the structure of similarity graphs](https://reader034.fdocuments.us/reader034/viewer/2022052619/5550b401b4c90504628b4a6b/html5/thumbnails/42.jpg)
mklab.iti.gr #42