An Unsupervised Learning Approach to Content-Based Image Retrieval
description
Transcript of An Unsupervised Learning Approach to Content-Based Image Retrieval
![Page 1: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/1.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
1
An Unsupervised Learning Approach to Content-Based Image Retrieval
Yixin Chen & James Z. Wang
The Pennsylvania State University
![Page 2: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/2.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
2
Outline
Introduction
Cluster-based retrieval of images
Experiments
Conclusions and future work
![Page 3: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/3.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
3
Image Retrieval
The driving forces InternetStorage devicesComputing power
Two approachesText-based approachContent-based approach
![Page 4: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/4.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
4
Text-Based Approach
Input keywords descriptions
Text-BasedImage
RetrievalSystem
Elephants
ImageDatabase
![Page 5: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/5.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
5
Text-Based Approach
Index images using keywords(Google, Lycos, etc.)Easy to implementFast retrievalWeb image search (surrounding text)Manual annotation is not always availableA picture is worth a thousand wordsSurrounding text may not describe the
image
![Page 6: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/6.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
6
Content-Based Approach
Index images using low-level features
Content-based image retrieval (CBIR): search pictures as pictures
CBIRSystem
ImageDatabase
![Page 7: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/7.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
7
A Data-Flow Diagram
ImageDatabase
FeatureExtraction
ComputeSimilarityMeasure
Visualization
Histogram, colorlayout, sub-images,regions, etc.
Euclidean distance,intersection, shapecomparison, region matching, etc.
Linear ordering,Projection to 2-D, etc.
![Page 8: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/8.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
8
Open Problem
Nature of digital images: arrays of numbers Descriptions of images: high-level concepts
Sunset, mountain, dogs, ……
Semantic gap Discrepancy between low-level features and high-
level concepts High feature similarity may not always correspond
to semantic similarity
![Page 9: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/9.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
9
Outline
Introduction
Cluster-based retrieval of images
Experiments
Conclusions and future work
![Page 10: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/10.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
10
Motivation
A query image and its top 29 matches returned by a CBIR system
Horses (11 out of 29), flowers (7 out of 29), golf player (4 out of 29)
![Page 11: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/11.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
11
CLUE: CLUsters-based rEtrieval of images by unsupervised learning
HypothesisIn the “vicinity” of a query image, images tend to be semantically clustered
CLUE attempts to capture high-level semantic concepts by learning the way that images of the same semantics are similar
![Page 12: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/12.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
12
System Overview
A general diagram of a CBIR system using CLUE
ImageDatabase
FeatureExtraction
ComputeSimilarityMeasure
DisplayAnd
Feedback
SelectNeighboring
Images
ImageClustering
CLUE
![Page 13: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/13.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
13
Neighboring Images Selection
Nearest neighbors method Pick k nearest
neighbors of the query as seeds
Find r nearest neighbors for each seed
Take all distinct images as neighboring images
k=3, r=4
![Page 14: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/14.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
14
Weighted Graph Representation
Graph representation Vertices denote images Edges are formed
between vertices Nonnegative weight of
an edge indicates the similarity between two vertices
![Page 15: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/15.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
15
Clustering
Graph partitioning and cut
Normalized cut (Ncut) [Shi et al., IEEE Trans. PAMI 22(8)]
Recursive Ncut
![Page 16: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/16.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
16
Outline
Introduction
Cluster-based retrieval of images
Experiments
Conclusions and future work
![Page 17: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/17.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
17
An Experimental System
Similarity measureUFM [Chen et al. IEEE PAMI 24(9)]
DatabaseCOREL60,000
![Page 18: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/18.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
18
User Interface
![Page 19: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/19.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
19
Query Examples
Query Examples from 60,000-image COREL DatabaseBird, car, food, historical buildings, and soccer game
CLUE UFM
Bird, 6 out of 11 Bird, 3 out of 11
![Page 20: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/20.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
20
Query Examples
CLUE UFM
Car, 8 out of 11 Car, 4 out of 11
Food, 8 out of 11 Food, 4 out of 11
![Page 21: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/21.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
21
Query Examples
CLUE UFM
Historical buildings, 10 out of 11 Historical buildings, 8 out of 11
Soccer game, 10 out of 11 Soccer game, 4 out of 11
![Page 22: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/22.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
22
Clustering WWW Images
Google Image SearchKeywords: tiger, BeijingTop 200 returns4 largest clustersTop 18 images within each cluster
![Page 23: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/23.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
23
Clustering WWW Images
Tiger Cluster 1 (75 images)
Tiger Cluster 3 (32 images)
Tiger Cluster 2 (64 images)
Tiger Cluster 4 (24 images)
![Page 24: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/24.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
24
Clustering WWW Images
Beijing Cluster 1 (61 images) Beijing Cluster 2 (59 images)
Beijing Cluster 3 (43 images) Beijing Cluster 4 (31 images)
![Page 25: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/25.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
25
Retrieval Accuracy
![Page 26: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/26.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
26
Outline
Introduction
Cluster-based retrieval of images
Experiments
Conclusions and future work
![Page 27: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/27.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
27
Conclusions
Retrieving image clusters by unsupervised learning
Tested using 60,000 images from COREL and images from WWW
![Page 28: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/28.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
28
Future Work
Recursive Ncut
Representative image
Other graph theoretic clustering techniques
Nonlinear dimensionality reduction
![Page 29: An Unsupervised Learning Approach to Content-Based Image Retrieval](https://reader035.fdocuments.us/reader035/viewer/2022062217/568148cd550346895db5ea6e/html5/thumbnails/29.jpg)
IEEE Int'l Symposium on Signal Processing and its Applications
29
Thank You!