Image Retrieval: Current Image Retrieval: Current Techniques, Promising Techniques, Promising

Directions, and Open IssuesDirections, and Open Issues

Yong Rui, Thomas Huang and Shih-Fu Yong Rui, Thomas Huang and Shih-Fu ChangChang

Published in the Journal of Visual Published in the Journal of Visual Communication and Image Communication and Image

Representation.Representation.Presented by: Deepak Bote

Presentation OutlinePresentation Outline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

History of Image History of Image RetrievalRetrieval

Traditional text-based image search Traditional text-based image search enginesengines Manual annotation of imagesManual annotation of images Use text-based retrieval methodsUse text-based retrieval methods

E.g. E.g.

Water lilies

Flowers in a pond

<Its biological name>

Limitations of text-based Limitations of text-based approachapproach

Problem of image annotationProblem of image annotation Large volumes of databasesLarge volumes of databases Valid only for one language – with image Valid only for one language – with image

retrieval this limitation should not existretrieval this limitation should not exist Problem of human perceptionProblem of human perception

Subjectivity of human perceptionSubjectivity of human perception Too much responsibility on the end-userToo much responsibility on the end-user

Problem of deeper (abstract) needsProblem of deeper (abstract) needs Queries that cannot be described at all, but Queries that cannot be described at all, but

tap into the visual features of images.tap into the visual features of images.

OutlineOutline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

What is CBIR?What is CBIR?

Images have rich content.Images have rich content. This content can be extracted as This content can be extracted as

various content features:various content features: Mean color , Color Histogram etc…Mean color , Color Histogram etc…

Take the responsibility of forming Take the responsibility of forming the query away from the user.the query away from the user.

Each image will now be described by Each image will now be described by its own features.its own features.

CBIR – A sample search CBIR – A sample search queryquery

User wants to search for, say, many rose User wants to search for, say, many rose imagesimages He submits an existing rose picture as query.He submits an existing rose picture as query. He submits his own sketch of rose as query.He submits his own sketch of rose as query.

The system will extract image features for The system will extract image features for this query.this query.

It will compare these features with that of It will compare these features with that of other images in a database.other images in a database.

Relevant results will be displayed to the Relevant results will be displayed to the user.user.

Sample QuerySample Query

Sample CBIR Sample CBIR architecturearchitecture

OutlineOutline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

Feature ExtractionFeature Extraction

What are image features?What are image features? Primitive featuresPrimitive features

Mean color (RGB)Mean color (RGB) Color HistogramColor Histogram

Semantic featuresSemantic features Color Layout, texture etc…Color Layout, texture etc…

Domain specific featuresDomain specific features Face recognition, fingerprint matching Face recognition, fingerprint matching

etc…etc…

General features

Mean ColorMean Color

Pixel Color Information: R, G, BPixel Color Information: R, G, B Mean component (R,G or B)= Mean component (R,G or B)=

Sum of that component for all Sum of that component for all pixels pixels

Number of pixelsNumber of pixels

Pixel

HistogramHistogram

Frequency count of each individual Frequency count of each individual colorcolor

Most commonly used color feature Most commonly used color feature representationrepresentation

Image

Corresponding histogram

Color LayoutColor Layout

Need for Color LayoutNeed for Color Layout Global color features give too many false Global color features give too many false

positivespositives How it works:How it works:

Divide whole image into sub-blocksDivide whole image into sub-blocks Extract features from each sub-blockExtract features from each sub-block

Can we go one step further?Can we go one step further? Divide into regions based on color Divide into regions based on color

feature concentrationfeature concentration This process is called segmentation.This process is called segmentation.

Example: Color layoutExample: Color layout

** Image adapted from Smith and Chang : Single Color Extraction and Image Query

TextureTexture Texture – innate property of all surfacesTexture – innate property of all surfaces

Clouds, trees, bricks, hair etc…Clouds, trees, bricks, hair etc… Refers to visual patterns of homogeneityRefers to visual patterns of homogeneity Does not result from presence of single colorDoes not result from presence of single color Most accepted classification of textures based Most accepted classification of textures based

on psychology studies – Tamura on psychology studies – Tamura representationrepresentation

• Coarseness

• Contrast

• Directionality

• Linelikeness

• Regularity

• Roughness

Segmentation issuesSegmentation issues

Considered as a difficult problemConsidered as a difficult problem Not reliableNot reliable Segments regions, but not objectsSegments regions, but not objects Different requirements from Different requirements from

segmentation:segmentation: Shape extraction: High Accuracy Shape extraction: High Accuracy

requiredrequired Layout features: Coarse segmentation Layout features: Coarse segmentation

may be enoughmay be enough

Presentation OutlinePresentation Outline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

Problem of high Problem of high dimensionsdimensions

Mean Color = RGB = 3 dimensional Mean Color = RGB = 3 dimensional vectorvector

Color Histogram = 256 dimensionsColor Histogram = 256 dimensions Effective storage and speedy retrieval Effective storage and speedy retrieval

neededneeded Traditional data-structures not Traditional data-structures not

sufficientsufficient R-trees, SR-Trees etc…R-trees, SR-Trees etc…

2-dimensional space2-dimensional space

D1

D2

Point A

3-dimensional space3-dimensional space

Now, imagine…Now, imagine…

An N-dimensional An N-dimensional box!!box!!

We want to conduct We want to conduct a nearest neighbor a nearest neighbor query.query.

R-trees are designed R-trees are designed for speedy retrieval for speedy retrieval of results for such of results for such purposespurposes

Designed by Designed by Guttmann in 1984Guttmann in 1984

Presentation OutlinePresentation Outline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

IBM’s QBICIBM’s QBIC

QBIC – Query by Image ContentQBIC – Query by Image Content First commercial CBIR system.First commercial CBIR system. Model system – influenced many others.Model system – influenced many others. Uses color, texture, shape featuresUses color, texture, shape features Text-based search can also be Text-based search can also be

combined.combined. Uses R*-trees for indexingUses R*-trees for indexing

QBIC – Search by colorQBIC – Search by color

** Images courtesy : Yong Rao

QBIC – Search by shapeQBIC – Search by shape

** Images courtesy : Yong Rao

QBIC – Query by sketchQBIC – Query by sketch

** Images courtesy : Yong Rao

VirageVirage

Developed by Virage inc.Developed by Virage inc. Like QBIC, supports queries based Like QBIC, supports queries based

on color, layout, textureon color, layout, texture Supports arbitrary combinations of Supports arbitrary combinations of

these features with weights attached these features with weights attached to eachto each

This gives users more control over This gives users more control over the search processthe search process

VisualSEEkVisualSEEk

Research prototype – University of Research prototype – University of ColumbiaColumbia

Mainly different because it considers Mainly different because it considers spatial relationships between objects.spatial relationships between objects.

Global features like mean color, color Global features like mean color, color histogram can give many false positiveshistogram can give many false positives

Matching spatial relationships between Matching spatial relationships between objects and visual features together objects and visual features together result in a powerful search.result in a powerful search.

ISearch – my own systemISearch – my own system

ISearch – my own systemISearch – my own system

ISearch – my own systemISearch – my own system

Feature selection in Feature selection in ISearchISearch

Database Admin facility in Database Admin facility in ISearchISearch

Presentation OutlinePresentation Outline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

Open issuesOpen issues

Gap between low level features and Gap between low level features and high-level conceptshigh-level concepts

Human in the loop – interactive Human in the loop – interactive systemssystems

Retrieval speed – most research Retrieval speed – most research prototypes can handle only a few prototypes can handle only a few thousand images.thousand images.

A reliable test-bed and measurement A reliable test-bed and measurement criterion, please!criterion, please!

Presentation OutlinePresentation Outline

History of image retrieval – Issues facedHistory of image retrieval – Issues faced Solution – Content-based image retrievalSolution – Content-based image retrieval Feature extractionFeature extraction Multidimensional indexingMultidimensional indexing Current SystemsCurrent Systems Open issuesOpen issues ConclusionConclusion

ConclusionConclusion

Satisfactory progress, but still…Satisfactory progress, but still…

A long way to go…!!A long way to go…!!

AcknowledgementsAcknowledgements

Dr. Padma MundurDr. Padma Mundur Mr. Yong RaoMr. Yong Rao Mr. Sumit Jain, Software Engineer, Mr. Sumit Jain, Software Engineer,

KPIT Cummins, IndiaKPIT Cummins, India Mr. Ajay Joglekar, Software Mr. Ajay Joglekar, Software

Engineer, Veritas India.Engineer, Veritas India.

ReferencesReferences

Y. Rui, T. S. Huang, and S.-F. Chang, Y. Rui, T. S. Huang, and S.-F. Chang, “Image retrieval: Current techniques, “Image retrieval: Current techniques, promising directions, and open issues”promising directions, and open issues”

S. Jain, A. Joglekar, and D. Bote, S. Jain, A. Joglekar, and D. Bote, ISearch: A Content-based Image ISearch: A Content-based Image Retrieval (CBIR) Engine, as Retrieval (CBIR) Engine, as Bachelor Bachelor of Computer Engineering final year of Computer Engineering final year thesis, Pune University, thesis, Pune University, 20022002

THANK YOU!!!THANK YOU!!!

Questions?Questions?