Semantically Relevant Visual Dictionary

Semantically Relevant Visual Dictionary

Ashish Gupta (CVSSP)

University of Surrey

[email protected]

July 10,2012

Ashish Gupta (CVSSP) Semantically Relevant Visual Dictionary

Contents

Introduction: Visual Category Recognition

Current practice: Visual Dictionary

Problem: inter-mixed feature vectors

Approach: Over-partition + Co-cluster image-word matrix

Solution: Group estimated categorically related partitions

Experiments:

Summary


Visual Category Recognition

Definition

Detect presence of an instance of avisual category in an image.


Challenges

Several variations in visual category appearance render categoryrecognition very difficult.


Visual Dictionary

Visual Word

Representative feature vector(generally centroid) of eachcluster.

Image Histogram

Histogram of assignments ofimage feature vectors to visualwords.


Problems with Visual Dictionary

Inter-mixed

Categorically dissimilar feature vectors inter-mixed in feature space.

Semantic scatter

Feature vectors pertaining to same category part scattered infeature space.


Inter-mixed Feature Vectors

Categorically equivalentvectors mapped to naturallyoccurring clusters

Easily partitioned to yielddiscriminative dictionaryelements



Categorically dissimilar vectorsinter-mixed

Partitioning yieldsnon-discriminative dictionary



Over-partition feature space into tiny clusters.

Build a dictionary using these tiny clusters.


Semantic Scatter

Small variations in instances of object part causes associateddescriptors to get scattered in feature space.

Combine visual words which are related and create a visualtopic.


Hypothesis

Semantically related words can be discovered by analysingimage-word distribution.


Visual Topic Dictionary ← Visual Word Dictionary


Co-Clustering

Formulate the image-word matrix as a joint probability distribution.

CX : {x1, x2, . . . , xm} → {x1, x2, . . . , xk}CY : {y1, y2, . . . , yn} → {y1, y2, . . . , yl}the tuple (CX ,CY ) is referred to as co-clustering.

‘re-order’ rows and columns of the matrix, which gives rise toblocks, referred to as co-clusters.


Co-clustering contd.

Optimal co-clustering minimizes loss in mutual informationI (X ;Y )− I (X ; Y ), given number of row (k) and column (l)clusters.

For a (CX ,CY ), loss in mutual information can be expressed byKL-divergence between p(X ,Y ) and an approximation q(X ,Y ).I (X ;Y )− I (X ; Y ) = DKL(p(X ,Y ) ‖ q(X ,Y ))


Conceptual view

Image histogram feature vectors in high-dimensional visual wordsspace are projected to lower dimensional visual topic space.

The distance between feature vectors from the same category isreduced.


Experiment

Feature descriptor

SIFT : Affine co-variant local image patch descriptor.

Data sets

Scene-15; Pascal VOC 2006; VOC 2007; VOC 2010.

Classifier

k-NN : Verify if mutual distance between categorically equivalentfeature vectors is reduced.

Performance metric

F1-score: harmonic mean of precision and recall. Popularly used inclassification and retrieval communities.


Scene-15 Dataset

It has 15 visual categories of natural indoor and outdoor scenes.Each category has about 200 to 400 images and the entire datasethas 4485 images.


PASCAL VOC2006 Dataset

It has 10 visual categories with about 175 to 650 images percategory. There are a total of 5304 images.



It has 20 visual categories. Each category contains images rangingfrom 100 to 2000, with 9963 images in all.



It has 20 visual categories and 300 to 3500 images in eachcategory. Combines data from VOC2008 and VOC2009.


Dictionary Size

10,000 words → n Topics. Appropriate number of Topics?

Large dictionary becomes category dependent.


Summary

Visual dictionary in limited: unsupervised clustering.

Significant intra-category appearance variation: semantic scatter.Feature vectors from different visual categories inter-mixed infeature space.

Visual Topic ←∑

Visual Word: grouping over-partitioned featurespace.

Co-clustering Image-Word distribution: discover optimal groupingof words with minimal loss in mutual information.

Semantic dimensionality reduction.


Thank you.

Acknowledgement


Semantically Relevant Visual Dictionary

Technology

Transcript of Semantically Relevant Visual Dictionary