MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Mining Multiple Queries for Image Retrieval

Authors: Basura Fernando and Tinne Tuytelaars

Presented: Manish Sinha

manish.sinha@asu.edu

December 3, 2013

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Introduction

Presents a new method for object retrieval from multiplequery images.

This method is based on pattern mining using theminimum length descriptor principle.

It tries to find the most suitable set of patterns to describethe query objects with patterns corresponding to localfeatures configuration.

The archive is maintained using two inverse file systemsfor searching patterns.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Related Work

Total recall: Automatic query expansion with agenerative feature model for object retrieval by OChum, J Philbin, J Sivic, M Isard (Computer Vision, 2007)

The Minimum Description Length Principle by P. D.Grunwald (THE MIT PRESS, 2007)

Krimp: mining itemsets that compress by J Vreeken,M Van Leeuwen, A Siebes (Mining and KnowledgeDiscovery, 2011)

Effective use of frequent itemset mining for imageclassification by B Fernando, E Fromont, T Tuytelaars(Computer VisionECCV 2012)

Efficient mining of frequent and distinctive featureconfigurations by V Ferrari, B Leibe, L Van Gool ( ICCV2007)

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Pattern Based Image Retrieval: Introduction

Let I be set of items, in our case items are visual words. Atransaction t is a set of items e.g. t = {i1, i2, i5, i9} andeach ij is a visual word. Let D = {t1, t2, t3, t4} known as adatabase of transactions.

Pattern can be considered as subsets of transactions. Forexample, x1 = {i1, i2, i9} could be a pattern. We say apattern x is matched/mapped to transaction t if x ⊆ t

A visual vocabulary is created using K-means clusteringand each descriptor extracted from a key point is assignedto one of the visual words.

We use local bag-of-words (LBOW) as representation for alocal region surrounding a key point.

Finally, LBOWs are transformed into transactions byconsidering each non-zero bin as an item.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Pattern Based Image Retrieval: Overall Process

For the archive images, we process the data offline. Weindex the transaction using two inverted file systems.For first inverted file system, IFS1, the key is a visual wordlabel and the corresponding entry consists of a list oftransactions. The second inverted file system, IFS2 , mapseach transaction id to the set of corresponding images.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Pattern Based Image Retrieval: Pattern Mining

If x is a pattern in database D containing transactions,then usage(x/D) = | {t ∈ D : x ∈ cover(H, t)} |, wherecover : {H × t} → PowerSet(PowerSet(I))

A model H is a set of patterns. We construct all the set ofmodels (Hall) and select the best model (H∗) based onminimum description length principle using the formula.H∗ = argminH∈Hall

L(H) + L(D/H)

Where L(H) is length of model in bits and L(D/H) islength of query image data once it is encoded with H.Where L(H) =

∑x∈H

[L(x/H) + L(x/Hst)]

Hst is the standard model consisting of only singletonitems and L(x/H) = - log(P(x/D)), P (x/D) =

usage(x/D)∑∀y∈H

usage(y/H)

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Pattern Based Image Retrieval: Pattern Mining

L(D/H) =∑t∈D

L(t/H), L(t/H) =∑

x∈cover(H,t)

L(x/H)

We L2 normalize the tf-idf weighted bag of patterns andobtain final score for each image using square roothistogram intersection.

Selecting consistent keypoints: This preprocessing stepidentifies a set of key points that are consistent acrossmultiple query images. Inconsistent key points, that arefound only in a single query image, are filtered out. Wecall this step Multiple query basic matching

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Experiment Setup

Two data sets named Oxford-Buildings with 5K imagesand Paris-Dataset containing 100K images, totally 105K.The third dataset contains old and new images ofbuildings, statues, churches, castles and other landmarksin Belgium, thereby called as the ”Belgium landmarksdataset”.

Each query consists of five images of a specific place orobject from different time periods, viewing conditions andviewpoints.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Experiment Results

Comparison with other algorithms: The above chartshows that our KRIMP based method outperforms allother algorithms with highest Mean Average Precision.Execution Time and Complexity Analysis: ProcessingQuery image took around 2-3 secs for 8 query images andpattern based image retrieval from inverted file systemtook 0.1 seconds on a CPU with 2.6GHz processor and250K images archive and inverted filesystem tok 1500Mbfor 105K images.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Advantages

It is an unsupervised method and does not need anynegative images

The method is robust to noisy images as it exploresregularity in query images using MDL principle.

The resulting KRIMP based representation is compact.

Using this method, relevant images can be accurately andefficiently even without geometric verifications.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Disadvantages

Performance is non-significant for lesser number of images.

Multiple Query Basic Matching step is still not immune tonoise even after removing inconsistent key points foundonly in one image.

MiningMultiple

Queries forImage

Retrieval

Authors:Basura

Fernando andTinne

Tuytelaars

Introduction

Related Work

Pattern BasedImageRetrieval

Experiment

Advantages

Disadvantages

Conclusion

Summary and Conclusion

A multiple query based image retrieval method thatsystematically outperforms other MQIR methods evenwithout costly spatial verification.

Used a minimum description length based pattern miningalgorithm to discover local structural patterns for specificobject retrieval.

Introduces a novel query expansion method based onpattern mining which we call pattern based queryexpansion.

KRIMP algorithm selects the best model (set of patterns)

Ranks the results using tf-idf weighted bag of patterns andthen runs L2 normalization of the tf-idf results.