Query Expansion for Hash-based Image Object Retrieval
Yin-Hsi Kuo, Kuan-Ting Chen,Chien-Hsing Chiang, Winston H. Hsu
National Taiwan UniversityTaipei, Taiwan
2
Our Goal–Effective and Efficient Object-level Image Retrieval
Ranking list
Object-level imageQuery image
- Object-level image retrieval
- Effective image retrieval
- Time efficiency
3
Problems in Conventional Content-Based Image Retrieval (CBIR)
• Global feature, e.g., color histogram– Not suitable for object-level image retrieval– Time-consuming for finding nearest neighbors
bin
Color histogram
bin
Color histogram
Query image Target image
4
Image (Many-to-many) Matching by Local Features
• Local feature, e.g., SIFT feature [D. Lowe. IJCV 2004]
– Measuring image similarities by local feature– Time-consuming for many-to-many matching– Requiring efficient indexing methods
• e.g., KD-tree, hashing
Query image Target image
5
• Hash-based indexing method (HB)
• Hash-based indexing + Query expansion (HQ)
5
Query image
Our Proposal–Query Expansion for Hash-based Image Retrieval
After query expansionPrecision-Recall curve
Recall
Precision
HB
HQ
Ideal case
Suffering from high precision but low recall
6
Two Strategies for Query Expansion
• Intra-expansion– Expanding more target feature points similar to query
• Inter-expansion– Mining feature points that shall co-occur with targets
• For example, query– New York– Musicals
- NYC, NewYork, New York City
- musical, show
Intra-expansion
- Cats- Mamma Mia- The Lion King- Directors- Times square
Inter-expansion
7
Highlights of Our Proposal
• Propose intra- and inter-expansion for hash-based image retrieval
• Object-level image retrieval• Hash-based method consumes <1 second to
retrieve images among 5000+ photos• Relative 76.3% (average) improvements over
the original hash-based method
8
Hash-based Indexing Methods
• E.g., Locality Sensitive Hashing (LSH)– Efficient similarity search of high-dimensional data– Similar feature points are mapped to the same
buckets with high probability
• Applicable for different hash functions– Stable distribution [M. Datar. SoCG 2004]
– Random projection [M. Charikar. STOCI 2002]
9
• Hash function: – v: feature point– a: stable distribution
• E.g., L2(Gaussian), L1(Cauchy)– b: 0 ~ W
Illustration of Locality sensitive hashing
⎥⎦⎥
⎢⎣⎢ +⋅
=W
bvavh )( v
Inner product
[M. Datar. SoCG 2004]
Hash table 1
1 2 3 4
3
2
1
Hash table 2
1
2
3
43
2
1High-dimensional feature space
Bucket
W
a
10
• Collecting all local features from images• Hashing them into buckets by LSH• Ranking result by score
LSH for Bags of Feature Points
Ranking scoreImageImageImage
H, I, J, K, L
A, B, C, D
E, F, G
Feature ID Query featureDatabase images Hash tables
[Y. Ke. ACM MM 2004]
11
Illustration of Query Expansion—Intra- and Inter-expansion
LSH
Target feature points:ABCD
B
B
B
A
A
C D
D
Ranking result
C
C
A
C
B
D
Query image
A’, A’’B’, B’’C’, C’’D’
Intra-expansion
B
B
B
A
A
C D
D
Intra-expansion
C
C
B’
B’’
A’’
A’C’
C’’
D’B’’
A’’
A’C’
C’’
D’
Inter-expansion
EFG
B
B
B
A
A
C D
D
Inter-expansion
C
C
E
G F
E
E
G
G
F
E
G F
E
E
G
G
F
A’AA’’
Hash table
12
Intra-expansion in Hashing
• Expanding more feature points similar to query• Possible solutions
– Matched points– Near buckets [Q. Lv. VLDB 2007]
– Meta-featureHash table 1
A C
DE G
F
H I J
B
R R
Hash table 2
C G
HA D
FE
I
J
B
13
Intra-expansion in Hashing—Matched Points
• Expanding more feature points from retrieved features within R radius distance– Red star → E, F, G
• E and G are matched points
– E → B (in hash table 2)
Hash table 1
A C
DE G
F
H I J
B
R R
Hash table 2
C G
HA D
FE
I
J
B
14
Intra-expansion in Hashing—Near Buckets
• Expanding more feature points from near buckets– Retrieve nearest bucket to the query feature– Then retrieve the second nearest bucket
Hash table 1
A C
DE G
F
H I J
B
R R
Hash table 2
C G
HA D
FE
I
J
B
[Q. Lv. VLDB 2007]
15
Intra-expansion in Hashing—Meta-feature
• Expanding more feature points base on the cluster center– Generate cluster centers to represent meta-features– Use meta-features to retrieve more possible
candidates
A C
DE G
F
H I J
B
Meta-feature
Hash table 1
A C
DE G
F
H I J
B
R
16
Inter-expansion in Hashing
• Mining feature points that shall co-occur with targets• Possible solutions
– Pseudo-relevance feedback (Top N)– Utilizing spatial verification to avoid false positive
images• Expanding feature
points from– Full image– Matched Region
Query image
Target image
Region B
Region A
17
Inter-expansion in Hashing—New Feature Points Retrieved from Target Images
Query image
Correct images(Inlier number ≧δ)
Ranking list 2B(Intra-
expansion) ……
Ranking list 1
(Intra-expansion)
A………
Ranking list 3
Inter-expansion
(Intra-expansion) ……
18
Fusion Methods for Inter-expansion Ranking Results
Ranking list 1
……
Ranking list 2
……
Ranking list 3
……
Final ranking list
…
• Average score• Maximum score• Borda count• Average inlier number• Maximum inlier number
Fusion
19
Experimental Dataset–Oxford Buildings Dataset
• Oxford buildings dataset– 5,062 images of particular Oxford landmarks
• 7M (7,162,122) feature points• 55 queries
– Object-based retrieval and landmark recognition
Ashmolean
Balliol
20
Experimental Dataset–Flickr11K Dataset
• Flickr11K dataset– 11,282 images downloaded from Flickr
• 10M (10,632,711) feature points• 56 queries
– More diverse and complicated
Torre PendenteDi Pisa
Starbucks logo
21
Experimental Result on Best Configuration
• Baseline– Hash-based indexing method
• Intra-expansion– Near buckets
• Inter-expansion– Matched region
• Intra- & inter-expansion– Using intra-expansion to retrieve initial ranking result– Doing inter-expansion base on initial ranking result
A C
DE G
F
H I J
B
R
Matched region
22
Experimental Result—Intra- and Inter-Expansion Complement Each Other
• MAP: mean average precision• %: relative improvement
• Combining both intra- and inter-expansion can achieve better results
OxfordMAP %
Baseline 0.261 -
75.00.2255.20.13567.30.215
-0.128%MAP
Flickr11K
52.00.396Intra-expansion
13.10.295Inter-expansion76.30.460Intra- & Inter-expansion
23
(a) Object-level query image
(f) Hash-based + Intra- and inter-expansion32 (3363) 33 (516) 34 (1948) 35 (445) 37 (727)
10 (45) 24 (88) 25 (116) 26 (641) 29 (67)(e) Hash-based + Inter-expansion
14 (779) 16 (22) 18 (72) 19 (37) 24 (548)(d) Hash-based + Intra-expansion
(c)(e) (d) (f)
Precision
Recall
1
0 1(b) PR curve
1 2 12 15 19(c) Hash-based ranking list
Oxford buildings
24
(c)
(e) (d) (f)
Precision
Recall
1
0 1(b) PR curve
1 2 4 6 7(c) Hash-based ranking list
4 (14) 7 (66) 8 (196) 9 (29) 11 (105)(e) Hash-based + Inter-expansion
5 (69) 8 (394) 9 (893) 11 (58) 12 (31)(d) Hash-based + Intra-expansion
(f) Hash-based + Intra- and inter-expansion8 (238) 22 (1387) 25 (183) 29 (1031) 33 (2065)
Flickr11K
(a) Object-level query image
25
Conclusions & Future Work
• Propose two novel expansion strategies–intra-expansion and inter-expansion
• Collaboratively boost the performance significantly in consumer photo benchmarks
• Optimize efficient implementations for the expansions and the spatial verification process
• Seek parallel solutions for further speeding up large-scale image object retrieval
Thank you for your attention!
Q & A
Top Related