Recent Advances of Compact Hashing for Large-Scale Visual Search Shih-Fu Chang Columbia University...
-
Upload
debra-tate -
Category
Documents
-
view
214 -
download
1
Transcript of Recent Advances of Compact Hashing for Large-Scale Visual Search Shih-Fu Chang Columbia University...
Recent Advances of Compact Hashing for Large-Scale Visual Search
Shih-Fu Chang
Columbia University
October 2012
Joint work with Junfeng He (Facebook), Sanjiv Kumar (Google), Wei Liu (IBM Research), and Jun Wang (IBM Research)
digital video | multimedia lab
Outline Lessons learned in designing hashing functions
The importance of balancing hash bucket size How to incorporate supervised information
Prediction of NN search difficulty & hashing performance
Demo: Bag of hash bits for Mobile Visual Search
Fast Nearest Neighbor Search• Applications: image search, texture synthesis, denoising … • Avoid exhaustive search ( time complexity)
3
Dense matching, Coherence sensitive hashing (Korman&Avidan ’11)
Photo tourism patch search
Image search
Locality-Sensitive Hashing
• hash code collision probability proportional to original similarityl: # hash tables, K: hash bits per table
0
1
0
10
1
4
hash function
random
101 Index by compact code
[Indyk, and Motwani 1998] [Datar et al. 2004]
Hash Table based Search
5
• O(1) search time by table lookup• bucket size is important (affect accuracy & post processing
cost)
xi
n
q01101
01101
01110
01111
01100
hash table
hash bucket address
Different Approaches
6
Unsupervised Hashing
LSH ‘98, SH ‘08, KLSH ‘09,AGH ’10, PCAH, ITQ ‘11
Semi-Supervised Hashing
SSH ‘10, WeaklySH ‘10
Supervised Hashing
RBM ‘09, BRE ‘10, MLH ‘11, LDAH ’11,ITQ ‘11, KSH ‘12
PCA + Minimize Quantization Errors
• PCA to maximize variance in each hash dimension• find optimal rotation in the subspace to minimize
quantization error
ITQ method, Gong&Lazebnik, CVPR 11
Effects of Min Quantization Errors
• 580K tiny images PCA-ITQ, Gong&Lazebnik, CVPR 11
PCA-random rotation PCA-ITQ optimal alignment
Utilize supervised labelsSemantic Category Supervision
9
Metric Supervision
similar
dissimilardissimilar
similar
dissimilar
Design Hash Codes to Match Supervised Information
10
similar
dissimilar
0
1• Preferred hashing function
Adding Supervised Labels to PCA Hash
Relaxation:
Wang, Kumar, Chang, CVPR ’10, ICML’10
“adjusted” covariance matrix
• solution W: eigen vectors of adjusted covariance matrix• If no supervision (S=0), it is simply PCA hash
Fitting labels PCA covariance matrix
dissimilar pairsimilar pair
Semi-Supervised Hashing (SSH)1 Million GIST Images1% labels, 99% unlabeled
Supervised RBM
Random LSH
Unsupervised SH
SSHPrecision @ top 1K
Problem of orthogonal projections
• Many buckets become empty when # bits increases.
• Need to search many neighbor buckets at query time
Precision @ hamming radius 2
• Explicitly optimize two terms– Preserve similarity (accuracy)– Balanced bucket size max entropy min mutual info I (search time)
Search accuracy
ICA Type Hashing
2
, 1
( ) || ||N
pq p qp q
D Y W Y Y
Balanced bucket size
1
1
min ( ,..., ,..., )
while ( ) 0
k M
N
pp
I y y y
E y Y
SPICA Hash, He et al, CVPR 11
Fast ICA to find non-orthogonal projections
The Importance of balanced size
Bucket index
Buck
et s
ize LSHSPICA HashBalanced bucket size
Simulation over 1M tiny image samples
The largest bucket of LSH contains 10% of all 1M samples
Different Approaches
16
Unsupervised Hashing
LSH ‘98, SH ‘08, KLSH ‘09,AGH ’10, PCAH, ITQ ‘11
Semi-Supervised Hashing
SSH ‘10, WeaklySH ‘10
Supervised Hashing
RBM ‘09, BRE ‘10, MLH ‘11, LDAH ’11,ITQ ‘11, KSH ‘12
Better ways to handle supervised information?
17
MLH [Norouzi & Flee, ‘11]
BRE [Kulis & Darrell, ‘10]Hamming distance between H(xi) and H(xj)
hinge loss
But optimizing Hamming Distance (DH, XOR) is not easy!
A New Supervision Form: Code Inner Products
18
S
x2
x3
x1
dis
sim
ilar
similar
supervised hashing
labeled data
dissim
ilar
1 -1 1
1 -1 1
-1 1 -1
1 1 1
-1 -1 1
1 1 -1Х
Tcode matrix
1 1 -1
1 1 -1
-1 -1 1
x1
x2
x3
x1 x2 x3
pair-wise label matrix
code inner products
rx1
x2
x3
code matrix
fitting
Liu, Wang, Ji, Jiang, Chang, CVPR’12
proof: code inner product ≡ Hamming distance
Code Inner Product enables efficient optimization
• Much easier/faster to optimize and extend to kernels
19
sample
hash bitHashing:
Design hash codes to match
supervised information
Liu, Wang, Ji, Jiang, Chang, CVPR2012
Extend Code Inner Product to Kernel• Following KLSH, construct a hash function using a kernel
function and m anchor samples:
zero-mean normalization applied to k(x).
20
1 -1 1
1 -1 1
-1 1 -1
1 1 -1
=sgn
hash coefficientskernel matrix
×l samples
m anchors
Benefits of Code Inner Product
21
•CIFAR 10, 60K object images from 10 classes, 1K query images.
•1K supervised labels. •KSH0 Spec Relax, KSH Sigmoid hashing function
Supervised Methods
Open Issue: empty buckets and balance not addressed
Speedup by Inner Code Product
22CVPR 2012
Method
Train Time Test Time
48 bits 48 bits
SSH 2.1 0.9×10−5
LDAH 0.7 0.9×10−5
BRE 494.7 2.9×10−5
MLH 3666.3 1.8×10−5
KSH0 7.0 3.3×10−5
KSH 156.1 4.3×10−5
Significant speedup
25
Tiny-1M: Visual Search Results
CVPR 2012
More visuallyrelevant
Comparison of Hashing vs. KD-Tree
Supervised Hashing
Photo Tourism Patch set (Norte Dame subset, 103K samples)512D GIFTAnchor Graph
Hashing
KD Tree
• How difficult is approximate nearest neighbor search in a dataset?
Understand Difficulty of Approximate Nearest Neighbor Search
Toy example
q
x is an ε-approximate NN if
Search not meaningful!
A concrete measure of difficulty of search in a dataset?
He, Kumar, Chang, ICML 2012
• A naïve search approach: Randomly pick a point and compare that to the NN
Relative Contrast
q
Relative Contrast
• High Relative Contrast easier search
• If , search not meaningful
He, Kumar, Chang, ICML 2012
• With CLT, and binomial approximation
Estimation of Relative Contrast
ϕ - standard Gaussian cdf
σ' – a function of data properties (dimensionality and sparsity)
n: data sizep: Lp distance
• Data sampled randomly from U[0,1]
Synthetic Datare
lati
ve c
ontr
ast
rela
tive
con
tras
t
higher dimensionality bad sparser vectors good
s: prob. of non-zero element in each dim.d: feature dimension
• Data sampled randomly from U[0,1]
Synthetic Data
rela
tive
con
tras
t
rela
tive
con
tras
t
lower p goodLarger database good
Predict Hashing Performance of Real-World Data
16 bits LSH
Dataset Dimensionality (d)
Sparsity (s)
Relative Contrast (Cr) for p = 1
SIFT 128 0.89 4.78
Gist 384 1.00 1.83
Color Hist 1382 0.027 3.19
Imagenet BoW 10000 0.024 1.90
28 bits LSH
Mobile Search System by Hashing
34
Light Computing Low Bit Rate Big Data Indexing
He, Feng, Liu, Cheng, Lin, Chung, Chang. Mobile Product Search with Bag of Hash Bits and Boundary Reranking, CVPR 2012.
Estimate the Complexity
• 500 local features per image– Feature size ~128 Kbytes– more than 10 seconds for transmission over 3G
• Database indexing– 1 million images need 0.5 billions local features– Finding matched features becomes challenging
• Idea: directly compute compact hash codes on mobile devices
Approach: hashing• Each local feature coded as hash bits
– locality sensitive, efficient for high dimensions• Each image is represented as Bag of Hash Bits
011001100100111100…
110110011001100110…
36
Bit Reuse for Multi-Table Hashing• To reduce transmission size
– Reuse a single hash bit pool by random subsampling
37
1 0 0 1 1 1 0 0 0 0 1 0 1 0 1 0 . . . 0 0 1 1 0 1 1 1
Optimal hash bit pool (e.g., 80 bits, PCA Hash or SPICA hash)
Random subset
Random subset
Random subset
Random subset. . .
Table 1 Table 2 Table 11 Table 12. . . 32 bits
Union Results
Rerank Results with Boundary Features• Use automatic salient object segmentation for every
image in DB [Cheng et al, CVPR 2011]
• Compute boundary features: normalized central distance, Fourier magnitude
• Invariance: translation, scaling, rotation
38
Boundary Feature – Central Distance
Distance to Center D(n) FFT: F(n) 39
Reranking with boundary feature
40
Server:• 1 million product images crawled from
Amazon, eBay and Zappos• Hundreds of categories; shoes, clothes,
electrical devices, groceries, kitchen supplies, movies, etc.
Speed• Feature extraction: ~1s • Transmission:
80 bits/feature, 1KB/image• Serer Search: ~0.4s• Download/display: 1-2s
Mobile Product Search System: Bags of Hash Bits and Boundary features
video demo (52”)
He, Feng, Liu, Cheng, Lin, Chung, Chang. Mobile Product Search with Bag of Hash Bits and Boundary Reranking, CVPR 2012.
Performance• Baseline [Chandrasekhar et al CVPR ‘10]:
Client: compress local features with CHoGServer: BoW with Vocabulary Tree (1M codes)
30% higher recall and 6X-30X search speedup
42
Summary• Some Ideas Discussed
– bucket balancing is important– code inner product – an efficient form of supervised
hashing– insights on search difficulty prediction– Large mobile search – a good test case for hashing
• Open Issues– supervised hashing vs. attribute discovery– hashing beyond point-to-point search– hashing to incorporate structured relation (spatio-
temporal)
43
References• (Supervised Kernel Hash)
W. Liu, J. Wang, R. Ji, Y. Jiang, and S.-F. Chang, Supervised Hashing with Kernels, CVPR 2012.
• (Difficulty of Nearest Neighbor Search)J. He, S. Kumar, S.-F. Chang, On the Difficulty of Nearest Neighbor Search, ICML 2012.
• (Hash Based Mobile Product Search)J. He, T. Lin, J. Feng, X. Liu, S.-F. Chang, Mobile Product Search with Bag of Hash Bits and Boundary Reranking, CVPR 2012
• (Hashing with Graphs)W. Liu, J. Wang, S. Kumar, S.-F. Chang. Hashing with Graphs, ICML 2011.
• (Iterative Quantization)Y. Gong and S. Lazebnik, Iterative Quantization: A Procrustean Approach to Learning Binary Codes, CVPR 2011.
• (Semi-Supervised Hash)J. Wang, S. Kumar, S.-F. Chang. Semi-Supervised Hashing for Scalable Image Retrieval. CVPR 2010.
• (ICA Hashing)J.He, R. Radhakrishnan, S.-F. Chang, C. Bauer. Compact Hashing with Joint Optimization of Search Accuracy and Time. CVPR 2011. 44