BinGAN: Learning Compact Binary Descriptors with a Regularized GANMaciej Ziebaa,b, Piotr Sembereckia,b, Tarek El-Gaalyc, Tomasz Trzcinskia,d

aTooploox bWroclaw University of Technology cVoyage dWarsaw University of Technology

Outline

I GAN for unsupervised learning of image binary descriptors.

I Binary representation of the GAN’s discriminator.I Two regularizations to improve binary representation.

I First transfers distances from high-dimensional to low-dimensional space.

I The second one increases the diversity of binary vectors.

I Evaluation on two applications: image matching and retrieval.I Our approach significantly outperforms state-of-the-art methods.

Contributions

I Compact binary image descriptors learned in an unsupervised manner.I Two regularization methods to achieve binary embeddings on discriminator.

I SOTA results in two tasks: image retrieval and matching.

Learning binary descriptors

Binary descriptors are feature embeddings that are: low-dimensional, quantized.

I Past: hand-crafted feature descriptors have been used for representing images.

I Present: deep neural networks are used to learn binary features.

I Ours: using GAN’s generator to learn binary codes.

0

1

0

1

1

0

1

0

1

GAN model

We consider two networks: generator (G), discriminator (D).

The generator is trained using feature matching:||Ex∼pdata(x)f(x)− Ez∼pz(z)f(G(z))||22.

The discriminator is trained using typical criterion:LD = −Ex∼pdata(x)[log (D(x))]− Ez∼pz(z)[log (1− D(G(z)))].

BinGAN model

I The generator is trained using feature matching.

I The discriminator is trained with the aggregated criterion:

Training BinGAN

I Distance Matching Regularizer (DMR) is responsible for matching distances from high-dimensional (h(·)) tolow-dimensional (f(·)) space.

I Adjusted Binarization Representation Entropy (BRE) Regularizer is increasing the diversity of binary vectorsin the low-dimensional layer:I LME is responsible for enforcing uniform distribution of sf .I LMAC is responsible for minimizing the weighted correlation.

softsign()

sign()

sign()

softsign()

I The functions f(x) and h(x) denote the low and high-dimensional intermediate layers of discriminator with thenumbers of hidden units equal to K and M, respectively.

I The corresponding binary codes (using sign(·)) are denoted by bf and bg.

I The corresponding quantized codes (using softsign(·)) are denoted by sf and sg.

Results - image matching (Brown)

The false positive rates at 95% true positives (FPR@95%).

BinGAN: Learning Compact Binary Descriptors with a Regularized GAN

Table 2. False positive rates at 95% true positives (FPR@95%) obtained for our BinGAN descriptor compared with the state-of-the-artbinary descriptors on Brown dataset (%). Real-valued SIFT features are provided for reference. We report all the results from (?), exceptfor L2-Net and BinGAN.

Train Yosemite Notre Dame Liberty AverageTest Notre Dame Liberty Yosemite Liberty Notre Dame Yosemite FPR@95%

SupervisedLDAHash (16 bytes) 51.58 49.66 52.95 49.66 51.58 52.95 51.40D-BRIEF (4 bytes) 43.96 53.39 46.22 51.30 43.10 47.29 47.54BinBoost (8 bytes) 14.54 21.67 18.96 20.49 16.90 22.88 19.24RFD (50-70 bytes) 11.68 19.40 14.50 19.35 13.23 16.99 15.86

Binary L2-Net (32 bytes) 2.51 6.65 4.04 4.01 1.9 5.61 4.12Unsupervised

SIFT (128 bytes) 28.09 36.27 29.15 36.27 28.09 29.15 31.17BRISK (64 bytes) 74.88 79.36 73.21 79.36 74.88 73.21 75.81BRIEF (32 bytes) 54.57 59.15 54.96 59.15 54.57 54.96 56.23DeepBit (32 bytes) 29.60 34.41 63.68 32.06 26.66 57.61 40.67

DBD-MQ (32 bytes) 27.20 33.11 57.24 31.10 25.78 57.15 38.59BinGAN (32 bytes) 16.88 26.08 40.80 25.76 27.84 47.64 30.76

Table 3. Ablation study. False positive rates at 95% true positives (FPR@95%) for three settings of � parameters when training BinGANfor image matching. Optimizing all three loss terms leads to the best performance on the Brown dataset.

Train Yosemite Notre Dame Liberty AverageTest Notre Dame Liberty Yosemite Liberty Notre Dame Yosemite FPR@95%

�DMR = �ME = �MAC = 0 32.72 39.44 39.44 27.92 27.24 50.48 36.21�DMR = 0 �ME = �MAC = 0.01 30.12 36.28 44.2 24.28 26.44 51.88 35.53�DMR = 0.05 �ME = �MAC = 0 24.68 26.96 40.16 27.00 27.28 45.28 31.90

�DMR = 0.05 �ME = �MAC = 0.01 16.88 26.08 40.80 25.76 27.84 47.64 30.76

vised method previously reporting state-of-the-art resultson this dataset, for 16, 32 and 64 bits. The performanceimprovement in terms of mean Average Precision reachesover 40%, 31% and 15%, respectively. The most significantperformance boost can be observed for the shortest binarystrings, as thanks to the loss terms introduced in our method,we explicitly model the distribution of the information in alow-dimensional binary space.

4.3. Image Matching

To evaluate the performance of our approach on imagematching task, we use the Brown dataset (?) and trainbinary local feature descriptors using our BinGAN methodand competing previous methods, applying the methodologydescribed in (?). The Brown dataset is composed of threesubsets of patches: Yosemite, Liberty and Notredame. Theresolution of the patches is 64⇥ 64, although we subsamplethem to 32⇥32 to increase the processing efficiency and usethe method to create binary descriptors in practice. The datais split into training and test sets according to the providedground truth, with 50,000 training pairs (25,000 matchedand 25,000 non-matched pairs) and 10,000 test pairs (5,000matched, and 5,000 non-matched pairs), respectively.

Tab. 2 shows the false positive rates at 95% true positives(FPR@95%) for binary descriptors generated with our Bin-GAN approach compared with several state-of-the-art de-scriptors. Among the compared approaches, Boosted SSC(?), BRISK (?), BRIEF (?), DeepBit (?) and DBD-MQ (?)are unsupervised binary descriptors while LDAHash (?),D-BRIEF (?), BinBoost (?) and RFD (?) are supervised.

(a) Notredame

(b) Yosemite

Figure 2. The set of randomly selected patches from the originaldata (odd columns) and corresponding synthetically generatedpatches (even columns) that are at the closest Hamming distanceto the true patch in the binary descriptor space.

Results - image retrieval (Cifar10)

Mean Average Precision (mAP) - top 1000.

BinGAN: Learning Compact Binary Descriptors with a Regularized GAN

L = LD + �ME · LME

+ �MAC · LMAC + �DMR · LDMR

(8)

where �ME ,�MAC ,�DMR are regularization pa-rameters and LD = �Ex⇠pdata(x)[log (D(x))] �Ez⇠pz(z)[log (1 � D(G(z)))] is loss for training thediscriminator. The generator in BinGAN model is trainedby minimizing the feature matching criterion provided byequation (??).

4. ResultsWe conduct experiments on two benchmark datasets, Browngray-scale patches (?) and CIFAR-10 color images (?).These benchmarks are used to evaluate the quality of ourapproach on image matching and image retrieval tasks, re-spectively.

4.1. Model Architecture and Parameter Settings

For both tasks, we use the same generator architecture andslight modifications of the discriminator. Below we outlinethe main features of both models and their parameters.

For the image matching task the discriminator is composedof 7 convolutional layers (3x3 kernels, 3 layers with 96 ker-nels and 4 layers with 128 kernels), two network-in-network(NiN) (?) layers (with 256 and 128 units respectively) anddiscriminative layer. For the low-dimensional feature spacebf we take the average-pooled NiN layer composed of256 units. For the high-dimensional space bh we take thereshaped output of the last convolutional layer that is com-posed of 9216 units.

For image retrieval the discriminator is composed of: 7 con-volutional layers (3x3 kernels, 3 layers with 96 kernels and4 layers with 192 kernels), two NiN layers with 192 units,one fully-connected layer with three variants of (16, 32, 64units) and discriminative layer. For the low-dimensionalfeature space bf we take fully-connected layer, and for thehigh-dimensional space bh we take average-pooled last NiNlayer.

In all our experiments, we fix the � parameters to: �DMR =0.05, �ME = 0.01 and �MAC = 0.01. The other hyper-paramters are set to: � = 0.001 and � = 0.5.

4.2. Image Retrieval

In this experiment we use CIFAR-10 dataset to evaluatethe quality of our approach in image retrieval. CIFAR-10dataset has 10 categories and each of them is composedof 6,000 pictures with a resolution 32 ⇥ 32 color images.The whole dataset has 50,000 training and 10,000 testingimages.

Table 1. Performance comparison (mAP, %) of different unsuper-vised hashing algorithms on the CIFAR-10 dataset. This tableshows the mean Average Precision (mAP) of top 1000 returnedimages with respect to different number of hash bits. We reportthe results for all the methods except for BinGAN after (?).

Method 16 bit 32 bit 64 bitKHM 13.59 13.93 14.46SphH 13.98 14.58 15.38SpeH 12.55 12.42 12.56SH 12.95 14.09 13.89PCAH 12.91 12.60 12.10LSH 12.55 13.76 15.07PCA-ITQ 15.67 16.20 16.64DH 16.17 16.62 16.96DeepBit 19.43 24.86 27.73DBD-MQ 21.53 26.50 31.85BinGAN 30.05 34.65 36.77

Figure 1. Top retrieved image matches from CIFAR-10 dataset forgiven query images from test set - first column.

To compare the binary descriptor generated with our Bin-GAN model with the competing approaches, we evaluateseveral unsupervised state-of-the methods, such as: KMH(?), Spherical Hashing (SphH)(?), PCAH (?), Spectral Hash-ing (SpeH)(?), Semantic Hashing (SH) (?), LSH (?), PCT-ITQ (?), Deep Hashing (DH)(?), DeepBit(?), deep binarydescriptor with multiquantization (DBD-MQ)(?). For allmethods except DH, DeepBit, DBD-MQ and ours, we fol-low (?) and compute hashes on 512-d GIST descriptors.The table ?? shows the CIFAR10 retrieval results based onthe mean Average Precision (mAP) of the top 1000 returnedimages with respect to different bit lengths. Fig. ?? showstop 10 images retrieved from a database for given queryimage from our test data.

Our method outperforms DBD-MQ method, the unsuper-

Query Top 10 retrieved base images

Image matching - qualitative analysis

True patches (left) and the closest generated (right).

Thirty-second Conference on Neural Information Processing Systems (NIPS 2018)