Deep Graphical Feature Learning for Face Sketch SynthesisMingrui Zhu†, Nannan Wang‡∗, Xinbo Gao†, Jie Li†† State Key Laboratory of Integrated Services Networks,

School of Electronic Engineering, Xidian University, Xi’an 710071, China‡ State Key Laboratory of Integrated Services Networks,

School of Telecommunications, Xidian University, Xi’an 710071, Chinamrz.edu@gmail.com, nnwang@xidian.edu.cn, {leejie,xbgao}@mail.xidian.edu.cn

AbstractThe exemplar-based face sketch synthesis methodgenerally contains two steps: neighbor selectionand reconstruction weight representation. Pixel in-tensities are widely used as features by most of theexisting exemplar-based methods, which lacks ofrepresentation ability and robustness to light varia-tions and clutter backgrounds. We present a novelface sketch synthesis method combining genera-tive exemplar-based method and discriminativelytrained deep convolutional neural networks (dC-NNs) via a deep graphical feature learning frame-work. Our method works in both two steps byusing deep discriminative representations derivedfrom dCNNs. Instead of using it directly, we boostits representation capability by a deep graphicalfeature learning framework. Finally, the optimalweights of deep representations and optimal recon-struction weights for face sketch synthesis can beobtained simultaneously. With the optimal recon-struction weights, we can synthesize high qualitysketches which is robust against light variationsand clutter backgrounds. Extensive experiments onpublic face sketch databases show that our methodoutperforms state-of-the-art methods, in terms ofboth synthesis quality and recognition ability.

1 IntroductionFace sketch synthesis aims at solving two problems: 1)givena face photo, synthesize a sketch drawing; 2)given a queryface sketch drawing, retrieve face photos in the database. Itcan effectively assist law enforcement [Wang et al., 2014]. Inmany criminal cases, suspects are well disguised so that theclear photo images of their faces are difficult and sometimeseven impossible to acquire. Under these circumstances, nor-mal face recognition methods based on photo images wouldnot be effective. The best substitute is often a sketch draw-ing based on the recollection of an eyewitness [Tang andWang, 2009]. Due to the great texture discrepancy betweenthe query sketch drawing and gallery mug shots, the speciallydevised face sketch synthesis method is extremely important.

∗Corresponding author: Nannan Wang (nnwang@xidian.edu.cn)

In addition, transforming photos into sketches is also a usefulapplication in the entertainment [Song et al., 2014].

Over the past decade, substantial advances have been madein face sketch synthesis. Among the various face sketchsynthesis methods, exemplar-based methods achieves thebest performance which benefit from their success in detect-ing and exploiting patch correspondences within a trainingdatabase or calculating optimized dictionaries allowing forhighly sparse data representation. Another growing branchformulate the map between photos and sketches as a regres-sion problem [Zhu and Wang, 2016], whose advantage liesin their computational speed. Recently, convolutional neuralnetwork (CNNS) has also been used as a nonlinear regressionmodel [Zhang et al., 2015]. However, due to the limited rep-resentation ability of their loss function (mean squared error,MSE), their results got blur effect and thus have less satisfiedperceptual quality.

The most striking successes in deep learning have involveddiscriminative models for tasks such as image classificationand object recognition [Goodfellow et al., 2014]. It wasshown that such models could learn to extract high-level im-age content in generic feature representations which simul-taneous possess discriminative capacity. Gatys et al. [2016]have recently utilized the filter pyramid of the VGG network[Simonyan and Zisserman, 2014] as a higher-level featurerepresentations for image style transfer. The method yieldsvery impressive artistic style results. However, feature pyra-mid at different layers are directly used to capture only pixelcorrelations of the image ”style” and ”content” so that the re-sults lack of structural similarity.

In this paper, we combine the the generative exemplar-based method with discriminatively trained deep convolu-tional neural networks for face sketch synthesis. Featuremaps (deep representations) derived from the VGG network,which was trained to perform ImageNet [Deng et al., 2009]classification, are used to represent the face photos. Such fea-ture maps with multi channels will have a large amount of re-dundant information. So we linearly combine those channelsweighted by a weight vector. At the beginning, all channelshave the same proportion, but a better weight combinationwill be learnt later. All face photos and sketches are dividedinto patches with overlap between the neighboring patchesso that the transformation problem of the whole image is re-duced to the local patch level. Similarly, deep representations

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3574

GreyPixel

Deep Representations

Image4Image3Image1 Image2 Image6Image5

ProbePatch

√

Figure 1: Comparison of neighbor selection between deeprepresentations and pixel intensity.

of the whole photo is reduced to the patch level. Then, foreach test photo patch, K nearest patches are selected fromthe training photo patches according to the Euclidean distancebetween their deep representations. Figure 1 shows that can-didate patches selected based on their deep representations issuperior to the candidates selected based on their pixel inten-sities. Sketch patches corresponding to the candidate photopatches are taken as the candidates for sketch patch synthesisand the prevalent way to synthesize the target sketch patch isthe linear combination of selected candidate sketch patches.In order to jointly model the distribution over the weights fordeep representations and the distribution over the weights forsketch reconstruction, we designed a graphical model-baseddeep feature learning framework.

The main contributions of this work are summarized as fol-lows:

1) Deep discriminative representations derived from thedeep Convolutional Neural Networks, which performs morerobust respect to light variations and clutter backgrounds, areused to represent face photos. More accurate candidate sketchpatches and weight combination for sketch patch reconstruc-tion could be obtained benefiting from the deep representa-tions.

2) A deep feature learning framework based on graphicalmodel are proposed to jointly learn weights for deep repre-sentations and reconstruction weights.

3) Extensive experimental results illustrate the superiorperformances of our method compared with other state-of-the-art approaches in terms of both synthesis quality andrecognition performance.

The rest of this paper is organized as follows. Section 2outlines the existing literature on face sketch synthesis. Sec-tion 3 introduces the proposed deep graphical feature learning(DGFL) method. Experimental results and analysis are givenin section 4 and section 5 concludes this paper.

2 Related WorkRecent works on face sketch synthesis include exemplar-based approaches and regression-based approaches.

Tang and Wang [2003] pioneered exemplar-based ap-proach by the work of Eigen-transformation which assumedthat the mapping between a photo and its corresponding

sketch is a linear transformation when their shape and texturewere processed separately. The reconstruction weights arelearned by projecting the input photo onto the training photosthrough principal component analysis. However, whole facephotos and face sketches cannot be simply represented by alinear transformation, especially when the hair region is con-sidered. Liu et al. [2005] presented a locally linear embed-ding (LLE) based face sketch synthesis approach, with theidea of locally linear approximating global nonlinear. Thismethod works on the image patch level. Each patch of thetest photo was reconstructed from the linear combination ofseveral selected training photo patches. Then the correspond-ing synthesized sketch patches could obtained from the lin-ear combination of corresponding candidate sketch patches inthe training set. This approach had the weaknesses that eachpatch was independently synthesized and thus neglected com-patible relationships between the neighboring image patches.Therefore, neighborhood information is not well utilized andblock effect was produced. Liu et al. [2007] developed a sta-tistical Bayesian Tensor Inference between the photo patchspace and the sketch patch space to capture and learn theinter-space dependencies. In order to introducing neighbor-hood information, Wang and Tang [2009] employed Markovrandom field (MRF) to model the distribution from two as-pects: the distribution between test photo patches and nearestphoto patches and the distribution between adjacent synthe-sized sketch patches. But this approach can not synthesizenew patches existing not in the training set and its optimiza-tion is NP hard. Zhou et al. [2012] introduce the linear com-bination into the MRF model (namely Markov weight field,MWF) by formulated their model into a convex quadratic pro-gramming (QP) problem and proposed a cascade decompo-sition method (CDM) to solve it. Wang et al. [2017] pro-posed a Bayesian framework which provided an interpreta-tion to existing face sketch synthesis methods from a prob-abilistic graphical view. Another branch of exemplar-basedapproaches is sparse representation-based methods. Sparserepresentation has been applied to various computer visiontasks, in which subsets weighted by a sparse vector are se-lected to represent the input signal. Chang et al. [2010] builda coupled dictionary with the photo and sketch patch pairs us-ing sparse coding. The input test photo was decomposed onthe photo elements in the coupled dictionary using sparse co-efficients. The sketch patch could then be computed using thesketch elements in the coupled dictionary and the previouslyobtained sparse coefficients. Gao et al. [2012] proposedto adaptively determine the number of nearest neighbors bysparse representation and proposed a sparse-representation-based enhancement strategy to enhance the quality of the syn-thesized photos and sketches.

Regression-based approaches formulate the map betweenphotos and sketches as a regression problem. They firstlearn a regression model between face photos (photo patches)and their corresponding face sketches (sketch patches) in thetraining stage. Then given a test face photo (photo patch), theregression model can predict the synthesized sketch (sketchpatch). These approaches usually has a very fast speed intest stage because all time-consuming works are completedin training stage. Zhu and Wang [2016] proposed to use a

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3575

Relu1_2 Relu2_2

VGG 16 Net

Input Photo

Output Sketch

Blocking

Training Photos

Training Sketches

Feature maps

,n lu L1,n lu

L

m,luL

m 1,lu

L

1,m kw

Km,kw K

1,n kw

K,n kwK

Weights for channels of feature maps

Weights for sketch patches

DGFL

1(w , w )n n

(w , w )m n

1(u , w )m m

1 1(u , w )n n

Figure 2: Framework of the proposed deep graphical featurelearning for face sketch synthesis.

ridge regression model to learn the mapping between photopatches and their corresponding sketch patches in the samecluster. Convolutional neural networks (CNNs) has broughtwidespread attention in recent years and it also can be usedas a nonlinear regression model. Zhang et al. [2015] pro-posed an end-to-end sketch generation model via fully convo-lutional networks (FCN) to directly model the complex non-linear mapping between face photos and sketches. However,due to the lack of deep layers, this model failed to fully ex-press the nonlinearity between face photos and sketches andthus its results have poor perceptual quality. In addition,the loss function of mean squared error (MSE) of regressionmodel leads to blur effect.

3 Deep Graphical Feature Learning for FaceSketch Synthesis

In this section, we would introduce in detail how to repre-sent photos with deep feature maps and then how to optimizethe reconstruction weights and deep representations jointly inthe proposed deep graphical feature learning framework. Theoverall framework of the proposed method is shown in Figure2.

3.1 Deep Feature ExtractionSupposing there are M training photo-sketch pairs and agiven test photo, which are geometrically aligned accordingto three points: two eye centers and the mouth center. Eachimage is cropped to the size of 250× 200. We put all photosinto a pre-trained 16-layer VGG network, which was trainedto perform ImageNet classification and is described exten-sively in the original work [Gatys et al., 2016], and performforward propagation. Feature maps (deep representations)which have 128 channels derived from the relu2 2 layer ofthe network are used to represent the input photos. Each fea-ture map generated by a convolution kernel could be regardedas a feature that concerned with a specific characteristic of thephotos. Figure 3 provides some channels of deep representa-tions for a test photo. As the size of convolution kernel in

...

Deep representations

with 128 channels

Input photo

Figure 3: Deep representations of a test photo derived fromVGG 16 Net.

VGG-16 network is 3 × 3, the feature maps we get woulddecrease in size. Thus, we resize each feature map back to250×200 in order to consistent with the size of input photos.Then we divide all the photos, feature maps and sketches intoN overlapping patches. Let xi be the ith test photo patch,where i = 1, 2, ..., N . The deep representations of the testphoto patch can be represented by the linear combination of128 feature maps weighted by the 128 dimensional vector ui:

D(xi) =L∑

l=1

ui,ldl(xi) (1)

where dl(xi) means lth feature map of patch xi, ui,l de-notes the weight of lth feature map, l = 1, 2, ..., L, and∑L

l=1 ui,l = 1. Here L equals 128. Note that the weightsof deep representations at various locations in photo patchesare different, and each feature map has uniform weight in ini-tial. Convergent weights for deep representations and weightsfor sketch reconstruction will be learnt later.

3.2 Deep Graphical Feature LearningGiven a test photo patch xi and its deep representationsD(xi), our target is to synthesize the corresponding sketchpatch yi, where i = 1, 2, ..., N . Firstly, we find K candidatephoto patches {xi,1,xi,2, ...,xi,K} that most like xi accord-ing to the Euclidean distance of their deep representationsand their corresponding sketch patches {yi,1,yi,2, ...,yi,K}within the search region around the location of xi. We as-sume that deep representations of a test photo patch and itstarget sketch patch can be represented by the same linearcombination of the K candidate photo patches’ deep repre-sentations and sketch patches respectively. Then, the targetsketch patch yi can be obtained by the linear combination ofK candidate sketch patches weighted by the K-dimentionalvector wi:

yi =K∑

k=1

wi,kyi,k (2)

wherewi,k denotes the weight of the kth candidate sketch and∑Kk=1 wi,k = 1.Now, we get two weight vectors to be optimized: weights

of deep representations ui and weights of sketch patch recon-

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3576

struction wi. wi directly determines the quality of the syn-thesized sketch patch while ui determines the representationability and indirectly influence the reconstruction weights.Similar with the graphical network in the work [Zhou et al.,2012], we design a deep graphical feature learning frame-work to jointly model the distribution of the weights fordeep representations and the distribution of the weights forsketch reconstruction. The joint probability of ui and wi ,∀i ∈ {1, ..., N}, is formulated as:

p(u1, ...,uN ,w1, ...,wN )

∝N∏i=1

Φ(ui,wi)∏

(i,j)∈Ξ

Ψ(wi,wj)N∏i=1

Υ(ui),(3)

where Φ(ui,wi) is the local evidence function:

Φ(ui,wi)

= exp{−L∑

l=1

ui,l‖dl(xi)−K∑

k=1

wi,kdl(xi,k)‖2/2δ2D}

(4)

and Ψ(wi,wj) is the neighboring compatibility function:

Ψ(wi,wj)

= exp{−‖K∑

k=1

wi,koji,k −

K∑k=1

wj,koij,k‖2/2δ2

S}(5)

and Υ(ui) is the regularization function:

Υ(ui) = exp{−λ‖ui‖2}. (6)

Here dl(xi) means the lth feature map of patch xi anddl(xi,k) means the lth feature map of kth candidate patch.(i, j) ∈ Ξ means the ith and jth patches are neighbors. oj

i,k

represents the overlapping area of the candidate sketch patchyi,k with the jth patch. λ balances the regularization termwith the other two terms.

In order to obtain the optimal weights ui and wi, the jointprobability in (3) should be maximized. We detail the opti-mization strategy in Section 3.3.

3.3 OptimizationThe problem of maximizing the posteriori probability (3) canbe converted to minimizing the following problem:

minu,w

1

2δ2S

∑(i,j)∈Ξ

‖K∑

k=1

wi,koji,k −

K∑k=1

wj,koij,k‖2

+1

2δ2D

N∑i=1

L∑l=1

ui,l‖dl(xi)−K∑

k=1

wi,kdl(xi,k)‖2

+N∑i=1

λ‖ui‖2

s.t.k∑

k=1

wi,k = 1, 0 ≤ wi,k ≤ 1

L∑l=1

ui,l = 1, 0 ≤ ui,l ≤ 1

(7)

where i = 1, 2, ..., N , k = 1, 2, ...,K , l = 1, 2, ..., L.An alternating optimization strategy can be used to solve

this problem.1)Fix u, the problem (6) can be simplified to:

minw

N∑i=1

‖D(xi)−K∑

k=1

wi,kD(xi,k)‖2

+ α∑

(i,j)∈Ξ

‖K∑

k=1

wi,koji,k −

K∑k=1

wj,koij,k‖2

(8)

where α = δ2D/δ

2S . This problem can be solved using the

CDM proposed in [Zhou et al., 2012].2) Fix w, the problem (6) can be simplified to:

minui

L∑l=1

ui,lpi,l + λ‖ui‖2

⇒ minui

λuTi ui + pT

i ui

(9)

where

pi,l =1

2δ2D

‖dl(xi)−K∑

k=1

wi,kdl(xi,k)‖2 (10)

The problem (9) is a standard convex QP problem which canbe effectively solved.

The initial ui,l are set to 1/L andwi,k are set to 1/K. Thenwe alternate execute 1) and 2) until convergence. Once weget the optimal weights wi, the target sketch patch can besynthesized by (2). After obtaining all target sketch patches{y1,y2, ...,yN}, they are stitched into a whole sketch withoverlapping area averaged. We summarize the proposedDGFL algorithm in Algorithm 1.

Algorithm 1 Deep Graphical Feature Learning for FaceSketch Synthesis

Input: Deep representations of training photo patches,training sketch patches, test photo patches, deep represen-tations of test photo patches, the number of neighboringpatches K, the search region, α, λ;Initialize: ui,l = 1/L, wi,k = 1/K;Step 1: For each test photo patch xi, find its K candidateneighboring patches from the training photo patches withinthe search region around the location of xi according to theEuclidean distance of their deep representations;repeat

Step 2: Fix u, optimize the problem (8) to compute w;Step 3: Fix w, optimize the problem (8) to compute u;

until convergenceStep 4: Synthesized all target sketch patches withw. Stitchthem into a whole sketch with overlapping area averaged.Output: The target sketch.

4 Experimental Results and AnalysisIn this section, we report the experimental results of the pro-posed DGFL method quantitatively and qualitatively. We

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3577

conducted our experiments on the Chinese University ofHong Kong (CUHK) face sketch database (CUFS) [Tang andWang, 2009]. The CUFS database consists of face photosfrom three databases: the CUHK student database [Tang andWang, 2002] (188 persons), the AR database [Martinez andBenavente, 1998] (123 persons) and the XM2VTS database[K. Messer and J. Luettin, 1999] (295 persons). Persons in theXM2VTS database are different in ages, skins (races) and hairstyles. Each person in the database has a face photo and corre-sponding face sketch. All these face photos and sketches aregeometrically aligned relying on three points: two eye centersand the mouth center and they are cropped to the size of 250× 200. In the following context, we firstly explain the exper-imental settings. Then we perform face sketch synthesis onthe CUFS database to qualitatively illustrate the superiorityof the proposed DGFL method compared with state-of-the-arts. Subsequently, objective image quality assessment andface recognition experiments are conducted to quantitativelyillustrate the superiority of the proposed method.

Photo LLE MRF MWF Bayesian FCN DGFL

Figure 4: Synthesized sketches on the CUFS database byLLE, MRF, MWF, Bayesian, FCN, the proposed DGFLmethod.

Table 1: AVERAGE SSIM SCORE (%)Methods LLE MRF MWF Bayesian FCN DGFL

SSIM (%) 52.58 51.32 53.93 55.43 52.13 56.45

4.1 Experimental SettingsThe parameters used were set as follows: the image patchsize was 10, the overlap size was 5, the size of search regionwas 5, the number of candidate patches K was set to 10, theα was set to 0.25, the λ was set to 2. For the CUHK student

Photo LLE MRF MWF Bayesian DGFL

Figure 5: Synthesized sketches of the photos with extremelighting variance.

database, 88 pairs of face photo-sketch are taken for trainingand the rest for testing. For the AR database, we randomlychoose 80 pairs for training and the rest 43 pairs for testing.For the XM2VTS database, we randomly choose 100 pairsfor training and the rest 195 pairs for testing. Five state-of-the-art methods are compared: the LLE method [Liu et al.,2005], the MRF method [Tang and Wang, 2009], the MWFmethod [Zhou et al., 2012], the Bayesian method [Wang etal., 2017] and the FCN method [Zhang et al., 2015]. All syn-thesized sketches by the MWF method and Bayesian Methodare generated from the source codes provided by the authors.For the MRF method, we use the codes from the implementa-tion provided by authors of [Song et al., 2014]. Results of theLLE method and FCN method is based on our implementa-tions. All experiments are conducted using Python on Ubuntu14.04 system with i7-4790 3.6G CPU and 12G NVIDIA Ti-tan X GPU. GPU was only used in the deep feature extractionstage. The deep feature extraction of training photos can beconducted offline. With the help of GPU, deep feature ex-traction of a input test photo can be finished less than 1s. So,the most time-consuming part of our proposed method liesin two phases: 1) the neighbor selection phase; and 2) theoptimization phase. The neighbor selection phase dependson the number of training photos and the size of search re-gions. More training photos and larger search regions wouldincrease in time-consuming. We use kd-tree [Beis and Lowe,1997] to conduct K-neighbor searches in order to improve ef-ficiency. The optimization phase mainly depends on the num-ber of iterations. In our experiments, it always converges afterfive to eight iterations.

4.2 Face Sketch SynthesisFigure 4 shows some synthesized face sketches from differ-ent methods on the CUFS database. As we can see, resultsfrom other methods contain significant amount of noise onthe nose, mouth and the hair region while the proposed DGFLmethod achieves much better performance. We have further

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3578

validated the performance of the proposed DGFL method onface photos with extreme lighting variance, as shown in Fig-ure 5. When the test photos is affected by extreme environ-mental noises, such as strong lighting variance, other meth-ods failed to discern the noises and thus produced distortionresults. Deep representations is more robust to these noisesthan pixel intensity, therefore the proposed DGFL methodcould synthesize sketches with virtually no distortion.

4.3 Objective Image Quality AssessmentWe utilize structural similarity index metric (SSIM) [Wanget al., 2004] to evaluate the quality of synthesized sketchesby different methods on CUFS. There are 338 (100 + 43 +195) synthesized sketches for each method generated fromthe CUFS database. Figure 6 gives the statistics of SSIMscores on the databases. The horizontal axis labels repre-sent the SSIM score from 0 to 1. The vertical axis means thepercentage of synthesized sketch whose SSIM scores are notsmaller than the score marked on the horizontal axis. Table Igives the average SSIM score on the CUFS database. It canbe seen from Figure 6 and Table I that the proposed DGFLmethod outperform five other state-of-the-art methods.

0.2 0.3 0.4 0.5 0.6 0.7 0.8

SSIM Score

0

10

20

30

40

50

60

70

80

90

100

Pe

rce

nta

ge

(%

)

SSIM Score on CUFS

LLE

MRF

MWF

FCN

Bayesian

DGFL

Figure 6: Statistics of SSIM scores on the CUFS database.

4.4 Face RecognitionSketch based face recognition is always used to assist lawenforcement. Given a sketch drawn by the artist, we can re-trieve the corresponding photo in the mug shots based on itssynthesized sketch. Null-space linear discriminant analysis(NLDA) [Chen et al., 2000] is employed to conduct the facerecognition experiments. As aforementioned, we have 338synthesized sketches for each face synthesis method. We ran-domly choose 150 synthesized sketches and their correspond-ing sketches drawn by the artist for classifier training and therest 188 synthesized sketches are as the gallery set. We re-peat each face recognition experiment 20 times by randomlypartition the data. Figure 7 gives the face recognition accu-racy against variations of the number of reduced dimensionsby NLDA on the CUFS database. It can be seen that on theCUFS database, the proposed DGFL method has the highestrecognition rate. This accuracy can be further improved byusing some advanced face recognition methods [Lei et al.,

2014] and heterogeneous face recognition methods [Klare etal., 2011].

20 40 60 80 100 120 140

The reduced number of dimensions

0

10

20

30

40

50

60

70

80

90

100

Re

co

gn

itio

n r

ate

(%

)

NLDA on CUFS

LLE

MRF

MWF

FCN

Bayesian

DGFL

Figure 7: Face recognition accuracy against variations ofthe number of reduced dimensions by NLDA on the CUFSdatabase.

5 ConclusionIn this paper, we presented a deep feature learning frameworkfor face sketch synthesis. Deep discriminative representa-tions derived from the deep Convolutional Neural Networksare used to represent face photos, which performs morerobust respect to the noises than pixel intensity. In orderto jointly model the distribution of the weights for deeprepresentations and the distribution of the weights for sketchreconstruction, we present a deep feature learning frame-work based on the graph model and adopt an alternatingoptimization strategy to solve this model. Optimal weightcombination for sketch reconstruction can be obtained afterconvergence. Quantitative and qualitative results showed thatthe proposed method outperforms existing state-of-the-artmethods, especially under the noise environment. In thefuture, we would further explore the discrimination abilityof the optimal deep representations and apply it to theidentification problem.

AcknowledgmentsThis work was supported in part by the National Natu-ral Science Foundation of China (under Grant 61671339,61501339, 61432014, U1605252, and 61601158), in partby the Fundamental Research Funds for the Central Uni-versities under Grant JB160104, in part by the Programfor Changjiang Scholars, in part by the Leading Talent ofTechnological Innovation of Ten-Thousands Talents Programunder Grant CS31117200001, in part by the China Post-Doctoral Science Foundation under Grant 2015M580818 andGrant 2016T90893, and in part by the Shaanxi Province Post-Doctoral Science Foundation.

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3579

References[Beis and Lowe, 1997] Jeffrey S. Beis and David G. Lowe.

Shape indexing using approximate nearest-neighboursearch in high-dimensional spaces. In Proceedings ofIEEE Computer Society Conference on Computer Visionand Pattern Recognition, pages 1000–1006, 1997.

[Chen et al., 2000] Li-Fen Chen, Hong-Yuan Mark Liao,Ming-Tat Ko, Ja-Chen Lin, and Gwo-Jong Yu. A new lda-based face recognition system which can solve the smallsample size problem. Pattern Recognition, 33(10):1713–1726, 2000.

[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scalehierarchical image database. In Proceedings of IEEEConference on Computer Vision and Pattern Recognition,2009.

[Gao et al., 2012] Xinbo Gao, Nannan Wang, Dacheng Tao,and Xuelong Li. Face sketch-photo synthesis and retrievalusing sparse representation. IEEE Transactions on Cir-cuits and Systems for Video Technology, 22(8):1213–1226,2012.

[Gatys et al., 2016] Leon A. Gatys, Alexander S. Ecker, andMatthias Bethge. Image style transfer using convolutionalneural networks. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, pages 2414–2423, 2016.

[Goodfellow et al., 2014] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen-erative adversarial nets. In Advances In Neural Informa-tion Processing systems, pages 2672–2680, 2014.

[K. Messer and J. Luettin, 1999] J. Kittler K. Messer,J. Matas and G. Maitre J. Luettin. Xm2vtsdb: theextended m2vts database. In Proceedings of the Interna-tional Conference on Audio- and Video-Based BiometricPerson Authentication, pages 72–77, 1999.

[Klare et al., 2011] Brendan Klare, Zhifeng Li, and Anil K.Jain. On matching forensic sketches to mugshot photos.IEEE Transactions on Pattern Analysis and Machine In-telligence, 33(3):639–646, 2011.

[Lei et al., 2014] Zhen Lei, Matti Pietikainen, and Stan Z. Li.Learning discriminant face descriptor. IEEE Transactionson Pattern Analysis and Machine Intelligence, 36(2):289–302, 2014.

[Liang Chang et al., 2010] Mingquan Zhou Liang Chang,Yanjun Han, and Xiaoming Deng. Face sketch synthe-sis via sparse representation. In Proc. 20th Int. Conf. Pat-tern Recognit, pages 2146–2149, Istanbul, Turkey, August2010.

[Liu et al., 2005] Qingshan Liu, Xiaoou Tang, HongliangJin, Hanqing Lu, and Songde Ma. A nonlinear approachfor face sketch synthesis and recognition. In Proceed-ings of IEEE Conference on Computer Vision and PatternRecognition, pages 1005–1010, 2005.

[Liu et al., 2007] Wei Liu, Xiaoou Tang, and JianzhuangLiu. Bayesian tensor inference for sketch-based facialphoto hallucination. In Proceedings of International JointConference on Artificial Intelligence, pages 2141–2146,2007.

[Martinez and Benavente, 1998] Aleix Martinez and RobertBenavente. The ar face database. In CVC, Barcelona,Spain, 1998.

[Simonyan and Zisserman, 2014] Karen Simonyan and An-drew Zisserman. Very deep convolutional networks forlarge-scale image recognition. CoRR, abs/1409.1556,2014.

[Song et al., 2014] Yibing Song, Linchao Bao, QingxiongYang, and Ming-Hsuan Yang. Real-time exemplar-basedface sketch synthesis. In Proceedings of Eureopean Con-ference on Computer Vision, pages 800–813, 2014.

[Tang and Wang, 2002] Xiaogang Tang and Xiaoou Wang.Face photo recognition using sketch. In Proceedingsof IEEE International Conference on Image Processing,pages 257–260, 2002.

[Tang and Wang, 2003] Xiaoou Tang and Xiaogang Wang.Face sketch synthesis and recognition. In Proceedings ofIEEE International Conference on Computer Vision, pages687–694, 2003.

[Tang and Wang, 2009] Xiaoou Tang and Xiaogang Wang.Face photo-sketch synthesis and recognition. IEEE Trans-actions on Pattern Analysis and Machine Intelligence,31(11):1955–1967, November 2009.

[Wang et al., 2004] Zhou Wang, A.C. Bovik, H.R. Sheikh,and E.P. Simoncelli. Image quality assessment: from errorvisibility to structural similarity. IEEE Transactions onImage Processing, 13(4):600–612, 2004.

[Wang et al., 2014] Nannan Wang, Dacheng Tao, XinboGao, Xuelong Li, and Jie Li. A comprehensive surveyto face hallucination. International Journal of ComputerVision, 106(1):9–30, January 2014.

[Wang et al., 2017] Nannan Wang, Xinbo Gao, Leiyu Sun,and Jie Li. Bayesian face sketch synthesis. IEEE Transac-tions on Image Processing, PP:1–11, January 2017.

[Zhang et al., 2015] Liliang Zhang, Liang Lin, Xian Wu,Shengyong Ding, and Lei Zhang. End-to-end photo-sketchgeneration via fully convolutional representation learning.In Proceedings of the 5th ACM on International Confer-ence on Multimedia Retrieval, pages 627–634, 2015.

[Zhou et al., 2012] Hao Zhou, Zhanghui Kuang, and Kwan-Yee K. Wong. Markov weight fields for face sketch syn-thesis. In Proceedings of IEEE Conference on ComputerVision and Pattern Recognition, pages 1091–1097, 2012.

[Zhu and Wang, 2016] Mingrui Zhu and Nannan Wang. Asimple and fast method for face sketch synthesis. In In-ternational Conference on Internet Multimedia Computingand Service, pages 168–171, 2016.

Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17)

3580