Inter-modality Face Sketch Recognition

Hamed Kiani

2

Outline

• Overview• Previous Works• Proposed Approach• Results• Summary

Inter-modality Face Sketch Recognition ICME'12

Inter-modality Face Sketch Recognition ICME'12

3

Overview

• Face Recognition

Known Face Images

Face Recognition

System

Identity

Input Face

Inter-modality Face Sketch Recognition ICME'12

4

Overview• Face sketch recognition

Known Face Photos

(Mug shot)

Photo-Sketch

Matching

Suspect’s identity

Viewing

Verbal description Drawin

g

Eyewitness

Police artist

Sketch

Inter-modality Face Sketch Recognition ICME'12

5

Overview

–Modality Gap: the difference of visual cues between face sketch and photo.

Intra-modality approaches

Inter-modality approaches

Matching face photos and sketches in a same modality by (photo or sketch) Synthesis

Matching photo and sketch of different modalities (direct matching).

Tang and Wang [ECCV’03] Liu et al. [CVPR’5]

Wang and Tang [PAMI’09]

Klare and Jain [SPIE ‘10]Klare et al. [PAMI’11]

Zhang et al. [CVPR’11]

Image synthesisNo modality gap

Modality gap No image synthesis

Inter-modality Face Sketch Recognition ICME'12

6

Overview

• Visual cues of face come from:– Fine texture (appearance):low contrast details, flaws, moles, wrinkles , etc.–Coarse texture (shape):high contrast boundaries of facial

components eyes, mouth, etc

Inter-modality Face Sketch Recognition ICME'12

7

Overview

• Face textures and modality gap: Fine textures of a face photo captured by camera

(true pixels) Fine texture of a sketch is rendered by artist,

depending on drawing style and tools Fine textures of face photo and sketch are not

equivalent: high amount of modality gap Coarse texture (facial component and

boundaries) exists in both sketch and photo modality gap is not affected significantly by

coarse texture

Inter-modality Face Sketch Recognition ICME'12

8

Proposed Approach

• Histogram of Averaged Oriented Gradients (HAOG): a modified version of Histogram of Oriented Gradients (HOG)

• HOG for sketch recognition: Modeling local appearance and shapeBased on fine and coarse textures. “Fine texture leads to a high amount of modality gap”

Inter-modality Face Sketch Recognition ICME'12

9

Proposed Approach

• Idea of HAOG:Emphasizing coarse texture much more than fine texture in feature extraction.

• How?By averaged gradient vector (dominant gradient) instead of pixel’s gradient vector (orientation and magnitude).

Inter-modality Face Sketch Recognition ICME'12

10

Proposed Approach

• But: Local gradients cannot directly be averaged, opposite gradient vectors cancel each other

• Solution: Doubling the angles of the gradient vectors before averaging: equal to squaring the length of gradient vectors [Bazen and Grez, 2002].

Inter-modality Face Sketch Recognition ICME'12

11

Proposed Approach

• Thus, we define squared gradient vectors

Inter-modality Face Sketch Recognition ICME'12

12

Proposed Approach

• HAOG x-gradient

y-gradient

Inter-modality Face Sketch Recognition ICME'12

13

Proposed Approach• HAOG

HAOG

Inter-modality Face Sketch Recognition ICME'12

14

Proposed Approach• Given a query sketch and a gallery of face

photos , face sketch recognition is done by:

: HAOG descriptor , :chi-square

Inter-modality Face Sketch Recognition ICME'12

15

Proposed Approach

Figure 1. (a1) Face photo, (a2) Face sketch, (b1,b2) Gradient magnitudes of (a1,a2), Squared gradient magnitudes of

(a1,a2).

Inter-modality Face Sketch Recognition ICME'12

16

Proposed Approach

Figure 2. Face sketch (top), photo (bottom), (b,c,d) local patches (first row), HAOG descriptors (second row) and HOG descriptors

(third row).

Inter-modality Face Sketch Recognition ICME'12

17

Results

• Results on CUHK dataset with 606 pairs of face photo/sketch

Inter-modality Face Sketch Recognition ICME'12

18

Summary

• Face sketch recognition vs. face recognition

• Modality gap• HOG vs. HAOG• Future work

Inter-modality Face Sketch Recognition ICME'12

19