Face Recognition:An Introduction

Face Recognition

• Face is the most common biometric used by humans

• Applications range from static, mug-shot verification to

a dynamic, uncontrolled face identification in a cluttered

background

• Challenges:

• automatically locate the face

• recognize the face from a general view point under

different illumination conditions, facial expressions,

and aging effects

Authentication vs Identification• Face Authentication/Verification (1:1 matching)

• Face Identification/Recognition (1:N matching)

www.viisage.com

Access Control

Applications

www.visionics.com

Video Surveillance (On-line or off-line)

Applications

Face Scan at Airports

www.facesnap.de

Face Recognition Application

Why is Face Recognition Hard?

Many Faces of the same person

Face Recognition Difficulties

• Identify similar faces (inter-class similarity)

• Accommodate intra-class variability due to:

• head pose

• illumination conditions

• expressions

• facial accessories

• aging effects

• Cartoon faces

Inter-class Similarity

• Different persons may have very similar appearance

Twins Father and son

www.marykateandashley.com news.bbc.co.uk/hi/english/in_depth/americas/2000/us_elections

Intra-class Variability• Faces with intra-subject variations in pose, illumination,

expression, accessories, color, occlusions, and brightness

Sketch of a Pattern Recognition Architecture

FeatureExtraction

ClassificationImage

(window)ObjectIdentityFeature

Vector

Example: Face Detection

• Scan window over image

• Classify window as either:– Face– Non-face

ClassifierWindowFace

Non-face

Example: Finding skin

• Skin has a very small range of (intensity independent) colors, and little texture– Compute an intensity-independent color measure, check if

color is in this range, check if there is little texture (median filter)

– See this as a classifier - we can set up the tests by hand, or learn them.

– get class conditional densities (histograms), priors from data (counting)

• Classifier is

Face Detection

Face Detection Algorithm

Face Localization

Lighting Compensation

Skin Color Detection

Color Space Transformation

Variance-based Segmentation

Connected Component &Grouping

Face Boundary Detection

Verifying/ WeightingEyes-Mouth Triangles

Eye/ Mouth Detection

Facial Feature Detection

Input Image

Output Image

Face Recognition: 2-D and 3-D

2-D

Face Database

Time(video)

2-D

RecognitionData

3-D 3-D

RecognitionComparison

Prior knowledgeof face class

Pose-dependent

Algorithms

Pose-invariant

Pose-dependency

Matching features

Appearance-based (Holistic)

Feature-based (Analytic)

Hybrid

Viewer-centered Images

Object-centered Models

Face representation

Taxonomy of Face Recognition

Image as a Feature Vector

• Consider an n-pixel image to be a point in an n-dimensional space, x Rn.

• Each pixel value is a coordinate of x.

x1

x2

x3

Nearest Neighbor Classifier { { RRjj } } are set of training images.

x1

x2

x3RR11

RR22

II

),(minarg IRdistID jj

Comments• Sometimes called “Template Matching”• Variations on distance function (e.g. L1,

robust distances)• Multiple templates per class- perhaps many

training images per class.• Expensive to compute k distances, especially

when each image is big (N dimensional).• May not generalize well to unseen examples

of class.• Some solutions:

– Bayesian classification– Dimensionality reduction

EigenfacesEigenfacesA set of eigenfaces can be generated by performing a mathematical process called principal component analysis (PCA) on a large set of images depicting different human faces.

Informally, eigenfaces can be considered a set of "standardized face ingredients", derived from statistical analysis )(covariance matrix ) of many pictures of faces. Any human face can be considered to be a combination of these standard faces.

EigenfacesEigenfacesFor example, one's face might be composed of the average face plus 10% from eigenface 1, 55% from eigenface 2, and even -3% from eigenface 3.

The eigenfaces that are created will appear as light and dark areas that are arranged in a specific pattern.

EigenfacesEigenfacesThese eigenfaces can now be used to represent both existing and new faces: we can project a new (mean-subtracted) image on the eigenfaces and thereby record how that new face differs from the mean face.

The eigenvalues associated with each eigenface represent how much the images in the training set vary from the mean image in that direction.

EigenfacesEigenfacesWe lose information by projecting the image on a subset of the eigenvectors, but we minimize this loss by keeping those eigenfaces with the largest eigenvalues.

For instance, if we are working with a 100 x 100 image, then we will obtain 10,000 eigenvectors. In practical applications, most faces can typically be identified using a projection on between 100 and 150 eigenfaces, so that most of the 10,000 eigenvectors can be discarded.

How do you construct Eigenspace?

[ ] [ ][ x1 x2 x3 x4 x5 ] W

Construct data matrix by stacking vectorized images and then apply Singular Value Decomposition (SVD)

EigenfacesEigenfaces

Eigenfaces• Modeling

1. Given a collection of n labeled training images,2. Compute mean image and covariance matrix.3. Compute k Eigenvectors (note that these are

images) of covariance matrix corresponding to k largest Eigenvalues.

4. Project the training images to the k-dimensional Eigenspace.

• Recognition1. Given a test image, project to Eigenspace.2. Perform classification to the projected training

images.

Eigenfaces: Training Images

Eigenfaces

Mean Image Basis Images

Difficulties with PCA

• Projection may suppress important detail– smallest variance directions may not be

unimportant

• Method does not take discriminative task into account– typically, we wish to compute features that allow

good discrimination– not the same as largest variance

Fisherfaces: Class specific linear projection

• An n-pixel image xRn can be projected to a low-dimensional feature space yRm by

y = Wx

where W is an n by m matrix.

• Recognition is performed using nearest neighbor in Rm.

• How do we choose a good W?

PCA & Fisher’s Linear Discriminant

• Between-class scatter

• Within-class scatter

• Total scatter

• Where– c is the number of classes

i is the mean of class i

– | i | is number of samples of

i..

Tii

c

iiBS ))((

1

c

i x

TikikW

ik

xS1

))((

WB

c

i x

TkkT SSxS

ik

1

))((

1

2

12

PCA & Fisher’s Linear Discriminant

• PCA (Eigenfaces)

Maximizes projected total scatter

• Fisher’s Linear Discriminant

Maximizes ratio of projected between-class to projected within-class scatter

WSWW TT

WPCA maxarg

WSW

WSWW

WT

BT

Wfld maxarg

12

PCA

FLD

Four Fisherfaces Database

Eigenfaces and Fisherfaces