A Two-Step Approach to Hallucinating Faces:Global Parametric Model and

Local Nonparametric Model

Ce Liu Heung-Yeung Shum Chang Shui Zhang

CVPR 2001

Face hallucination

Face Hallucination — to infer high resolution face image from low resolution input

(a) Input 24×32 (a) Input 24×32 (b) Hallucinated result(b) Hallucinated result (c) Original 96×128(c) Original 96×128

Why to study face hallucination?

Applications Video conference

To use very low band to transmit face image sequence To repair damaged images in transmission

Face image recovery To recover low-quality faces in old photos To recover low-resolution monitoring videos

Research Information recovery

How to formulate and learn prior knowledge of face How to apply face prior to infer the lost high frequency details

Super resolution How to model the bridge from low-resolution to high-resolution

Difficulties and solution strategy

Difficulties Sanity Constraint

The result must be close to the input image when smoothed and down-sampled Global Constraint

The result must have common characteristics of a human face, e.g., eyes, mouth, nose and symmetry

Local ConstraintThe result must have specific characteristics of this face image, with photorealistic local features

Solution strategyWe choose learning based method aided by a large set of various face images to hallucinate face

Previous learning-based super-resolution methods

Multi-resolution texture synthesisDe Bonet. SIGGRAPH 1997

Markov networkFreeman and Pasztor. ICCV 1999

Face hallucinationBaker and Kanade. AFGR 2000, CVPR 2000

Image analogiesHertzmann, Jacobs, Oliver, Curless and Salesin. SIGGRAPH 2001

They all use local feature transfer or inference in Markov random field, without any global correspondence taken into account.

Decouple high-resolution face image to two parts

— high resolution face image — global face — local face Two-step Bayesian inference

1. Inferring global face

2. Inferring local face

Finally adding them together

Our method

gHI l

HIHIHI g

HI lHI

)|()()|(maxarg

),(),|(maxarg

)()|(maxarg

,

,

*

gH

lH

gH

gHL

II

lH

gH

lH

gHL

II

HHLI

H

IIpIpIIp

IIpIIIp

IpIIpI

lH

gH

lH

gH

H

1. Inferring global face

2. Inferring local face

Finally adding them together

)()|(maxarg*

* gH

gHL

I

gH IpIIpI

gH

)|(maxarg ***

gH

lH

I

lH IIpI

lH

*** gH

lHH III

= +

HI

LI

?

Flowchart of hallucinating face

Learning Process

Inference Process

Training dataset

Global faces Local faces

Learning(a) Learn the prior of global face by PCA(b) Build Markov network between global and local faces Inference(c) Infer global face by linear regression(d) Infer local face by Markov network

(c) (d)

(a)

(b)

Input Output

Inferring global face

PriorAssume the prior of global face to be Gaussian and learn it by PCA. The global face is the principal components of the high-resolution face image.(Many other methods such as Gaussian mixture, ICA, kernel PCA, TCA can be used to model the face prior. We choose PCA because it could get simple solution)

LikelihoodTreat low resolution input as a soft constraint to the global face. The likelihood turns out to be a Gaussian distribution again.

Posteriori The energy of the posteriori has a quadratic form. The MAP solution is converted to linear regression by SVD.

How to compute global face

Prior Distributionof Global Face

Likelihood from InputLow Resolution Face

MAP Solution ofthe Global Face Posteriori of Global

Face Given Input

Local face is pursued by minimizing the energy of Markov network

Two terms of energies:

external potential — to model the connective statistics between two linked patches in and .

internal potential — to make adjacent patches in well connected.

Energy minimization by simulated annealing

Inferring local face by Markov network

gHI

)(vI gH

)(vN lH

lHI

)(vS

)(vI lH

An inhomogeneous patch-based nonparametric Markov network

lHI

lHI

gHI

Experimental results (1)

(a)

(b)

(c)

(d)

(a) Input low 24×32

(b) Inferred global face

(c) Hallucinated result

(d) Original high 96×128

Experimental results (2)

Experimental results (3)

Comparison with other methods

(a)

(b)

(c)

(d)

(e)

(f)

(a) Input

(b) Hallucinated by our method

(c) Cubic B-spline

(d) Hertzmann et al.

(e) Baker et al.

(f) Original

Summary

• Hybrid modeling of face (global plus local) Global: the major information of face, lying in middle and low frequency band Local: the residue between real data and global model, lying in high frequency band

• The sanity constraint is added to the global part• The global face is modeled by PCA and inferred by linear regression• The conditional distribution of the local face given the global face is

modeled upon a patch-based nonparametric Markov network, and inferred by energy minimization

• Both of the two steps in inference are global optimal Global part: optimizing a quadratic energy function by SVD Local part: optimizing the network energy by simulated annealing

Thank you!