Post on 27-Mar-2015
A Two-Step Approach to Hallucinating Faces:Global Parametric Model and
Local Nonparametric Model
Ce Liu Heung-Yeung Shum Chang Shui Zhang
CVPR 2001
Face hallucination
Face Hallucination — to infer high resolution face image from low resolution input
(a) Input 24×32 (a) Input 24×32 (b) Hallucinated result(b) Hallucinated result (c) Original 96×128(c) Original 96×128
Why to study face hallucination?
Applications Video conference
To use very low band to transmit face image sequence To repair damaged images in transmission
Face image recovery To recover low-quality faces in old photos To recover low-resolution monitoring videos
Research Information recovery
How to formulate and learn prior knowledge of face How to apply face prior to infer the lost high frequency details
Super resolution How to model the bridge from low-resolution to high-resolution
Difficulties and solution strategy
Difficulties Sanity Constraint
The result must be close to the input image when smoothed and down-sampled
Global ConstraintThe result must have common characteristics of a human face, e.g., eyes, mouth, nose and symmetry
Local ConstraintThe result must have specific characteristics of this face image, with photorealistic local features
Solution strategyWe choose learning based method aided by a large set of various face images to hallucinate face
Previous learning-based super-resolution methods
Multi-resolution texture synthesis
De Bonet. SIGGRAPH 1997
Markov network
Freeman and Pasztor. ICCV 1999
Face hallucination
Baker and Kanade. AFGR 2000, CVPR 2000
Image analogies
Hertzmann, Jacobs, Oliver, Curless and Salesin. SIGGRAPH 2001
They all use local feature transfer or inference in Markov random field, without any global correspondence taken into account.
Decouple high-resolution face image to two parts
— high resolution face image — global face — local face
Two-step Bayesian inference1. Inferring global face
2. Inferring local face
Finally adding them together
Our method
gHI l
HIHIHI g
HI lHI
)|()()|(maxarg
),(),|(maxarg
)()|(maxarg
,
,
*
gH
lH
gH
gHL
II
lH
gH
lH
gHL
II
HHLI
H
IIpIpIIp
IIpIIIp
IpIIpI
l
H
g
H
l
H
g
H
H
1. Inferring global face
2. Inferring local face
Finally adding them together
)()|(maxarg*
* gH
gHL
I
gH IpIIpI
gH
)|(maxarg **
*
gH
lH
I
lH IIpI
lH
*** gH
lHH III
= +
HI
LI
?
Flowchart of hallucinating face
Learning Process
Inference Process
Training dataset
Global faces Local faces
Learning(a) Learn the prior of global face by PCA(b) Build Markov network between global and local faces
Inference(c) Infer global face by linear regression(d) Infer local face by Markov network
(c) (d)
(a)
(b)
Input Output
Inferring global face
PriorAssume the prior of global face to be Gaussian and learn it by PCA. The global face is the principal components of the high-resolution face image.
(Many other methods such as Gaussian mixture, ICA, kernel PCA, TCA can be used to model the face prior. We choose PCA because it could get simple solution)
Likelihood
Treat low resolution input as a soft constraint to the global face. The likelihood turns out to be a Gaussian distribution again.
Posteriori
The energy of the posteriori has a quadratic form. The MAP solution is converted to linear regression by SVD.
How to compute global face
Prior Distributionof Global Face
Likelihood from InputLow Resolution Face
MAP Solution ofthe Global Face Posteriori of Global
Face Given Input
Local face is pursued by minimizing the energy of Markov network
Two terms of energies:
external potential — to model the connective statistics between two linked patches in and .
internal potential — to make adjacent patches in well connected.
Energy minimization by simulated annealing
Inferring local face by Markov network
gHI
)(vI gH
)(vN lH
lHI
)(vS
)(vI lH
An inhomogeneous patch-based nonparametric Markov network
lHI
lHI
gHI
Experimental results (1)
(a)
(b)
(c)
(d)
(a) Input low 24×32
(b) Inferred global face
(c) Hallucinated result
(d) Original high 96×128
Experimental results (2)
Experimental results (3)
Comparison with other methods
(a)
(b)
(c)
(d)
(e)
(f)
(a) Input
(b) Hallucinated by our method
(c) Cubic B-spline
(d) Hertzmann et al.
(e) Baker et al.
(f) Original
Summary
• Hybrid modeling of face (global plus local) Global: the major information of face, lying in middle and low frequency band Local: the residue between real data and global model, lying in high frequency band
• The sanity constraint is added to the global part
• The global face is modeled by PCA and inferred by linear regression
• The conditional distribution of the local face given the global face is modeled upon a patch-based nonparametric Markov network, and inferred by energy minimization
• Both of the two steps in inference are global optimal Global part: optimizing a quadratic energy function by SVD Local part: optimizing the network energy by simulated annealing
Thank you!