Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face...
Transcript of Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face...
![Page 1: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/1.jpg)
Review: Image warping and morphing• Forward vs. inverse warping
![Page 2: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/2.jpg)
f(x,y) g(x’,y’)
Forward warping
Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image
x x’
T(x,y)
Q: what if pixel lands “between” two pixels?
y y’
Slide credit: S. Seitz
![Page 3: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/3.jpg)
f(x,y) g(x’,y’)
Forward warping
Send each pixel f(x,y) to its corresponding location (x’,y’) = T(x,y) in the second image
x x’
T(x,y)
Q: what if pixel lands “between” two pixels?
y y’
A: distribute color among neighboring pixels (x’,y’)– Known as “splatting”– Check out griddata in Matlab Slide credit: S. Seitz
![Page 4: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/4.jpg)
f(x,y) g(x’,y’)xy
Inverse warping
Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image
x x’
Q: what if pixel comes from “between” two pixels?
y’T-1(x,y)
Slide credit: S. Seitz
![Page 5: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/5.jpg)
f(x,y) g(x’,y’)xy
Inverse warping
Get each pixel g(x’,y’) from its corresponding location (x,y) = T-1(x’,y’) in the first image
x x’
T-1(x,y)
Q: what if pixel comes from “between” two pixels?
y’
A: Interpolate color value from neighbors– nearest neighbor, bilinear, Gaussian, bicubic– Check out interp2 in Matlab Slide credit: S. Seitz
![Page 6: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/6.jpg)
Forward vs. inverse warpingQ: which is better?
A: usually inverse—eliminates holes• however, it requires an invertible warp function—not always possible...
Slide credit: S. Seitz
![Page 7: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/7.jpg)
Review: Image morphing
Michael Jackson’s “Black or White” (1991)
![Page 8: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/8.jpg)
How do we create a morphing sequence?
1. Define corresponding control points on both images2. For each intermediate image:
1. Create an intermediate shape (by interpolation)2. Warp both images towards it3. Cross-dissolve the colors between the newly warped
images
![Page 9: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/9.jpg)
morphing
Morphing example
image #1 image #2
warp warp
![Page 10: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/10.jpg)
Morphing example
![Page 11: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/11.jpg)
Fun with faces
![Page 12: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/12.jpg)
How do we average faces?
http://www2.imm.dtu.dk/~aam/datasets/datasets.html
![Page 13: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/13.jpg)
Sir Francis Galton: Composite Portraits
Composite portraits showing "features common among men convicted of crimes of violence," by Francis Galton, 1885
![Page 14: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/14.jpg)
Sir Francis Galton: Composite portraits
“The Jewish Type”
![Page 15: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/15.jpg)
Sir Francis Galton: Composite portraits
“The Jewish Type”
![Page 16: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/16.jpg)
Jason Salavon: “The Late Night Triad”
http://www.salavon.com/
![Page 17: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/17.jpg)
Jason Salavon: “100 Special Moments”
![Page 18: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/18.jpg)
Jason Salavon: “Every Playboy Centerfold”
![Page 19: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/19.jpg)
Is there a better way to average faces?
![Page 20: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/20.jpg)
Average Face
1. Warp to mean shape2. Average pixels
http://graphics.cs.cmu.edu/courses/15-463/2004_fall/www/handins/brh/final/ Slide credit: A. Efros
![Page 21: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/21.jpg)
Representing the shape of a face
Slide credit: A. Efros
![Page 22: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/22.jpg)
Representing the shape of a face
Vector of normalized landmark coordinates (or inter-landmark distances)
Slide credit: A. Efros
![Page 23: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/23.jpg)
Subpopulation means
• 250 control points per face manually specified
Average male(32 faces)
Average female(64 faces)
http://www.beautycheck.de
![Page 24: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/24.jpg)
Manipulating faces
• How can we make a face look more male/female, young/old, happy/sad, etc.?1. Obtain the two prototypes spanning the desired axis
(gender, age, expression, etc.) and find the difference vector between them
2. Add scaled versions of this vector to a given image
Prototype 1
Prototype 2Current face
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
![Page 25: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/25.jpg)
Changing gender
Deform shape and/or color of an input face in the direction of “more female”
original shape
color both
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
![Page 26: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/26.jpg)
Enhancing gender
more same original androgynous more opposite
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
![Page 27: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/27.jpg)
Changing age
Face becomes “rounder” and “more textured” and “grayer” original shape
color both
D. Rowland and D. Perrett, “Manipulating Facial Appearance through Shape and Color,” IEEE CG&A, September 1995
Slide credit: A. Efros
![Page 28: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/28.jpg)
Which face is more attractive?
http://www.beautycheck.de
![Page 29: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/29.jpg)
Which face is more attractive?
attractive0.5(attractive + average)
![Page 30: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/30.jpg)
Which face is more attractive?
http://www.beautycheck.de
![Page 31: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/31.jpg)
Which face is more attractive?
0.5(adult+child) adult
![Page 32: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/32.jpg)
Effect of skin texture
average shape + unattractive texture
average shape + average texture
![Page 33: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/33.jpg)
Digital beautification – Video http://www.youtube.com/watch?v=iYhCn0jf46U
![Page 34: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/34.jpg)
Data-Driven Enhancement of Facial Attractiveness
T. Leyvand, D. Cohen-Or, G. Dror, and D. Lischinski
SIGGRAPH 2008
![Page 35: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/35.jpg)
System Overview
![Page 36: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/36.jpg)
Feature extraction• Active shape model (Cootes et al. 1995)
• Represents distribution of shape (facial features represented by splines) and appearance (pattern of local gradients)
• Initialized from manually labeled training data• At test time, need good initial estimate of face location, scale
and orientation – use a face detector• 84 keypoints are converted into a planar graph of 234 inter-
point distances
Average and scatterfrom training set
Results of ASM search
Graph of distances
![Page 37: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/37.jpg)
Regression of facial attractiveness model• Input: training set of distance vectors of faces rated
on a scale of 1 to 7 by human subjects (Eisenthal et al. 2006)• 92 female and 33 male photographs
• Support vector regression: learn smooth function f(v) to estimate attractiveness ranking
![Page 38: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/38.jpg)
Beautification• Problem: given a vector v and estimated attractiveness
rating f(v), find nearby vector v’ such that f(v’) > f(v)• Method 1 (KNN): move the face vector in the direction of the
beauty-weighted average of K nearest neighbors of that face:
• Method 2 (SVR): perform gradient ascent of f starting at v• Results
• NN-based beautification worked better on male faces, but SVR-based beautification worked better on female faces
• On average, estimated beauty score was increased by 30%
,beauty score of the ith neighbor
![Page 39: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/39.jpg)
Choosing K in KNN beautification
![Page 40: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/40.jpg)
Comparison of KNN and SVR beautification
![Page 41: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/41.jpg)
Additional details• Distance embedding
• Given a vector of distances v’ = {dij}, find a set of points {qi} that conform to these distances:
• Avoiding pupil distortion• Add post-process to enforce similarity transform for feature
points on the eyes
• Warping: Multilevel Free-Form Deformation (Lee et al. 1996)• Uses B-splines, avoids fold-overs
![Page 42: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/42.jpg)
Interactive system: Video
![Page 43: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/43.jpg)
Which face is more attractive?
![Page 44: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/44.jpg)
Which face is more attractive?
originalbeautified
![Page 45: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/45.jpg)
Which face is more attractive?
![Page 46: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/46.jpg)
Which face is more attractive?
original beautified
![Page 47: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/47.jpg)
Which face is more attractive?
![Page 48: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/48.jpg)
Which face is more attractive?
original beautified
![Page 49: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/49.jpg)
Which face is more attractive?
![Page 50: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/50.jpg)
Which face is more attractive?
beautified original
![Page 51: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/51.jpg)
Which face is more attractive?
![Page 52: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/52.jpg)
Which face is more attractive?
original beautified
![Page 53: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/53.jpg)
Which face is more attractive?
![Page 54: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/54.jpg)
Which face is more attractive?
original beautified
![Page 55: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/55.jpg)
Which face is more attractive?
![Page 56: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/56.jpg)
Which face is more attractive?
beautified original
![Page 57: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/57.jpg)
Results of user study• 92 face pairs (47 female, 45 male)• 68 raters, males and females aged between 25 and 40
• “For all 47 portraits of females, the beautified versions were chosen as more attractive by most raters, and in half of the cases the beautified versions were preferred by more than 80 percent of the raters. As for the male portraits, 69 percent of the beautified versions were chosen as more attractive.”
![Page 58: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/58.jpg)
Application: Construction of facial collages
initial collage Poisson blending beautification
![Page 59: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/59.jpg)
Face Swapping: Automatically Replacing Faces in Photographs
D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, S. K. Nayar
SIGGRAPH 2008
![Page 60: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/60.jpg)
Outline• Offline: create face library• Given a new query:
• Detect faces• For each face:
– Retrieve candidate replacement faces from the library– Apply recoloring and relighting– Perform face replacement– Rank replacement results
![Page 61: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/61.jpg)
Creating a face library• Download images from the Internet and process them
using a commercial face detector• Approximately 33,000 faces found• Detector returns face pose (yaw, pitch, roll) and six fiducial points• Bin faces according to yaw and pitch angles, align to a common
coordinate system using fiducial points
![Page 62: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/62.jpg)
Candidate selection• Detect faces in new image, estimate their pose
angles and fiducial points• Matching criteria:
• Pose: yaw and pitch angles• Resolution: inter-eye distance • Blur: histogram of gradient magnitudes in the eye region• Lighting: approximate the face by a Lambertian cylinder
under orthographic projection, estimate face albedos and coefficients for an orthonormal illumination basis
• Seam signature: ribbon of pixels within a small radius of the replacement boundary (resampled, intensity normalized)
• Retrieval• SQL query for pose, resolution, and albedo• Further prune using blur and lighting distance• Select top 50 final candidates using seam distance=> Less than 1 sec to generate list of candidate replacements
![Page 63: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/63.jpg)
Candidate selection
![Page 64: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/64.jpg)
• Approximate face images:
• Relit replacement:
• After relighting, match the color histograms of the two faces
Relighting and recoloring
albedo
illumination coefficient
illumination basis image
![Page 65: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/65.jpg)
Relighting and recoloring
![Page 66: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/66.jpg)
Face replacement: Final steps• Rank replacements by color difference along the
overlap region• Perform feathering along the overlap region
![Page 67: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/67.jpg)
Face replacement results
![Page 68: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/68.jpg)
Face replacement results
![Page 69: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/69.jpg)
Face replacement results
![Page 70: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/70.jpg)
Face de-identification
![Page 71: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/71.jpg)
Face de-identification
![Page 72: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/72.jpg)
More results
![Page 73: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/73.jpg)
More results
![Page 74: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/74.jpg)
Improving family portraits
![Page 75: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/75.jpg)
Limitations
![Page 76: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/76.jpg)
Limitations
![Page 77: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/77.jpg)
User study• 50 face images, 12 subjects, 5 seconds per image
![Page 78: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/78.jpg)
Aside: Detecting composites of faces
M.K. Johnson and H. Farid, “Exposing Digital Forgeries Through SpecularHighlights on the Eye,” 9th International Workshop on Information Hiding, 2007
![Page 79: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/79.jpg)
More fun with faces
Example-Based Cosmetic Transfer
W.-S. Tong, C.-K. Tang, M. Brown, Y.-Q. XuPacific Graphics 2007
![Page 80: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/80.jpg)
More fun with faces
A Conditional Random Field for Automatic Photo EditingM. Brand, P. Pletscher
CVPR 2008
![Page 81: Forward vs. inverse warping - Computer Sciencelazebnik/research/fall08/lec08_faces.pdf• 92 face pairs (47 female, 45 male) • 68 raters, males and females aged between 25 and 40](https://reader034.fdocuments.us/reader034/viewer/2022042909/5f3c2d9840146e007f5d2509/html5/thumbnails/81.jpg)
More fun with faces
Expressive Expression Mapping with Ratio ImagesZ. Liu, Y. Shan, Z. Zhang
SIGGRAPH 2001
Original image Expression transfer by warping
Proposed method