Appearance Models
description
Transcript of Appearance Models
![Page 1: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/1.jpg)
Appearance Models
• Shape models represent shape variation• Eigen-models can represent texture
variation• Combined appearance models represent
both
![Page 2: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/2.jpg)
Appearance Models
• Statistical model of shape and texture
• Generative model– general
– specific
– compact
![Page 3: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/3.jpg)
Building Appearance Models
• For each example extract shape vector
• Build statistical shape model,
Shape, x = (x1,y1, … , xn, yn)T
ssbPxx
![Page 4: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/4.jpg)
Building Appearance Models
• For each example, extract texture vector
Shape, x = (x1,y1, … , xn, yn)T
Texture, gWarp tomeanshape
![Page 5: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/5.jpg)
Warping texture
• Problem:– Given corresponding points in two images,
how do we warp one into the other?
• Two common solutions1. Piece-wise linear using triangle mesh2. Thin-plate spline interpolation
![Page 6: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/6.jpg)
Interpolation using Triangles
Region of interest enclosed by triangles.
Moving nodes changes each triangle
Just need to map regions between two triangles
),( :points Control ii yx )','( :points Warped ii yx
![Page 7: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/7.jpg)
Barycentric Co-ordinates
cbax
a b
c
x
''' cbax '
'a
'b
'c
'x
1
10 and 10if triangle theinside is
βαx
![Page 8: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/8.jpg)
Barycentric Co-ordinates
a b
c
x)( ab
)( ac cba
cbaacabax
)1(
)()(
cbax
1
Three linear equations in 3 unknowns
1111yyy
xxx
cbacba
yx
![Page 9: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/9.jpg)
Interpolation using Triangles
• To find out where each pixel in new image comes from in old image
• Determine which triangle it is in• Compute its barycentric co-ordinates• Find equivalent point in equivalent triangle in
original image• Only well defined in region of `convex
hull’ of control points
![Page 10: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/10.jpg)
Thin-Plate Spline Interpolation
• Define a smooth mapping function (x’,y’)=f(x,y) such that
– It maps each point (x,y) onto (x’,y’) and does something smooth in between.
– Defined everywhere, even outside convex hull of control points
niyxyxf iiii ..1 allfor )','(),(
![Page 11: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/11.jpg)
Thin-Plate Spline Interpolation
• Function has form )),(),,((),( yxfyxfyxf yx
ii
ni
ixixxx rrwxbayxf log),( 2
1
ii
ni
iyiyyy rrwybayxf log),( 2
1
222 )()( where iii yyxxr
)','(),(bygiven equationslinear thesolvingby found are
),,,,,( parameters The
iiii
yiyyxixx
yxyxf
wbawba
![Page 12: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/12.jpg)
Building Texture Models
• For each example, extract texture vector
• Normalise vectors (as for eigenfaces)• Build eigen-model
Texture, g
Warp tomeanshape
ggbPgg
![Page 13: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/13.jpg)
Face Texture Model
1b12 12
2b22 22
3b32 32
![Page 14: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/14.jpg)
Textured Shape Modes
Shape variation (texture fixed)
Generate position of control points
Warp mean texture image
(Mean points go to new points, X)
)( ssT bPxX
![Page 15: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/15.jpg)
Textured Shape Model
1b12 12
2b22 22
3b32 32
![Page 16: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/16.jpg)
Combined Models
• Shape and texture often correllated– When smile, shadows change (texture) and
shape changes• Learning this correlation leads to more
compact (and specific) model
![Page 17: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/17.jpg)
Learning Correlations
sb
gbModel assuming shape and texture independent
Model accounting for correlations between shape and texture
![Page 18: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/18.jpg)
Learning Correlations
• For each image in training set we have best fitting shape and texture param.s
• Construct new vector,
• Apply PCA (mean + eigenvec.s of covar.)
gs bb ,
g
sc b
Wbb
cQQ
Qcb
g
sc
too.does thus
set, trainingover themean zero have and Both
c
gs
b
bb
![Page 19: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/19.jpg)
Combined Appearance Models
cQ
WQQc
bWb
b
g
s
g
sc
ggbPgg ssbPxx
cQggcQxx
g
s
Varying c changes both
shape and texture
![Page 20: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/20.jpg)
Combined Appearance Model
• Generate shape, X, and texture, g• Warp texture so mean control points lie on
new X
1b12 12
![Page 21: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/21.jpg)
Face Appearance Model
1b12 12
2b22 22
![Page 22: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/22.jpg)
Face Appearance Model
3b32 32
4b42 42
![Page 23: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/23.jpg)
Sub-cortical structures
• 72 examples• 123 points• 5000 pixel model
Ventricles
Lentiform Nucleus
Caudate Nucleus
![Page 24: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/24.jpg)
Shape and Texture Modes
Shape variation (texture fixed)
Texture variation (shape fixed)
![Page 25: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/25.jpg)
Combined Appearance Model
• Shape and texture correlated
![Page 26: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/26.jpg)
Full brain slice
Shape:
Texture:
![Page 27: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/27.jpg)
Full brain slice
CombinedMode 1
CombinedMode 2
![Page 28: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/28.jpg)
Problems with viewpoint
• Models require all points visible– Sometimes a problem for 2D images of 3D objects
• Small rotations (+/-30o) of face modelled well• Large rotations cause occlusions
– Eg eye hidden behind nose etc• Solutions
1. Use multiple `view based’ 2D models2. Use a full 3D model
![Page 29: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/29.jpg)
View-Based Models
• Build 3 distinct models– Exploit symmetry
Profile Profile (Reflected)
Frontal
Half-Profile Half-Profile (Reflected)
![Page 30: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/30.jpg)
Face Profile Model
Mode 1:
Mode 2:
![Page 31: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/31.jpg)
Half-Profile Model
Mode 1:
Mode 2:
![Page 32: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/32.jpg)
3D Models
• Use 3D shape model (3n-D vectors)
• Points control a polyhedral mesh
• Texture mapped onto mesh and modelled
• Reconstruct by generating new texture and mapping onto 3D mesh described by shape model
![Page 33: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/33.jpg)
3D Models
=+
Mesh
Texture
![Page 34: Appearance Models](https://reader036.fdocuments.us/reader036/viewer/2022062323/56815d93550346895dcbafd1/html5/thumbnails/34.jpg)
Interpreting Images (1)
Place model in image
Measure Difference
Update Model
Iterate