R&D Topics in Computer Film

j.p.lewis

. – p.1/??

R&D Topics in Computer Film

(About the author)

• Computer Graphics and Immersive Technology Lab,

U. Southern California; Digimax

• previously: ILM, ESC, Disney TSL, NYIT

• Film credits on Forest Gump, Godzilla, Matrix

Reloaded

• publications (Siggraph, TOG, CHI, GI)

• algorithms adopted by Softimage XSI, Houdini, Shake,

Commmotion, RenderDotC

. – p.2/??

Topics

• computer graphics research - in Hollywood?

• scattered interpolation, + applications (Pose Space,

101 Dalmations, Matrix Reloaded)

• virtual actors and face tracking (Gemini Man, Matrix

Reloaded, Digimax, USC)

. – p.3/??

R&D in Computer Film

• most algorithms are invented in academic labs

• some algorithms later adopted in commercial software

• programmers in Hollywood: usually doing

“uninteresting” projects, such as file format translators

• but there are exceptions!

. – p.4/??

102 Dalmations

J.P.Lewis, Lifting Detail from Darkness, SIGGRAPH 2001

. – p.5/??

Brightness-Detail Decomposition

image image

detail

intensity

• Separate detail by Wiener filter

• Represent brightness by membrane PDE

• “Unsharp masking 2.0”

Topics: Wiener filter, Laplace PDE, multigrid

. – p.6/??

Brightness-Detail Decomposition

image image

detail

intensity

• Modify detail, keep brightness. Example: texture

replacement (subject to limitations of 2D technique)

• Modify brightness, keep detail. Example: 102

Dalmatians spot removal

. – p.7/??

Scattered interpolation

. – p.8/??

Modeling

Deforming a face mesh

Images: Jun-Yong Noh and Ulrich Neumann, CGIT lab

. – p.9/??

Shepard Interpolation

d̂(x) =

∑

wk(x)dk∑

wk(x)

weights set to an inverse power of the distance:wk(x) = ‖x− xk‖

−p.

Note: singular at the data points xk.

. – p.10/??

Shepard Interpolation

. – p.11/??

Laplace/Poisson interpolation

minimizeR

(f ′(x))2dx should come out like d2f

dx2= ∇2 = 0

F (y, y′, x) = y′2

δF = ∂Fdy′

dy′

dεδε

= ∂Fdy′

q′δε ∂Fdy′

= 2y′ = 2 dfdx

dEdε

=R

∂Fdy′ q

′dx now change q’ to qR

∂Fdy′ q

′dx = ∂Fdy′ q −

R

ddx

∂Fdy′ qdx integration by parts

∂Fdy′ q

˛

˛

˛

b

a= 0 because q is zero at both ends

dEdε

= −R

ddx

∂Fdy′ qdx = 0 variation of functional is zero at minimum

= −2 ddx

dfdx

= −2 d2f

dx2= −2∇2f = 0

. – p.12/??

Laplace/Poisson Interpolation

Objective: Minimize a roughness measure, theintegrated derivative (or gradient) squared:

∫

df

dx

2

dx

∫ ∫

|∇f |2ds

. – p.13/??

Laplace/Poisson Interpolation

Discrete/matrix viewpoint: Encode derivativeoperator in a matrix D

D =

1

−1 1

−1 1

. . .

minf

fTDTDf

. – p.14/??

Laplace/Poisson Interpolation

minf

fTDTDf

2DTDf = 0

i.e.d2f

dx2= 0 or ∇2 = 0

f = 0 is a solution; last eigenvalue is zero,corresponds to a constant solution.

. – p.15/??

Laplace/Poisson: alternate derivation

Local viewpoint:

roughness R =

∫

|∇u|2du ≈∑

(uk+1 − uk)2

for a particular k:dR

duk

=d

duk

[(uk − uk−1)2 + (uk+1 − uk)

2]

= 2(uk − uk−1)− 2(uk+1 − uk) = 0

uk+1 − 2uk + uk−1 = 0→ ∇2u = 0

Notice: DT D = . . . 1,−2, 1

. – p.16/??

Laplace/Poisson: solution approaches

• direct matrix inverse (better: Choleski)

• Jacobi (because matrix is quite sparse)

• Jacobi variants (SOR)

• Multigrid

. – p.17/??

Jacobi iteration

matrix viewpoint

Ax = b

(D + E)x = b split into diagonal D, non-diagonal E

Dx = −Ex + b

x = −D−1Ex + D−1b call B = D−1E, z = D−1b

x← Bx + z D−1 is easy

hope that largest eigenvalue of B is less than 1

. – p.18/??

Jacobi iteration

Local viewpontJacobi iteration sets each fk to the solution of its row of the

matrix equation, independent of all other rows:

∑

Arcfc = br

→ Arkfk = bk −∑

j 6=k

Arjfj

fk ←bk

Akk

−∑

j 6=k

Akj/Akkfj

. – p.19/??

Jacobi iteration

apply to Laplace eqnJacobi iteration sets each fk to the solution of its row of the

matrix equation, independent of all other rows:

. . . ft−1 − 2ft + ft+1 = 0

2ft = ft−1 + ft+1

fk ← 0.5 ∗ (f [k − 1] + f [k + 1])

In 2D,

f[y][x] = 0.25 * ( f[y+1][x] + f[y-1][x] +

f[y][x-1] + f[y][x+1] )

. – p.20/??

But now let’s interpolate

1D case, say f3 is known. Three eqns involve f3.Subtract (a multiple of) f3 from both sides ofthese equations:

f1 − 2f2 + f3 = 0 → f1 − 2f2 + 0 = −f3

f2 − 2f3 + f4 = 0 → f2 + 0 + f4 = 2f3

f3 − 2f4 + f5 = 0 → 0− 2f4 + f5 = −f3

L =

2

6

6

6

6

6

4

1 −2 0

1 0 1

0 −2

. . .

3

7

7

7

7

7

5

one column is zeroed

. – p.21/??

Multigrid

Ax = b

x̃ = x + e

r is known, e is not r = Ax̃− b

r = Ax + Ae− b

r = Ae

For Laplace/Poisson, r is smooth. So decimate, solve for e,

interpolate. And recurse...

. – p.22/??

demo

. – p.23/??

Applications: Pose Space Deformation

Lewis/Cordner/Fong, SIGGRAPH 2000

incorporated in Softimage

(video)

. – p.24/??

Applications: Matrix virtual city

. – p.25/??

Radial Basis Functions

d̂(x) =N

∑

k

wkφ(‖x− xk‖)

. – p.26/??

Radial Basis Functions (RBFs)

• any function other than constant can be used!

• common choices:

• Gaussian φ(r) = exp(−r2/σ2)

• Thin plate spline φ(r) = r2 log r• Hardy multiquadratic

φ(r) =√

(r2 + c2), c > 0

Notice: the last two increase as a function of radius

. – p.27/??

RBF versus Shepard’s

. – p.28/??

Solving RBF interpolation

• if few known points: linear system

• if few unknown points: multigrid

. – p.29/??

Radial Basis Functions

d̂(x) =N

X

k

wkφ(‖x − xk‖)

e =X

j

(d(xj) − d̂(xj))2

=X

j

(d(xj) −

NX

k

wkφ(‖xj − xk‖))2

= (d(x1) −N

X

k

wkφ(‖x1 − xk‖))2 + (d(x2) −

NX

k

wkφ(‖x2 − xk‖))2 + · · ·

. – p.30/??

Radial Basis Functions

define Rjk = φ(‖xj − xk‖)

= (d(x1) − (w1R11 + w2R12 + w3R13 + · · ·))2

+(d(x2) − (w1R21 + w2R22 + w3R23 + · · ·))2 + · · ·

+(d(xm) − (w1Rm1 + w2Rm2 + w3Rm3 + · · ·))2 + · · ·

d

dwm

= 2(d(x1) − (w1R11 + w2R12 + w3R13 + · · ·))R1m

+2(d(x2) − (w1R21 + w2R22 + w3R23 + · · ·))R2m

+ · · ·

+2(d(xm) − (w1Rm1 + w2Rm2 + w3Rm3 + · · ·)) + · · · = 0

put Rk1, Rk2, Rk3, · · · in row m of matrix.

. – p.31/??

Radial Basis Functions

d̂(x) =N

∑

k

wkφ(‖x− xk‖)

e = ||(d−Φw)||2

e = (d−Φw)T (d−Φw)

de

dw= 0 = −Φ

T (d−Φw)

ΦTd = Φ

TΦw

w = (ΦTΦ)−1

ΦTd

. – p.32/??

Thin plate spline

Minimize the integrated second derivativesquared (approximate curvature)

minf

∫(

d2f

dx2

)2

dx

. – p.33/??

Where does TPS kernel come from

Fit an unknown function f to the data yk, regularized by

minimizing a smoothness term.

E[f ] =∑

(fk − yk)2 + λ

∫

||Pf ||2

e.g. ||Pf ||2 =

∫(

d2f

dx2

)2

dx

A similar discrete version.

E[f ] = (f − y)′S′S(f − y) + λ2P ′Pf

. – p.34/??

Where does TPS kernel come from

(continued) A similar discrete version.

E[f ] = (f − y)′S′S(f − y) + λ2P ′Pf

• To simplify things, here the data points to interpolate are required to be at discrete

sample locations in the vector y, so the length of this vector defines a “sample rate”

(reasonable).

• S is a “selection matrix” with 1s and 0s on the diagonal (zeros elsewhere). It has

1s corresponding to the locations of data in y. y can be zero (or any other value)

where there is no data.

• P is a diagonal-constant matrix that encodes the discrete form of the regularization

operator. E.g. to minimize the integrated curvature, rows of P will contain:

2

6

6

4

−2, 1, 0, 0, . . .

1,−2, 1, 0, . . .

0, 1,−2, 1, . . .

3

7

7

5

. – p.35/??

Where does TPS kernel come from

Take the derivative of E with respect to the vectorf ,

2S(f − y) + λ2P ′Pf = 0

P ′Pf = −1

λS(f − y)

Multiply by G, being the inverse of P ′P :

f = GP ′Pf = −1

λGS(f − y)

So the RBF kernel is G = (P ′P )−1.

. – p.36/??

Matrix regularization

Find w to minimize (Rw− b)T (Rw− b). If the training points

are very close together, the corresponding columns of R

are nearly parallel. Difficult to control if points are chosen

by a user.

Add a term to keep the weights small: wT w.

minimize (Rw − b)T (Rw − b) + λwT w

RT (Rw − b) + 2λw = 0

RT Rw + 2λw = RT b

(RT R + 2λI)w = RT b

w = (RTR + 2λI)−1RT b. – p.37/??

Virtual Actors - Disney

Disney’s Gemini Man test, 2001

(video)

. – p.38/??

Virtual Actors - Disney

Disney’s Gemini Man test, 2001

Optic flow, constrained by markers

. – p.39/??

Virtual Actors - Matrix Reloaded

• George Borshukov and J.P.Lewis Realistic Human

Face Rendering for "The Matrix Reloaded" Siggraph

2003 Technical Sketch

• George Borshukov, Dan Piponi, Oystein Larsen, J. P.

Lewis, Christina Tempelaar-Lietz Universal Capture:

Image-BasedFacial Animation for "The Matrix

Reloaded" Siggraph 2003 Technical Sketch

(pdfs)

. – p.40/??

Virtual Actors - Matrix Reloaded

. – p.41/??

Face tracking - Digimax

(live demo)

. – p.42/??

Virtual Actors - USC research

• T.Y.Kim and U. Neumann, Interactive Multiresolution

Hair Modeling and Editing, SIGGRAPH 2002 paper

(video)

• Z. Deng, JP Lewis, U. Neumann, Practical Eye

Movement Model using Texture Synthesis, SIGGRAPH

03 sketch

. – p.43/??

Conclusion

• learn the math

• numerical methods do matter

. – p.44/??