Analysis of Lighting Effects

Outline: The problem Lighting models Shape from shading Photometric stereo Harmonic analysis of lighting

Applications

Modeling the effect of lighting can be used for Recognition – particularly face recognition Shape reconstruction Motion estimation Re-rendering …

Lighting is Complex

Lighting can come from any direction and at any strength

Infinite degree of freedom

Issues in Lighting

Single light source (point, extended) vs. multiple light sources

Far light vs. near light Matt surfaces vs. specular surfaces Cast shadows Inter-reflections

Lighting

From a source – travels in straight lines Energy decreases with r2 (r – distance from

source) When light rays reach an object

Part of the energy is absorbed Part is reflected (possibly different amounts

in different directions) Part may continue traveling into the object,

if object is transparent / translucent

Specular Reflectance

When a surface is smooth light reflects in the opposite direction of the surface normal

Specular Reflectance

When a surface is slightly rough the reflected light will fall off around the specular direction

Lambertian Reflectance

When the surface is very rough light may be reflected equally in all directions

Lambertian Reflectance

When the surface is very rough light may be reflected equally in all directions

Lambertian Reflectance

Lambert Law

n̂

cos ( 90 !)

ˆ ˆ( , )

o

T

I E

I l n l E l n n

or

l̂

BRDF

A general description of how opaque objects reflect light is given by the Bidirectional Reflectance Distribution Function (BRDF)

BRDF specifies for a unit of incoming light in a direction (θi,Φi) how much light will be reflected in a direction (θe,Φe) . BRDF is a function of 4 variables f(θi,Φi;θe,Φe).

(0,0) denotes the direction of the surface normal. Most surfaces are isotropic, i.e., reflectance in any

direction depends on the relative direction with respect to the incoming direction (leaving 3 parameters)

n̂

Why BRDF is Needed?

Light from front Light from back

Most Existing Algorithms

Assume a single, distant point source All normals visible to the source (θ<90°) Plus, maybe, ambient light (constant lighting

from all directions)

Shape from Shading

Input: a single image Output: 3D shape Problem is ill-posed, many different shapes can

give rise to same image Common assumptions:

Lighting is known Reflectance properties are completely known –

For Lambertian surfaces albedo is known (usually uniform)

convex

concave

convex

concave

HVS Assumes Light from Above

HVS Assumes Light from Above

Lambertian Shape from Shading (SFS) Image irradiance equation

Image intensity depends on surface orientation It also depends on lighting and albedo, but those

assumed to be known

ˆ( , ) ( ( , ))I x y R n x y

Surface Normal

A surface z(x,y) A point on the surface: (x,y,z(x,y))T

Tangent directions tx=(1,0,p)T, ty=(0,1,q)T with p=zx, q=zy

2 2

1ˆ ( , ,1)

. 1

x y Tt tn p q

p q

ˆ ˆ( , , ) , 1Tx y zn n n n n

Lambertian SFS

We obtain

Proportionality – because albedo is known up to scale For each point one differential equation in two unknowns, p and q

But both come from an integrable surface z(x,y) Thus, py= qx (zxy=zyx). Therefore, one differential equation in one unknowns

2 2ˆ( , ) ( , )

1

x y zT l p l q lI x y R p q l n

p q

Lambertian SFS

cos 0.5

60

I

SFS with Fast Marching

Suppose lighting coincides with viewing direction l=(0,0,1)T, then

Therefore

For general l we can rotate the camera

2 2

1

1I

p q

2 2 1

x y zl p l q lI

p q

2 22

11p q z

I

Distance Transform

is called Eikonal equation Consider d(x) s.t. |dx|=1

Assume x0=0

2

11z

I

x0

d

x

Distance Transform

is called Eikonal equation Consider d(x) s.t. |dx|=1

Assume x0=0 and x0=1

2

11z

I

x0

d

xx1

SFS with Fast Marching

- Some places are more difficult to walk than others

Solution to Eikonal equations –using a variation of Dijkstra’s algorithm

Initial condition: we need to know z at extrema Starting from lowest points, we propagate a

wave front, where we gradually compute new values of z from old ones

( , )z F x y

Results

Photometric Stereo

Fewer assumptions are needed if we have several images of the same object under different lightings

In this case we can solve for both lighting, albedo, and shape

This can be done by Factorization Recall that

Ignore the case θ>90°

s.t. 0T TI l n l n

Photometric Stereo - FactorizationGoal: given M, find L and S

M L S

1 1 111 1

1 2

1 2

1 2

3

1 3

...

. ....

. ....

. ....

p x y z

x x

y y

z z pf f f

f fp x y zf p f

I I l l l

n n

n n

n n

I I l l l

What should rank(M) be?

Photometric Stereo - Factorization Use SVD to find a rank 3 approximation

Define So Factorization is not unique, since

, A invertible

To reduce ambiguity we impose integrability

, 3 3TM U V

, TL U S V

M L S

1 ˆˆ( ) ( )M LA AS L S 1 ˆL̂ LA S AS

Reducing Ambiguity

Assume We want to enforce integrability Notice that

Denote by the three rows of A, then

From which we obtain

n An

y xp q

, yx

z z

nnp q

n n

1 2

3 3

,a n a n

p qa n a n

1 2

3 3

a n a n

y a n x a n

1 2 3, ,a a a

Reducing Ambiguity

Linear transformations of a surface

It can be shown that this is the only transformation that maintains integrability

Such transformations are called “generalized bas relief transformations” (GBR)

Thus, by imposing integrability the surface is reconstructed up to GBR

( , ) ( , )

x

y

y y x x

z x y x y z x y

p z p

q z q

p p q q

Relief Sculptures

Illumination Cone Due to additivity, the set of images of an object

under different lighting forms a convex cone in RN

This characterization is generic, holds also with specularities, shadows and inter-reflections

Unfortunately, representing the cone is complicated (infinite degree of freedom)

= 0.5* +0.2* +0.3*

Eigenfaces

Photobook/Eigenfaces (MIT Media Lab)

Recognition with PCA

Amano, Hiura, Yamaguti, and Inokuchi; Atick and Redlich; Bakry, Abo-Elsoud, and Kamel; Belhumeur, Hespanha, and Kriegman; Bhatnagar, Shaw, and Williams; Black and Jepson; Brennan and Principe; Campbell and Flynn; Casasent, Sipe and Talukder; Chan, Nasrabadi and Torrieri; Chung, Kee and Kim; Cootes, Taylor, Cooper and Graham; Covell; Cui and Weng; Daily and Cottrell; Demir, Akarun, and Alpaydin; Duta, Jain and Dubuisson-Jolly; Hallinan; Han and Tewfik; Jebara and Pentland; Kagesawa, Ueno, Kasushi, and Kashiwagi; King and Xu; Kalocsai, Zhao, and Elagin; Lee, Jung, Kwon and Hong; Liu and Wechsler; Menser and Muller; Moghaddam; Moon and Philips; Murase and Nayar; Nishino, Sato, and Ikeuchi; Novak, and Owirka; Nishino, Sato, and Ikeuchi; Ohta, Kohtaro and Ikeuchi; Ong and Gong; Penev and Atick; Penev and Sirivitch; Lorente and Torres; Pentland, Moghaddam, and Starner; Ramanathan, Sum, and Soon; Reiter and Matas; Romdhani, Gong and Psarrou; Shan, Gao, Chen, and Ma; Shen, Fu, Xu, Hsu, Chang, and Meng; Sirivitch and Kirby; Song, Chang, and Shaowei; Torres, Reutter, and Lorente; Turk and Pentland; Watta, Gandhi, and Lakshmanan; Weng and Chen; Yuela, Dai, and Feng; Yuille, Snow, Epstein, and Belhumeur; Zhao, Chellappa, and Krishnaswamy; Zhao and Yang…

Ball Face Phone Parrot

#1 48.2 53.7 67.9 42.8

#2 84.4 75.2 83.2 69.7

#3 94.4 90.2 88.2 76.3

#4 96.5 92.1 92.0 81.5

#5 97.9 93.5 94.1 84.7

#6 98.9 94.5 95.2 87.2

#7 99.1 95.3 96.3 88.5

#8 99.3 95.8 96.8 89.7

#9 99.5 96.3 97.2 90.7

#10 99.6 96.6 97.5 91.7

Empirical Study

(Yuille et al.)

0 1 2 30

0.5

1

0 1 2 30

0.5

1

1.5

2

lighting

reflectance

Intuition

(Light → Reflectance) = Convolution

(Light → Reflectance) = Convolution

Spherical Harmonics

Z YX

23 1Z XZ YZ22 YX XY

2 2 2 1X Y Z Positive values

Negative values

Harmonic Transform of Kernel

1.023

0.495

-0.111

0.05

-0.029

0.886

-0.5

0

0.5

1

1.5

0 1 2 3 4 5 6 7 8

00

( ) max(cos ,0) n nn

k k Y

nk

n

37.5

87.599.22 99.81 99.93 99.97

0

20

40

60

80

100

0 1 2 3 4 5 6 7 8

Cumulative Energy

N

1

N

nn

E

(percents)

Second Order Approximation

21 1 15( ) cos cos 2

64 2 64k

Reflectance Near 9D

0

4( )

2 1

n

n nm nmn m n

r k k l Yn

Yields 9D linear subspace.

4D approximation (first order) can also be used = point source + ambient

0

2 4( )

2 1

n

n nm nmn m n

k l Yn

Harmonic Representations

2(3 1)zn 2 2( )x yn n x yn n x zn n y zn n

zn xn yn

Positive values

Negative values( , , )x y zn n n n

ρ Albedo

n Surface normal

Photometric Stereo

LM

S

Image n

:

Image 1

Light n

:

Light 1

*

SVD recovers L and S up to an ambiguity

nz

nz

ny

nz2-1)

nx2-ny

2)

nxny

nxnz

nynz

r r

Photometric Stereo

Photometric Stereo

Summary

Lighting effects are complex Algorithms for SFS and photometric stereo for

Lambertian object illuminated by a single light source

Harmonic analysis extends this to multiple light sources

Handling specularities, shadows, and inter-reflections is difficult