Photometric and Photogeometric Stereo

CS635 Spring 2010

Daniel G. AliagaDepartment of Computer Science

Purdue University

Photometric Stereo

• A technique for estimating the surface normals of objects by observing that object under different lighting conditions– Then, using the surface normals, a plausible surface geometry can be reconstructed

– Woodham in 1980

• Related: when using a single image, it is called shape from shading– B. K. P. Horn in 1989

Photometric Stereo

surface

point

in ],1[ Pi

What are the values for ?in

Photometric Stereo

surface

point

n

l

cln )(

Lambertian (single channel) model:

Photometric Stereo

surface

point

n

l

clnlnln zzyyxx )(Lambertian (single channel) model:

How many unknowns per point?

How to get more equations?

3+1

More lights!

(known lights)

Photometric Stereo

surface

point

n

1l

1111 )( clnlnln zzyyxx

Photometric Stereo

surface

point

n

2l

1111 )( clnlnln zzyyxx

2222 )( clnlnln zzyyxx

Photometric Stereo

surface

point

n3l

1111 )( clnlnln zzyyxx

2222 )( clnlnln zzyyxx

3333 )( clnlnln zzyyxx

Photometric Stereo1111 )( clnlnln zzyyxx

2222 )( clnlnln zzyyxx

3333 )( clnlnln zzyyxx

Using 1l Using 2l Using 3l

Photometric Stereo1111 )( clnlnln zzyyxx

2222 )( clnlnln zzyyxx

3333 )( clnlnln zzyyxx

Using 1l Using 2l Using 3l

1c 2c 3c

Photometric Stereo1111 )( clnlnln zzyyxx

2222 )( clnlnln zzyyxx

3333 )( clnlnln zzyyxx

What is ? zyx nnn ,,

cLn Write as and solve cLn 1

Where Tzyx nnnn

Tcccc 321

What is the surface normal at the point? nn /

What is (the albedo at the surface point)? nn

Lambertian Photometric Stereo with Known Lights

• Take three pictures of a static Lambertian object with a static camera

• In each picture move the light to a different but known position– For distant lights, could know light direction instead of light position

• At pixel i, solve • Use normals to “integrate” a surface

ii cLn 1

Normals ‐> Surface

• How?

Normals ‐> Surface),( yxzSurface (height field) is

Tyx zzyxn 1),( Surface normal is

))y,x(z)y,1x(z(zx

))y,x(z)1y,x(z(zy and

1122

yxyx

zyx zzzz

nnn

1 x

z

x znn

1 y

z

y znn

As we saw before

So and or zxx nnz / zyy nnz /and

Altogether: xzz nyxznyxzn ),(),1(yzz nyxznyxzn ),()1,(

Normals ‐> Surfacexzz nyxznyxzn ),(),1(yzz nyxznyxzn ),()1,(

Can setup as a large over-constrained linear system: bAz Azb

where has a bunch of normal values are the unknown heights (z-values)is a zero vector

and solve (e.g., use “lsqr” method)bAz 1

Normals ‐> Surface

[Basri et al., IJCV, 2007]

Fundamental Ambiguity

CNL

CLNRR 1

CLNAA 1

CLANA ))(( 1

RGA

Rotation matrix

Generalized Bas Relief (GBR) Ambiguity matrix

Fundamental Ambiguity

Belhumeur et al. 1999

Fundamental Ambiguity

Belhumeur et al. 1999

GBR Transform

• Consider– This means “flatten” by    and add a plane (      )

• When                this is classical “bas‐relief”• Else, it is a generalized bas‐relief that can be written as

yxyxzyxz ),(),(

,

0

010001

G

10000

11

G

Photometric Stereo Ambiguity

CLANA ))(( 1

010001

RRGA

• Interpretation: given a solution for the normals, we can transform the normals by A and the lights by A-1

and the resulting picture looks the same…

Photometric Stereo with Unknown Lights

• Question:– What if both the surface normals and the light directions are unknown?

– Can we reconstruct the light directions, surface normals, and thus surface geometry (up to the aforementioned ambiguity)?

• Answer:– Yes, and the problem is still linear!

PS with Unknown LightingcLn Recall

1)( 2/1 nnTThink of the normals as located at the origin, then we care about a linear transformation of the sphere

Note PccLn 1

So 1)( QccPcPcPcPc TTTT

And Q is a 3x3 symmetric positive definite matrix:

Geometrically, this means intensity’s c lie on an ellipsoid whose equation is

01222 3223311321122

3332

2222

111 ccqccqccqcqcqcq

332313

232212

131211

qqqqqqqqq

which only has unknowns6

PS with Unknown Lighting

• Given at least six pixels, can solve for q’s• In general, a large overconstrained linear solution is used

3231212

32

22

1

1312131112112

132

122

11

222..................

222

nnnnnnnnn ccccccccc

cccccccccM

Tqqqqqqq 231312332211

1...1b

bMq 1 bMMMq TT 1)( or

PS with Unknown Lighting

• Given Q, we can plug back and get the angles between light directions and their strengths

• Then “lights are known” and solve for normals as before…– but need to choose a reasonable R and G

Photogeometric Approach

• Combine photometric stereo with geometric stereo– High resolution of photometric stereo– Accuracy of geometric method– Can lead to self‐calibration of entire acquisition process

Photogeometric Upsampling

1. Integrate surface normals

photo-surface

Photogeometric Upsampling

2. Compute sparse geometric model

geo-surface

Photogeometric Upsampling

3. Warp photometric surface to geometric surface

geo-surface

photo-surface

Photogeometric Upsampling

3. Warp photometric surface to geometric surfacephoto-geo surface

Photogeometric Upsampling

4. Triangulate and proceed to optimizationphoto-geo surface

true surface

Photogeometric Optimization

• Linear system in the unknown 3D points (pi)• Supports multi‐view reconstruction• Weighted combination of three error terms:

where 

eg = error of reprojection

ep = error of perpendicularity of normal‐to‐tangent

er = error of  relative distance change

Photogeometric Optimization• Linear system in the unknown 3D points (pi)• Supports multi‐view reconstruction• Weighted combination of three error terms:

where 

Photogeometric Reconstruction

photographs reconstruction