Y. Moses 1

Combining Photometric and Geometric

Constraints

Yael Moses

IDC, Herzliya

Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion

Y. Moses 2

Recover the 3D shape of a general smooth surface from a set of calibrated images

Problem 1:

Y. Moses 3

Problem 2:

Recover the 3D shape of a smooth bilaterally symmetric object from a single image.

Y. Moses 4

Shape Recovery

Geometry: Stereo Photometry:

Shape from shading Photometric stereo

Main problems: Calibrations and Correspondence

Y. Moses 5

3D Shape Recovery

Photometry: Shape from

shading Photometric

stereo

Geometry:Stereo

Structure from motion

Y. Moses 6

Geometric Stereo

2 different images Known camera parameters Known correspondence

+ +

Y. Moses 7

Photometric Stereo

3D shape recovery: surface normals from two or more

images taken from the same viewpoint

Y. Moses 8

Photometric Stereo

Solution:

)ˆ(

3

2

1

3

2

1

n

l

l

l

I

I

I

IT

T

T

LnI

IL

ILn

1

1

nlyxI

nlyxI

nlyxI

ˆ),(

ˆ),(

ˆ),(

33

22

11

Three images Matrix notation

IL

ILn

1

Y. Moses 9

Photometric Stereo

3D shape recovery (surface normals) Two or more images taken from the

same viewpoint

Main Limitation:

Correspondence is obtained by a fixed viewpoint

IL

ILn

1

Y. Moses 10

Overview

Combining photometric and geometric stereo: Symmetric surface, single image Non symmetric: 3 images

Mono-Geometric stereo Mono-Photometric stereo Experimental results.

Y. Moses 11

The input Smooth featureless surface Taken under different viewpoints Illuminated by different light sources

The Problem: Recover the 3D shape from a set of calibrated images

Y. Moses 12

Assumptions

Given correspondence the normals can be computed (e.g., Lambertian, distant point light source …)

*

*

ln

nlI ˆ

n

Three or more images

Perspective projection

Y. Moses 13

Our method

Combines photometric and geometric stereo

We make use of:

Given Correspondence:

Can compute a normal

Can compute the 3D point

Y. Moses 14

IL

ILn

1

1

ˆ

Basic Method

GivenCorrespondence

Y. Moses 15

First Order Surface Approximation

Y. Moses 16

First Order Surface Approximation

Y. Moses 17

First Order Surface Approximation

P() = (1 - )O1 + P,

N)O - (P

N)O - (PT

1

T1

N (P() - P) = 0

Y. Moses 18

First Order Surface Approximation

Y. Moses 19

PMp ii

New Correspondence

Y. Moses 20

IL

ILn

1

1

ˆ

New Surface Approximation

Y. Moses 21

Dense Correspondence

Y. Moses 22

Basic Propagation

Y. Moses 23

Basic Propagation

Y. Moses 24

Basic method: First Order

Given correspondence pi and L

P and n Given P and n T Given P, T and Mi

a new correspondence qi

Y. Moses 25

Extensions

Using more than three images Propagation:

Using multi-neighbours Smart propagation

Second error approximation Error correction:

Based on local continuity Other assumptions on the surface

IL

ILn

1

Y. Moses 26

Multi-neighbors Propagation

Y. Moses 27

Smart Propagation

Y. Moses 28

Second Order: a Sphere

P()

N+N

N

N

P

(P-P())(N+N)=0

Y. Moses 29

Second Order Approximation

Y. Moses 30

Second Order Approximation

Y. Moses 31

Using more than three images

Reduce noise of the photometric stereo

Avoid shadowed pixels Detect “bad pixels”

Noise Shadows Violation of assumptions on the surface

Y. Moses 32

Smart Propagation

Y. Moses 33

Error correction

The compatibility of the local 3D shape can be used to correct errors of:

Correspondence Camera parameters Illumination parameters

Y. Moses 34

Score

Continuity: Shape Normals Albedo

The consistency of 3D points locations and the computed normals: General case: full triangulation Local constraints

Y. Moses 35

Extensions

Using more than three images Propagation:

Using multi-neighbours Smart propagation

Second error approximation Error correction:

Based on local continuity Other assumptions on the surface

Y. Moses 36

Real Images

Camera calibration Light calibration

Direction Intensity Ambient

Y. Moses 37

Error correction + multi-neighbor5 Images

Y. Moses 38

5pp

3pp 3nn

5nn5pn

Y. Moses 39

Y. Moses 40

Y. Moses 41

Y. Moses 42

Y. Moses 43

Y. Moses 44

Detected Correspondence

Y. Moses 45

Error correction + multi-neighbordMulti-neighborsBasic scheme (3 images)Error correction no multi-neighbors

Y. Moses 46

New ImagesSynthetic Images

Y. Moses 47

Sec a

Ground truthBasic schemeMulti-neighbors Error correction

Y. Moses 48

Sec b

Ground truthBasic schemeMulti-neighbors Error correction

Y. Moses 49

Sec c

Ground truthBasic schemeMulti-neighbors Error correction

Y. Moses 50

Ground truthBasic schemeMulti-neighbors

approx.Error correction

Sec d

Ground truthBasic schemeMulti-neighbors Error correction

Y. Moses 51

Combining Photometry and

Geometry

Yields a dense correspondence and dense shape recovery of the object

in a single path

Y. Moses 52

Assumptions

Bilaterally Symmetric object Lambertian surface with constant

albedo Orthographic projection Neither occlusions nor

shadows Known “epipolar geometry”

Y. Moses 53

Geometric Stereo

2 different images Known camera parameters Known viewpoints Known correspondence

3D shape recovery

Y. Moses 54

Computing the Depth from Disparity

pl

pr

P

ql

qrZ

Z

Orthographic

Projection

Y. Moses 55

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses 56

Symmetry and Geometric Stereo

Non frontal view of a symmetric object

Two different images of the same object

Y. Moses 57

Geometry

Weak perspective projection:

)(~ tsRpprojp

Around X Around Z Around Y

zyx RRRR

xR zR yRI I

Y. Moses 58

Geometry

Projection of Ry:

010

)sin(0)cos()(' yRprojR

Around Y

is the only pose parameter

PRp '~

Image point

Object point

Y. Moses 59

objectx

z

image

),,( zyxP l Tr zyxP ),,(

)~,~(~ rrr yxp )~,~(~ lll yxp

Correspondence

Assume YxZ is the symmetry plane.

x~

Y. Moses 60

Mono-Geometric Stereo

3D reconstruction: given correspondence and ,

unknown

known

PRp '~

x~

z

image

x object

)sin()cos(~)sin()cos(~

~~

zxx

zxx

yyy

l

r

lr

xx

rx~lx~

Y. Moses 61

Viewpoint Invariant

Given the correspondence and unknown

2

1

22

11

~~

~~

x

x

pp

pplr

lr

*1

~p

*2~p

*3~p 3*

~p2*

~p1*

~p

lr pp 11~~

)sin()cos(~)sin()cos(~

~~

zxx

zxx

yyy

l

r

lr

Invariant

Y. Moses 62

Photometric Stereo

2 images Lambertian reflectance Known illuminations Known correspondence

(same viewpoint)

3D shape recovery

Y. Moses 63

Symmetry and Photometric Stereo

Non-frontal illumination of a symmetric object

Two different images of the same object

Y. Moses 64

Notation: Photometry

Corresponding object points:

Illumination:

Tyx

r zzn )1,,( Tyx

l zzn )1,,( Tzyx

r nnnn ),,(ˆ Tzyx

l nnnn ),,(ˆ

Tzyx eeeE ),,(

Tzyx

r nnnn ),,(ˆ Tzyx

l nnnn ),,(ˆ

Y. Moses 65

Mono-Photometric Stereo

3D reconstruction given correspondence and E (up to a twofold ambiguity):

unknown

known

zzyyxxr neneneI

zzyyxxl neneneI

)(2

2

zzyylr

xxlr

neneIII

neIII

Y. Moses 66

Invariance to Illumination

Given correspondence and E unknown

Invariant:x

x

n

n

I

I

2

1

2

1

)(2

2

zzyylr

xxlr

neneIII

neIII

Y. Moses 67

Mono-Photometric Stereo

3D reconstruction E unknown but correspondence is given

Frontal viewpoint with non-frontal illumination. Use image first derivatives.

Y. Moses 68

Mono-Photometric Stereo Using image derivatives

3 global unknowns: E For each pair:

5 unknowns zx zy zxx zxy zyy

6 equations 3 pairs are sufficient

Y. Moses 69

Mono-Photometric StereoUnknown Illumination

Y. Moses 70

Correspondence

No correspondence => no stereo. Hard to define correspondence in

images of smooth surfaces. Almost any correspondence is legal

when: Only geometric constraints are

considered. Only photometric constraints are

considered.

Y. Moses 71

Combining Photometry and

Geometry Yields a dense correspondence (dense shape recovery of the

object).

Enables recovering of the global parameters.

Y. Moses 72

Self-Correspondence A self-correspondence function:

lr ppC ~)~(

*~ lp

rp~*lrr xyxC ~)~,~(

Y. Moses 73

Dense Correspondence using Propagation

Assume correspondence between a pair of points, p0

l and p0r.

rr yxCydCxdpCpdpC ~~

~~)~()~~(

lp2~

rp0*~

rp1*~

rp2~

*1~ lp

*0~ lp

)~,~(~ ydxdpd

Y. Moses 74

Dense Correspondence using Propagation

* ***

**

lp2~

rp0*~

rp1*~

rp2~

*1~ lp

*0~ lp

Y. Moses 75

x

x~

z

image

objectn̂

dxdx

dz

)sin()cos(~ dzdxxd )cos(~ dxxd

xdxcxdxc ~)~()~~(

dx

dz

xCxdxCxdxC ~~)~()~~(

Y. Moses 76

Tyx zzn )1,,(

First derivatives of the Correspondence

Assume known Assume known E

)sin()cos(

)2sin(

;)sin()cos(

)sin()cos(

~

~

x

y

y

x

xx

z

zC

z

zC

r

r

Y. Moses 77

)sin()cos(~)sin()cos(~

~~

zxxzxx

yyy

l

r

lr

Computing and

ryy

ryxy

rxy

rxxx

yCxCC

yCxCC

rr

rr

~~

~~

~~

~~

rr yxCC ~~

Object coordinates:Given computing and is trivial

Moving from object to image coordinates depends on the viewing parameter

rr yxCCn̂

Y. Moses 78

Derivatives with respect to the object coordinates:

Derivatives with respect to the image coordinates:

Y. Moses 79

x

x~

z

image

object

xd~

yx cc ~~

E

)~,~(~ rrr yxp

)~,~(~ lll yxp

Y. Moses 80

Given a corresponding pair and E n=(zx,zy,-1)T Given and n

cx and cy

Given cx and cy

a new corresponding pair

General Idea

Y. Moses 81

Results on Real Images: Given global parameters

Y. Moses 82

Finding Global Parameters

Assume E and are unknown. Assume a pair of corresponding

points is given. Two possibilities:

Search for E and directly.

Compute E and from the image second derivatives.

Y. Moses 83

All roads lead to Rome …

Find and verify correct correspondence

Recover global parameters, E and

Integration Constraint:Circular Tour

Y. Moses 84

Finding Global Parameters

Consider image second derivatives Due to foreshortening effect:

and

We can relate image and object derivatives by

xll

xxrr

xIIII rr ~~

ry

r

y

ry

r

x

ry

rx

r

y

rx

r

x

rx

yIxII

yIxII

rr

rr

~~

~~

~~

~~

Y. Moses 85

For each corresponding pair:

and

Plus 4 linear equations in 3 unknown.

Where

Testing E and : Image second derivatives

),,,(34

EzzfA yxx

yy

xy

xx

l

y

r

y

l

x

r

x

zzz

A

I

III

r

r

r

r

~

~

~

~

rrll nEInEI ˆˆ

Y. Moses 86

Counting

5 unknowns for each pair: zx zy,zxx zxy zyy

4 global unknowns: E, For each pair: 6 equations. For n pairs: 5n+4 unknowns

6n equations. 4 pairs are sufficient

Y. Moses 87

Results on Simulated Data

Ground Truth Recovered Shape

Y. Moses 88

Recovering the Global Parameters

Y. Moses 89

Degenerate Case

Close to frontal view: problems with geometric-stereo.

reconstruction problem Close to frontal illumination:

problems with photometric-stereo. correspondence problem

Y. Moses 90

Future work

Perspective photometric stereo Use as a first approximation to global

optimization methods Test on other reflection models Recovering of the global parameters:

Light Cameras

Detect the first pair of correspondence

Y. Moses 91

Future Work

Extend to general 3 images under 3 viewpoints and 3 illuminations.

Extend to non-lambertian surfaces.

Y. Moses 4

Assumptions

Bilaterally Symmetric object Lambertian surface with constant

albedo Orthographic projection Neither occlusions nor shadows Known “epipolar geometry”

Y. Moses 92

Thanks

Y. Moses 93

x

x~

z

image

object

)cos(~

dx

xd

n̂

dx

dz

dx

dz

Y. Moses 94

Integration Constraint

ry

r

y

ry

r

x

ry

rx

r

y

rx

r

x

rx

yIxII

yIxII

rr

rr

~~

~~

~~

~~

l

xylx

l

y

r

xyrx

r

y

x

lxl

x

x

rxr

x

lr

rr

r

r

IznEI

IznEI

z

nEI

z

nEI

~~

~~

~

~

)sin(ˆ

)sin(ˆ

)sin()cos(

ˆ

)sin()cos(

ˆ

Y. Moses 95

Integration Constraint

)(2

2

zzyy

xx

neneI

neI

l

xylx

l

y

r

xyrx

r

y

x

lxl

x

x

rxr

x

lr

rr

r

r

IznEI

IznEI

z

nEI

z

nEI

~~

~~

~

~

)sin(ˆ

)sin(ˆ

)sin()cos(

ˆ

)sin()cos(

ˆ

Y. Moses 96

Searching for E

Illumination must satisfy:

E is further constrained by the image second derivatives.

),max( lr IIE

)(2 zzyy neneI

xxneI 2

Y. Moses 97

Image second derivatives:

xll

x

xrr

x

yrl

y

yrr

y

II

II

II

II

r

r

r

r

~

~

~

~

lx

l

x

rx

r

x

ly

l

y

ry

r

y

nEI

nEI

nEI

nEI

r

r

r

r

ˆ

ˆ

ˆ

ˆ

~

~

~

~

),,,,(ˆ yyxyxxyxrx zzzzzn

yy

xy

xx

l

y

r

y

l

x

r

x

zzz

A

I

III

r

r

r

r

~

~

~

~ll nEI ˆrr nEI ˆ

Where ),,(34

EzzfA yxx

4 linear equations in 3 unknown

Y. Moses 98

For each corresponding pair and E: 4 linear equations in 3 unknown.

Where

Image second derivatives

),,(34

EzzfA yxx

yy

xy

xx

l

y

r

y

l

x

r

x

zzz

A

I

III

r

r

r

r

~

~

~

~

zzyyxxr neneneI

zzyyxxl neneneI

Y. Moses 99

Counting

5 unknowns for each pair: zx,zy,zxx,zxy,zyy

3 global unknowns: E For each pair: 6 equations. For n pairs: 5n+3 unknowns

6n equations.3 pairs are sufficient

Y. Moses 100

Correspondence

Tr zyxP ),,( Tl zyxP ),,(

)~,~(~ rrr yxp )~,~(~ lll yxp

Y. Moses 101

Variations

Known/unknown distant light source Known/unknown viewpoint Symmetric/non-symmetric image

Frontal/non-frontal viewpoint Frontal/non-frontal illumination

Y. Moses 102

Correspondence

Epipolar geometry is the only geometric constraint on the correspondence.

Weak photometric constraint on the correspondence.

)(2

2

zzyy

xx

neneI

neI

Y. Moses 103

Lambertian Surface

Basic radiometric

)cos(ˆ),(Re EnEqpf I =

E

*

*P

EE

n2

n1

n̂

Y. Moses 104

E

Photometric Stereo

First proposed by Woodham, 1980. Assume that we have two images ..

nEInEI ˆˆ 2211