Lambertian model of reflectance I: shape from shading and...
Transcript of Lambertian model of reflectance I: shape from shading and...
![Page 1: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/1.jpg)
Lambertian model of reflectance I: shape from shading and
photometric stereo
Ronen Basri
Weizmann Institute of Science
![Page 2: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/2.jpg)
Variations due to lighting (and pose)
![Page 3: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/3.jpg)
Relief
Dumitru Verdianu Flying Pregnant Woman
![Page 4: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/4.jpg)
Edges
![Page 5: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/5.jpg)
Edges on smooth surfaces
![Page 6: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/6.jpg)
Bump or dip?
![Page 7: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/7.jpg)
What sticks out?
![Page 8: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/8.jpg)
Light field
• Rays travel from sources to objects
• There they are either absorbed or reflected
• Energy of light decreases with distance and number of bounces
• Camera is designed to capture the set of rays that travel through the focal center
• We will assume the camera response is linear
![Page 9: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/9.jpg)
Specular reflectance (mirror)
• When a surface is smooth light reflects in the opposite direction of the surface normal
![Page 10: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/10.jpg)
Specular reflectance
• When a surface is slightly rough the reflected light will fall off around the specular direction
![Page 11: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/11.jpg)
Diffuse reflectance
• When the surface is very rough light may be reflected equally in all directions
![Page 12: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/12.jpg)
Diffuse reflectance
• When the surface is very rough light may be reflected equally in all directions
![Page 13: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/13.jpg)
BRDF
• Bidirectional Reflectance Distribution Function
𝑓(𝜃𝑖𝑛, ∅𝑖𝑛; 𝜃𝑜𝑢𝑡, ∅𝑜𝑢𝑡)
• Specifies for a unit of incoming light in a direction (𝜃𝑖𝑛, ∅𝑖𝑛) how much light will be reflected in a direction (𝜃𝑜𝑢𝑡, ∅𝑜𝑢𝑡)
𝑛
![Page 14: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/14.jpg)
BRDF
Light from front Light from back
![Page 15: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/15.jpg)
Lambertian reflectance
![Page 16: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/16.jpg)
Lambertian reflectance
![Page 17: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/17.jpg)
Lambert’s law
𝑛 𝑙
𝐼 = 𝐸𝜌 cos 𝜃 𝜃 ≤ 900
𝐼 = 𝑙𝑇𝑛 (𝑙 = 𝐸𝑙 , 𝑛 = 𝜌𝑛 )
𝜃
![Page 18: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/18.jpg)
Preliminaries
• A surface is denoted 𝑧 𝑥, 𝑦
• A point on 𝑧 is (𝑥, 𝑦, 𝑧 𝑥, 𝑦 )
• The tangent plane is spanned by 1,0, 𝑧𝑥 (0,1, 𝑧𝑦)
• The surface normal is given by
𝑛 =1
𝑧𝑥2 + 𝑧𝑦
2 + 1
−𝑧𝑥, −𝑧𝑦 , 1
![Page 19: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/19.jpg)
Photometric stereo
• Given several images of the a lambertian object under varying lighting
• Assuming single directional source
𝑀 = 𝐿𝑆
![Page 20: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/20.jpg)
Photometric stereo
𝑀 = 𝐿𝑆 𝐼11 ⋯ 𝐼1𝑝
⋮ ⋮⋮ ⋮
𝐼𝑓1 … 𝐼𝑓𝑝 𝑓×𝑝
=
𝑙1𝑥 𝑙1𝑦 𝑙1𝑧
⋮⋮
𝑙𝑓𝑥 𝑙𝑓𝑦 𝑙𝑓𝑧 𝑓×3
𝑛𝑥1
𝑛𝑦1 ⋯
𝑛𝑧1
𝑛𝑥𝑝
⋯ 𝑛𝑥𝑝
𝑛𝑥𝑝 3×𝑝
• We can solve for S if L is known (Woodham) • This algorithm can be extended to more complex
reflection models (if known) through the use of a lookup table
![Page 21: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/21.jpg)
Factorization
• Use SVD to find a rank 3 approximation
𝑀 = 𝑈Σ𝑉𝑇
• Define Σ3 = 𝑑𝑖𝑎𝑔(𝜎1, 𝜎2, 𝜎3), where 𝜎1, 𝜎2, 𝜎3 are the largest
singular values of 𝑀
𝐿 = 𝑈 Σ3, 𝑆 = Σ3𝑉𝑇 and 𝑀 ≈ 𝐿 𝑆
• Factorization is not unique, since
𝑀 = 𝐿 𝐴−1 𝐴𝑆 , 𝐴 is 3 × 3 invertible
• We can reduce ambiguity by imposing integrability
(Hayakawa)
![Page 22: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/22.jpg)
Generlized bas-relief ambiguity
• Linearly related surfaces: given a surface 𝑧(𝑥, 𝑦) the surfaces related linearly to 𝑧 are:
𝑧 𝑥, 𝑦 = 𝑎𝑥 + 𝑏𝑦 + 𝑐𝑧(𝑥, 𝑦)
𝐺 =1 0 00 1 0𝑎 𝑏 𝑐
𝐺−1 =1
𝑐
𝑐 0 00 𝑐 0
−𝑎 −𝑏 1
• Forms a sub-group of GL(3)
(Belhumeur, Kriegman and Yuille)
![Page 23: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/23.jpg)
Bas-relief
![Page 24: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/24.jpg)
Integrability
• Objective: given a normal field 𝑛(𝑥, 𝑦) ∈ 𝑆2 recover depth 𝑧(𝑥, 𝑦) ∈ ℝ
• Recall that
𝑛 = (𝑛𝑥 , 𝑛𝑦 , 𝑛𝑧) =𝜌
𝑧𝑥2 + 𝑧𝑦
2 + 1
−𝑧𝑥 , −𝑧𝑦 , 1
• Given 𝑛, we set
𝜌 = 𝑛 ; 𝑝 = −𝑛𝑥
𝑛𝑧, 𝑞 = −
𝑛𝑦
𝑛𝑧
![Page 25: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/25.jpg)
Integrability
• Solve
min𝑧(𝑥,𝑦)
(𝑧 𝑥 + 1, 𝑦 − 𝑧 𝑥, 𝑦 − 𝑝)2+(𝑧 𝑥, 𝑦 + 1 − 𝑧(𝑥, 𝑦)−q)2
where
𝑝 = −𝑛𝑥
𝑛𝑧, 𝑞 = −
𝑛𝑦
𝑛𝑧
![Page 26: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/26.jpg)
Photometric stereo with matrix completion (Wu et al.)
![Page 27: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/27.jpg)
Shape from shading (SFS)
• What if we only have one image?
• Assuming that lighting is known and albedo is uniform
𝐼 = 𝑙𝑇𝑛 ∝ cos 𝜃
• Every intensity determines a circle of possible normals
• There is only one unknown (𝑧) for each pixel (since uniform albedo is assumed)
![Page 28: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/28.jpg)
Shape from shading
• Denote 𝑛 = −𝑧𝑥 , −𝑧𝑦 , 1
• Then
𝐸 =𝑙𝑇𝑛
𝑛
• Therefore 𝐸2𝑛𝑇𝑛 = 𝑛𝑇𝑙𝑙𝑇𝑛
• We obtain 𝑛𝑇 𝑙𝑙𝑇 − 𝐸2𝐼 𝑛 = 0
• This is a first order, non-linear PDE (Horn)
![Page 29: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/29.jpg)
Shape from shading
• Suppose 𝑙 = (0,0,1), then
𝑧𝑥2 + 𝑧𝑦
2 =1
𝐸2− 1
• This is called an Eikonal equation
![Page 30: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/30.jpg)
Distance transform
• The distance of each point to the boundary
![Page 31: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/31.jpg)
Shortest path
![Page 32: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/32.jpg)
Distance transform
• Posed as an Eikonal equation 𝛻𝑧 2 = 𝑧𝑥
2 + 𝑧𝑦2 = 1
![Page 33: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/33.jpg)
Update for shortest path
𝑇1
𝑇2
𝑤1
𝑤2
𝑇 = min 𝑇1 + 𝑤1, 𝑇2 + 𝑤2
𝑇 =?
![Page 34: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/34.jpg)
Update for fast marching
𝑇1
𝑇2
𝑓
𝑇 = 𝑇1 + 𝑇2 + 2𝑓2 − (𝑇1 − 𝑇2)2 if real
𝑓 + min 𝑇1, 𝑇2 otherwise
(Tsitsiklis, Sethian)
![Page 35: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/35.jpg)
SFS Solution
• Lambertian SFS produces an eikonal equation
𝛻𝑧 2 = 𝑧𝑥2 + 𝑧𝑦
2 =1
𝐸2− 1
• Right hand side determines “speed”
• Boundary conditions are required: depth values at local
maxima of intensity and possibly in shape boundaries
• The case 𝑙 ≠ 0,0,1 is handled by change of variables
(Kimmel & Sethian)
![Page 36: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/36.jpg)
Example
![Page 37: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/37.jpg)
Summary
• Understanding the effect of lighting on images is challenging, but can lead to better interpretation of images
• We considered Lambertian objects illuminated by single sources
• We surveyed two problems:
– Photometric stereo
– Shape from shading
![Page 38: Lambertian model of reflectance I: shape from shading and ...ayuille/courses/Stat238-Winter14/... · •Lambertian SFS produces an eikonal equation 𝛻 6= 6+ 6= 1 𝐸 6 −1 •Right](https://reader033.fdocuments.us/reader033/viewer/2022060603/6057ef1ff772f931ea45816b/html5/thumbnails/38.jpg)
Challenges
• General reflectance properties – Lambertian – specular – general BRDFs
• Generic lighting – multiple light sources (“attached shadows”) – near light
• Cast shadows • Inter-reflections • Dynamic scenes • Local approaches (eg. direction of gradients)
Can we hope to model this complexity?