The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination...
Transcript of The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination...
![Page 1: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/1.jpg)
Chun-Po Wang
CS7670: October 4, 2011
![Page 2: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/2.jpg)
Scene Illumination What is it?
Environment map
A sphere approximate incident light from long distance
A 2D radiance function in sphere coordinates
Assumptions
Far-field lighting
Independent to position
Grace Cathedral, San FranciscoAngular map formathttp://ict.debevec.org/~debevec/Probes/
![Page 3: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/3.jpg)
Azimuthal
Elev
atio
n
Unwarp
180-180
90
-90
![Page 4: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/4.jpg)
Applications Realistic 3D rendering/relighting
http://gl.ict.usc.edu/Research/RHL/
![Page 5: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/5.jpg)
Applications (cont.) Insert 3D objects into photos
“Webcam Clipart”, http://www.jflalonde.org/projects/webcamclipart/
![Page 6: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/6.jpg)
Applications (cont.) Help scene understanding
Illumination vastly changes scene appearance
Solution 1: use illumination invariant features (e.g., SIFT)
Solution 2: utilize information about scene illumination
![Page 7: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/7.jpg)
Example: Blind Reflectometry Ambiguity between illumination and reflectance
Find material properties by
Choosing an appropriate BRDF representation
Find the statistics of outdoor environment maps
Select most likely BRDF parameters under that distribution of environment maps
[Romeiro and Zickler, 2010]
![Page 8: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/8.jpg)
Example: Illumination-Aware Pedestrian Detection [Lalonde, PhD thesis, 2011]
![Page 9: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/9.jpg)
EstimatingScene Illumination Light Probe [Debevec et al., 1998]
![Page 10: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/10.jpg)
Outdoor Illuminationfrom Image Sequence Webcam time lapsed video
“What do color changes reveal about an outdoor scene?”[Sunkavalli et al., 2008]
“What Do the Sun and the Sky Tell Us About the Camera?”[Lalonde et al., 2008, 2010]
”Webcam Clip Art”[Lalonde et al., 2009]
![Page 11: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/11.jpg)
Outdoor Illumination from Single Image Estimating Natural Illumination from a Single Outdoor
Image [Lalonde et al., 2009]
Estimating sun position and sky color
Using image cues:
Sky color
Shadow lines
Shading of vertical surfaces
![Page 12: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/12.jpg)
[Lalonde et al., ICCV 2009]
![Page 13: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/13.jpg)
Geometric Context [Hoiem et al., 2005]
Pixel location Color
Texture Perspective
Classification/Labeling
Sky
Vertical facing rightVertical
Ground
![Page 14: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/14.jpg)
[Lalonde et al., ICCV 2009]
![Page 15: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/15.jpg)
Sun probability distribution map
Zenith
Le
ft
Rig
ht
Back
Forward [Lalonde et al., ICCV 2009]
![Page 16: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/16.jpg)
Sky Model
zenith angle angle with the sun
• [Perez et al., 1993] and [Preetham et al., 1999]• a,b,c,d,e can be approximated with a linear function of a single parameter, t (turbidity)• In this work, sky is assumed to be clear (t = 2.17)• Cloud is segmented by clustering based on color
![Page 17: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/17.jpg)
Original sky
Predicted sky at current sun position
[Lalonde et al., ICCV 2009]
![Page 18: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/18.jpg)
[Lalonde et al., ICCV 2009]
![Page 19: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/19.jpg)
Sun behind camera Sky not visible
[Lalonde et al., ICCV 2009]
![Page 20: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/20.jpg)
Ground shadows
[Lalonde et al., ICCV 2009]
![Page 21: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/21.jpg)
[Lalonde et al., ICCV 2009]
![Page 22: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/22.jpg)
[Lalonde et al., ICCV 2009]
![Page 23: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/23.jpg)
Shadow detection Non-vertical objects
[Lalonde et al., ICCV 2009]
![Page 24: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/24.jpg)
Surfaces shading
Vertical facing left
[Lalonde et al., ICCV 2009]
![Page 25: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/25.jpg)
[Lalonde et al., ICCV 2009]
![Page 26: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/26.jpg)
No flat surface
[Lalonde et al., ICCV 2009]
![Page 27: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/27.jpg)
Cue Combination
Sky color
Ground shadow
Vertical surfaceSun position prior
Sky pixels
Ground pixels
Vertical surface pixels
Bayes rule
Sun position
![Page 28: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/28.jpg)
P(sun|surfaces)
P(sun|sky)P(sun)
P(sun|shadow)
P(sun|image)
![Page 29: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/29.jpg)
[Lalonde et al., ICCV 2009]
![Page 30: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/30.jpg)
[Lalonde et al., ICCV 2009]
![Page 31: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/31.jpg)
[Lalonde et al., ICCV 2009]
![Page 32: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/32.jpg)
Quantitative evaluation
[Lalonde et al., ICCV 2009]
![Page 33: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/33.jpg)
Quantitative evaluation
Error (deg)
% o
f im
ages (
984 im
ages t
ota
l)
Data-driven prior
++
Scene cues + data
[Lalonde et al., ICCV 2009]
![Page 34: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/34.jpg)
[Lalonde et al., ICCV 2009]
![Page 35: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/35.jpg)
Code and Dataset
http://www.jflalonde.org/projects/outdoorIllumination/
Currently only the code for the Sky model is available
Extension in Lalonde’s PhD thesis [2011]
Find sun direction by person appearance
[0,90] [90,180][-90,0][-180,-90]
Binary SVM classifiers
Person Detector[Felzenszwalb et al., 2010]
![Page 36: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/36.jpg)
![Page 37: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/37.jpg)
![Page 38: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/38.jpg)
Shadow Detection Application
Estimating outdoor illumination
Shadow removal
Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et al., ECCV 2010]
Single-Image Shadow Detection and Removal using Paired Regions [Guo et al., CVPR 2011]
![Page 39: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/39.jpg)
Detecting Ground Shadows in Outdoor Consumer Photographs [Lalonde et al., ECCV 2010]
Observation: photometric methods do not work well on consumer images (not linear, lossy compression)
Hypothesis: appearances of shadows on the ground are less varied than shadows in general, and can be learned from labeled images.
![Page 40: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/40.jpg)
![Page 41: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/41.jpg)
![Page 42: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/42.jpg)
Single-Image Shadow Detection and Removal using Paired Regions [Guo et al., CVPR 2011]
![Page 43: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/43.jpg)
Shadow removalSegmentationSingle/Pairwise
region classification
Shadow Labeling
Soft shadow matting
Shadow mask
![Page 44: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/44.jpg)
Guo’s methodLalonde’s method
![Page 45: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/45.jpg)
Guo’s method
Lalonde’s method
![Page 46: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/46.jpg)
Practical Issues Applying to outdoor illumination estimation
Segmentation (region based) => does not work well on thin shadows
![Page 47: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/47.jpg)
Lalonde’s method
![Page 48: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/48.jpg)
Guo’s method
![Page 49: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/49.jpg)
Thank you Questions?
![Page 50: The Dimensionality of Scene AppearanceShadow Detection Application Estimating outdoor illumination Shadow removal Detecting Ground Shadowsin Outdoor Consumer Photographs [Lalonde et](https://reader034.fdocuments.us/reader034/viewer/2022050416/5f8c0b320d18511fca2c216b/html5/thumbnails/50.jpg)
Question
Bayes rule
Assuming conditionalindependence
Bayes rule
Can we ignore this?
I: illumination (sun direction)S: sky pixelsG: ground pixelsV: vertical surface pixelsS,G,V: input image