C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering...
Transcript of C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering...
![Page 1: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/1.jpg)
C280 Computer VisionC280, Computer Vision
Prof. Trevor [email protected]
Lecture 23: Segmentation II & Computational Photography Teaser
![Page 2: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/2.jpg)
Two presentations today:Two presentations today:
![Page 3: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/3.jpg)
Contours and Junctions in Natural Images
Jitendra MalikUniversity of California at BerkeleyUniversity of California at Berkeley
(with Jianbo Shi, Thomas Leung, Serge Belongie, Charless Fowlkes, David Martin, Xiaofeng Ren, Michael Maire, Pablo Arbelaez)
3
![Page 4: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/4.jpg)
From Pixels to Perceptionp
GrassWater
outdoorTiger
Sand
wildlife
b kSand back
Tigerhead
tail
eye
legs mouth
4
shadow
![Page 5: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/5.jpg)
I stand at the window and see a house, trees, sky Theoretically I might say there were 327sky. Theoretically I might say there were 327 brightnesses and nuances of colour.
Do I have "327"?
No. I have sky, house, and trees.No. I have sky, house, and trees.
---- Max Wertheimer, 1923Max Wertheimer, 1923
5
![Page 6: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/6.jpg)
Perceptual Organization
Grouping Figure/GroundGrouping Figure/Ground
6
![Page 7: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/7.jpg)
Key Research Questions in Perceptual OrganizationOrganization
• Predictive power– Factors for complex, natural stimuli ?– How do they interact ?
• Functional significance– Why should these be useful or confer some
evolutionary advantage to a visual organism?evolutionary advantage to a visual organism?
• Brain mechanismsHow are these factors implemented given what we– How are these factors implemented given what we know about V1 and higher visual areas?
7
![Page 8: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/8.jpg)
Attneave’s Cat (1954)Line drawings convey most of the g y
information
8
![Page 9: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/9.jpg)
Contours and junctions are fundamental…
• Key to recognition, inference of 3D scene properties, visually- guided manipulation and locomotion…
• This goes beyond local, V1-like, edge-detection. Contours are the result of perceptual organization grouping and figure/groundorganization, grouping and figure/ground processing
9
![Page 10: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/10.jpg)
Some computer vision history…p y
• Local Edge Detection was much studied in the 1970s and early 80s (Sobel, Rosenfeld, Binford-Horn, Marr-Hildreth, Canny …)
• Edge linking exploiting curvilinear continuity was studied as well (Rosenfeld, Zucker, Horn, Ullman )Ullman …)
• In the 1980s, several authors argued for perceptual organization as a precursor toperceptual organization as a precursor to recognition (Binford, Witkin and Tennebaum, Lowe, Jacobs …)
10
, )
![Page 11: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/11.jpg)
However in the 90s …
1. We realized that there was more to images than edges• Biologically inspired filtering approaches (Bergen & Adelson,
Malik & Perona..)• Pixel based representations for recognition (Turk & Pentland,
√Murase & Nayar, LeCun …)
2. We lost faith in the ability of bottom-up vision• Do minimal bottom up processing e g tiled orientationDo minimal bottom up processing , e.g. tiled orientation
histograms don’t even assume that linked contours or junctions can be extracted
• Matching with memory of previously seen objects then becomes
?Matching with memory of previously seen objects then becomes the primary engine for parsing an image.
11
![Page 12: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/12.jpg)
At Berkeley, we took a contrary view…y y
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for
grouping and figure-ground segmentation
12
![Page 13: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/13.jpg)
Berkeley Segmentation DataSet [BSDS]
13D. Martin, C. Fowlkes, D. Tal, J. Malik. "A Database of Human Segmented Natural Images and its
Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics", ICCV, 2001
![Page 14: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/14.jpg)
14
![Page 15: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/15.jpg)
Contour detection ~1970
1515
![Page 16: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/16.jpg)
Contour detection ~1990
1616
![Page 17: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/17.jpg)
Contour detection ~2004
1717
![Page 18: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/18.jpg)
Contour detection ~2008 (gray)
1818
![Page 19: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/19.jpg)
Contour detection ~2008 (color)
1919
![Page 20: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/20.jpg)
Outline
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for
grouping and figure-ground segmentation
20
![Page 21: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/21.jpg)
Contours can be defined by any of a number of cues (P. Cavanagh)
21
![Page 22: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/22.jpg)
Cue-Invariant RepresentationsGray level photographsGray level photographs
Objects from motion
Objects from luminance
Objects from disparityObjects from texture Line drawings
22Grill-Spector et al. , Neuron 1998
![Page 23: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/23.jpg)
Martin, Fowlkes, Malik PAMI 04PImage
Boundary CuesPb
BrightnessCue Combination
ModelBrightness
Color
Texture
Challenges: texture cue, cue combinationG l l h i b bili f b dGoal: learn the posterior probability of a boundary Pb(x,y,) from local information only
23
![Page 24: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/24.jpg)
Individual Features
• 1976 CIE L*a*b* colorspacer
• Brightness Gradient BG(x,y,r,) – Difference of L* distributions
r(x,y)
• Color Gradient CG(x,y,r,)– Difference of a*b* distributions
• Texture Gradient TG(x,y,r,)– Difference of distributions of
V1-like filter responses
24These are combined using logistic regression
![Page 25: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/25.jpg)
Various Cue CombinationsCombinations
25
![Page 26: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/26.jpg)
Outline
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for
grouping and figure-ground segmentation
26
![Page 27: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/27.jpg)
Exploiting global constraints:Image Segmentation as Graph Partitioning
Build a weighted graph G=(V,E) from image
V: image pixelsg p
E: connections between pairs of nearby pixelspairs of nearby pixels
Partition graph so that similarity within group is large and similarity between groups is small -- Normalized Cuts
27
y g p z[Shi & Malik 97]
![Page 28: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/28.jpg)
Wij small when intervening contour strong, small when weak..
Cij = max Pb(x,y) for (x,y) on line segment ij; Wij = exp ( - Cij / �
28
![Page 29: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/29.jpg)
Eigenvectors carry contour informationg y
29
![Page 30: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/30.jpg)
We do not try to find regions from the eigenvectors, so we avoid the “broken sky” artifacts of Ncuts …
30
![Page 31: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/31.jpg)
Key idea – compute edges on ncut eigenvectors, fi t ksum over first k:
where is the output of a Gaussian derivative on the j th eigenvectorwhere is the output of a Gaussian derivative on the j-th eigenvector of
31
![Page 32: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/32.jpg)
The Benefits of GlobalizationMaire, Arbelaez, Fowlkes, Malik, CVPR 08
32
![Page 33: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/33.jpg)
Comparison to other approaches
33
![Page 34: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/34.jpg)
34
![Page 35: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/35.jpg)
Outline
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3. Global framework to capture closure, continuity4. Detect and localize junctions5. Integrate low, mid and high-level information for
grouping and figure-ground segmentation
35
![Page 36: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/36.jpg)
Detecting Junctions
36
![Page 37: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/37.jpg)
37
![Page 38: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/38.jpg)
Benchmarking corner detection
38
![Page 39: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/39.jpg)
39
![Page 40: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/40.jpg)
Better object recognition using previous version of Pb
• Ferrari, Fevrier, Jurie and Schmid (PAMI 08)
• Shotton, Blake and Cipolla (PAMI 08)
40
![Page 41: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/41.jpg)
Outline
1. Collect Data Set of Human segmented images2. Learn Local Boundary Model for combining
brightness, color and texture3 Gl b l f k t t l ti it3. Global framework to capture closure, continuity4. Detect and localize junctions5 Integrate low mid and high level cues for5. Integrate low, mid and high-level cues for
grouping and figure-ground segmentation1. Ren, Fowlkes, Malik, IJCV ‘08, , ,2. Fowlkes, Martin, Malik, JOV ‘073. Ren, Fowlkes, Malik, ECCV ‘06
41
![Page 42: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/42.jpg)
Power laws for contour lengthsg
42
![Page 43: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/43.jpg)
Convexity [Metzger 1953, Kanizsa and Gerbino 1976]
ConvG = percentage of straight lines that lie completely within region G
Convexity(p) = log(ConvF / ConvG)
that lie completely within region G
pG F
43
![Page 44: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/44.jpg)
Figural regions tend to be convex
44
![Page 45: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/45.jpg)
Lower Region[Vecera, Vogel & Woodman 2002]
θ
p
L R i ( ) θ
45
LowerRegion(p) = θGcenter of mass
![Page 46: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/46.jpg)
Figural regions tend to lie below ground regions
46
![Page 47: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/47.jpg)
Ren, Fowlkes, Malik ECCV ‘06
Object and Scene Recognition
Grouping /Segmentation
Figure/GroundOrganization
• Human subjects label groundtruth figure/ground assignments in natural images.
• Shapemes encode high-level knowledge in a generic way, capturing local figure/ground cues.
• A conditional random field incorporates junction cues and enforces global consistency.
47
![Page 48: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/48.jpg)
Forty years of contour detectiony y
Roberts Sobel Prewitt Marr Canny Perona Martin Maire
4848
Roberts(1965)
Sobel(1968)
Prewitt(1970)
MarrHildreth(1980)
Canny(1986)
PeronaMalik(1990)
MartinFowlkesMalik(2004)
MaireArbelaezFowlkesMalik(2008)
![Page 49: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/49.jpg)
Forty years of contour detectiony y
???Roberts Sobel Prewitt Marr Canny Perona Martin Maire
4949
???(2013)
Roberts(1965)
Sobel(1968)
Prewitt(1970)
MarrHildreth(1980)
Canny(1986)
PeronaMalik(1990)
MartinFowlkesMalik(2004)
MaireArbelaezFowlkesMalik(2008)
![Page 50: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/50.jpg)
Curvilinear Groupingp g
• Boundaries are smooth in nature!Boundaries are smooth in nature!
• A number of associated visual phenomena
Good continuation Visual completion Illusory contours
50
![Page 51: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/51.jpg)
51
![Page 52: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/52.jpg)
Computational PhotographyComputational Photography
Computer VisionCSE 576, Spring 2008
Richard SzeliskiMicrosoft Research
![Page 53: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/53.jpg)
C t ti l Ph t hComputational Photography
photometric camera calibration• photometric camera calibration• high-dynamic range imaging & tone mapping
fl h h t h• flash photography
Richard Szeliski Computational Photography 53
![Page 54: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/54.jpg)
R diReadings• Debevec and Malik Recovering High Dynamic Range• Debevec and Malik, Recovering High Dynamic Range
Radiance Maps from Photographs. In SIGGRAPH 97.• S. B. Kang et al. High dynamic range video.g g y g
SIGGRAPH 2003.• D. Lischinski. Interactive local adjustment of tonal
values SIGGRAPH 2006values. SIGGRAPH 2006.• G. Petschnigg et al. Digital photography with flash and
no-flash image pairs. SIGGRAPH 2004.g p• P. Pérez et al. Poisson image editing. SIGGRAPH 2003
Richard Szeliski Computational Photography 54
![Page 55: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/55.jpg)
SSources
Some of m slides are fromSome of my slides are from:
Bill Freeman and Frédo Durandhttp://groups csail mit edu/graphics/classes/CompPhoto06/
Richard Szeliski Computational Photography 55
http://groups.csail.mit.edu/graphics/classes/CompPhoto06/
![Page 56: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/56.jpg)
SSources
Some of m slides are fromSome of my slides are from:
Alexei (Alyosha) Efroshttp://graphics cs cmu edu/courses/15 463/
Richard Szeliski Computational Photography 56
http://graphics.cs.cmu.edu/courses/15-463/
![Page 57: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/57.jpg)
But firstBut first, …
… for something (a little) different …
![Page 58: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/58.jpg)
P hPanography - http://www.flickr.com/search/?q=panography
Richard Szeliski Computational Photography 58
![Page 59: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/59.jpg)
P hPanography - http://www.flickr.com/search/?q=panograph
Richard Szeliski Computational Photography 59
![Page 60: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/60.jpg)
P hPanography
What kind of motion model?What kind of motion model?
Wh t ki d f iti ?What kind of compositing?
Can you do “global alignment”?
Richard Szeliski Computational Photography 60
![Page 61: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/61.jpg)
High Dynamic Range Imaging (HDR)
slides borrowed from15-463: Computational Photography
Alexei Efros, CMU, Fall 2007,Paul Debevec, and my talksy
![Page 62: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/62.jpg)
P bl D i RProblem: Dynamic Range
1 The real world ishigh dynamic range
1500
25 000
high dynamic range.
25,000
400,000
2,000,000,000
Richard Szeliski Computational Photography 62
![Page 63: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/63.jpg)
P bl D i RProblem: Dynamic Range
Typical cameras have limited dynamic range
What can we do?Solution: merge multiple exposures
Richard Szeliski Computational Photography 63
Solution: merge multiple exposures
![Page 64: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/64.jpg)
Varying ExposureVarying Exposure
Richard Szeliski Computational Photography 64
![Page 65: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/65.jpg)
HDR i lti l i tHDR images — multiple inputs
Pixel count
Richard Szeliski Computational Photography 65
Radiance
![Page 66: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/66.jpg)
HDR i dHDR images — merged
Pixel count
Richard Szeliski Computational Photography 66
Radiance
![Page 67: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/67.jpg)
C i t h t t !Camera is not a photometer!
Limited dynamic rangeLimited dynamic rangeUse multiple exposures?
Unknown nonlinear responseUnknown, nonlinear response Not possible to convert pixel values to radiance
Solution:Solution:• Recover response curve from multiple exposures,
then reconstruct the radiance mapp
Richard Szeliski Computational Photography 67
![Page 68: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/68.jpg)
Imaging system response functionImaging system response function
255
PixelPixelvaluevalue
0
l EE l (R di(R di ** t)t)log Exposure = Exposure = log (Radiance(Radiance * * t)t)(CCD photon count)
![Page 69: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/69.jpg)
C C lib tiCamera Calibration
Geometric• How pixel coordinates relate to directions in theHow pixel coordinates relate to directions in the
world
Photometric• How pixel values relate to radiance amounts in the
worldworld• Per-pixel transfer and blur
Richard Szeliski Computational Photography 69
![Page 70: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/70.jpg)
C i i liCamera sensing pipelineCamera
Irradiance
Camera Body
Optics Aperture Shutter
RAW
Sensor chip
Sensor(CCD/CMOS) A / DGain
(ISO)
Demosaic (Sharpen)
DSP
White Balance Gamma/curve Compress
JPEG
Richard Szeliski Computational Photography 70
![Page 71: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/71.jpg)
C i i liCamera sensing pipeline
Camera Body
CameraIrradiance Optics Aperture Shutter
Blur kern. & RD F-stop & Vignette Exposure T
RAW
Sensor chip
Sensor(CCD/CMOS) A / DGain
(ISO)
AA CFA Noise ISO Gain Q1
Demosaic (Sharpen)? ?
DSP
White Balance Gamma/curve Compress
JPEG
RGB Gain Q2
Richard Szeliski Computational Photography 71
RGB Gain Q2
![Page 72: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/72.jpg)
Recovering High Dynamic RangeRadiance Maps from Photographs
Paul DebevecPaul DebevecJitendra Malik
Computer Science DivisionUniversity of California at Berkeley
SIGGRAPH’97, August 1997
![Page 73: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/73.jpg)
W tWays to vary exposure
Sh tter Speed (*) Shutter Speed (*)
F/stop (aperture iris) F/stop (aperture, iris)
Neutral Density (ND) Filters Neutral Density (ND) Filters
Richard Szeliski Computational Photography 73
![Page 74: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/74.jpg)
Sh tt S dShutter Speed
Ranges Canon D30 30 to 1/4 000 secRanges: Canon D30: 30 to 1/4,000 sec.(1997) Sony VX2000: ¼ to 1/10,000 sec.
PPros:Directly varies the exposureUsually accurate and repeatable
Issues:Noise in long exposures
Richard Szeliski Computational Photography 74
![Page 75: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/75.jpg)
Sh tt S dShutter Speed
Note sh tter times s all obe a po er seriesNote: shutter times usually obey a power series –each “stop” is a factor of 2
¼, 1/8, 1/15, 1/30, 1/60, 1/125, 1/250, 1/500, 1/1000 sec
Usually really is:
¼, 1/8, 1/16, 1/32, 1/64, 1/128, 1/256, 1/512, 1/1024 sec
Richard Szeliski Computational Photography 75
![Page 76: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/76.jpg)
Th Al ithThe Algorithm•••• •••• ••
••
11 • •
22••
11 • •
22 ••
11 • •
22••
11 • •
22 ••
11 • •
22
t =t =t =t = t =t =t =t = t =t =
33
33
33
33
33
1 sec1 sec1/16 sec1/16 sec 4 sec4 sec1/64 sec1/64 sec 1/4 sec1/4 sec
Pixel Value Z = f(Exposure)Exposure = Radiance t
log Exposure = log Radiance log tRichard Szeliski Computational Photography 76
log Exposure log Radiance log t
![Page 77: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/77.jpg)
R CResponse CurveAssuming unit radiance After adjusting radiances toAssuming unit radiance
for each pixelAfter adjusting radiances to
obtain a smooth response curve
ue
33
ue
ixel
val
u
22
Pix
el v
alu
P
11P
Richard Szeliski Computational Photography 77
log Exposure log Exposure
![Page 78: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/78.jpg)
Th M thThe Math
Let g(z) be the discrete inverse response functionFor each pixel site i in each image j, want:
S l th d t i d li t
ln Radiancei lnt j g(Zij )
Solve the over-determined linear system:
ln Radiancei lntj g(Zij ) 2P
N
g (z)2Zmax
fi i h
ln Radiancei lntj g(Zij ) j1
i1 g (z)
zZmin
Richard Szeliski Computational Photography 78
fitting term smoothness term
![Page 79: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/79.jpg)
M tL b dMatLab codefunction [g,lE]=gsolve(Z,B,l,w)
n = 256;A = zeros(size(Z,1)*size(Z,2)+n+1,n+size(Z,1));b = zeros(size(A,1),1);
k = 1; %% Include the data-fitting equationsk 1; %% Include the data fitting equationsfor i=1:size(Z,1)
for j=1:size(Z,2)wij = w(Z(i,j)+1);A(k,Z(i,j)+1) = wij; A(k,n+i) = -wij; b(k,1) = wij * B(i,j);k=k+1;
endend
A(k,129) = 1; %% Fix the curve by setting its middle value to 0k k+1k=k+1;
for i=1:n-2 %% Include the smoothness equationsA(k,i)=l*w(i+1); A(k,i+1)=-2*l*w(i+1); A(k,i+2)=l*w(i+1);k=k+1;
end
Richard Szeliski Computational Photography 79
end
x = A\b; %% Solve the system using SVD
g = x(1:n);lE = x(n+1:size(x,1));
![Page 80: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/80.jpg)
R lt di it lResults: digital cameraRecovered responseRecovered responseKodak DCS460 Recovered response Recovered response
curvecurve1/30 to 30 sec
lue
lue
ixel
val
ixel
val
PiPi
Richard Szeliski Computational Photography 80log Exposurelog Exposure
![Page 81: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/81.jpg)
R t t d R di MReconstructed Radiance Map
Richard Szeliski Computational Photography 81
![Page 82: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/82.jpg)
R lt C l FilResults: Color Film
Kodak Gold ASA 100 PhotoCDKodak Gold ASA 100, PhotoCD
Richard Szeliski Computational Photography 82
![Page 83: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/83.jpg)
Recovered Response Curvesp
Red Green
RGBBlue
Richard Szeliski Computational Photography 83
![Page 84: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/84.jpg)
Th R di MThe Radiance Map
Richard Szeliski Computational Photography 84
![Page 85: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/85.jpg)
Th R di MThe Radiance Map
Linearly scaled toLinearly scaled todisplay devicedisplay device
Richard Szeliski Computational Photography 85
![Page 86: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/86.jpg)
P t bl Fl tM ( f )Portable FloatMap (.pfm)12 bytes per pixel 4 for each channel12 bytes per pixel, 4 for each channel
i t tisign exponent mantissa
Text header similar to Jeff Poskanzer’s .ppmPF768 5121<bi i d t >Floating Point TIFF similar
image format:
Richard Szeliski Computational Photography 86
<binary image data>Floating Point TIFF similar
![Page 87: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/87.jpg)
R di F t ( i hd )Radiance Format (.pic, .hdr)
32 bits / pixel
Red Green Blue Exponent
(145, 215, 87, 149) =
(145, 215, 87) * 2^(149-128) =
(145, 215, 87, 103) =
(145, 215, 87) * 2^(103-128) =
(1190000, 1760000, 713000) (0.00000432, 0.00000641, 0.00000259)
Ward, Greg. "Real Pixels," in Graphics Gems IV, edited by James Arvo, Academic Press, 1994
Richard Szeliski Computational Photography 87
Ward, Greg. Real Pixels, in Graphics Gems IV, edited by James Arvo, Academic Press, 1994
![Page 88: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/88.jpg)
ILM’ O EXR ( )ILM’s OpenEXR (.exr)6 bytes per pixel 2 for each channel compressed6 bytes per pixel, 2 for each channel, compressed
i t tisign exponent mantissa
• Several lossless compression options, 2:1 typicalCompatible with the “half” datatype in NVidia's Cg• Compatible with the “half” datatype in NVidia s Cg
• Supported natively on GeForce FX and Quadro FX• Available at http://www.openexr.net/
Richard Szeliski Computational Photography 88
![Page 89: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/89.jpg)
High Dynamic Range Video
Sing Bing Kang, Matt Uyttendaele, Simon Winder, Rick Szeliski
[SIGGRAPH’2003]
![Page 90: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/90.jpg)
HDR i dHDR images — merged
Pixel count
Richard Szeliski Computational Photography 90
Radiance
![Page 91: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/91.jpg)
Wh t b t ti ?What about scene motion?
Inputs Tonemapped outputInputs Tonemapped output(no compensation or consistency check)
Richard Szeliski Computational Photography 91
co s ste cy c ec )
![Page 92: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/92.jpg)
With ti tiWith motion compensation
Inputs Tonemapped outputInputs Tonemapped output(global+local compensation)
Richard Szeliski Computational Photography 92
![Page 93: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/93.jpg)
R i t ti ( l b l)Registration (global)
After global registration
Richard Szeliski Computational Photography 93
![Page 94: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/94.jpg)
R i t ti (l l)Registration (local)
After local registration
Richard Szeliski Computational Photography 94
![Page 95: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/95.jpg)
N Wh t?Now What?
Richard Szeliski Computational Photography 95
![Page 96: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/96.jpg)
Tone MappingTone Mapping
![Page 97: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/97.jpg)
T M iTone Mapping
Ho can e do this?
10 6 106Hi h d i
How can we do this?Linear scaling?, thresholding? Suggestions?
10-6 106Real WorldRay Traced World (Radiance)
High dynamic range
10 6 106
World (Radiance)
10-6 106Display/Printer
Richard Szeliski Computational Photography 97
0 to 255
![Page 98: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/98.jpg)
Si l Gl b l O tSimple Global OperatorCompression curve needs toCompression curve needs to
• Bring everything within range• Leave dark areas alone
In other wordsIn other words
• Asymptote at 255• Derivative of 1 at 0
Richard Szeliski Computational Photography 98
![Page 99: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/99.jpg)
Gl b l O t (R i h t t l)Global Operator (Reinhart et al)
ld
worlddisplay L
LL
1 worldL1
Richard Szeliski Computational Photography 99
![Page 100: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/100.jpg)
Gl b l O t R ltGlobal Operator Results
Richard Szeliski Computational Photography 100
![Page 101: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/101.jpg)
Richard Szeliski Computational Photography 101Darkest 0.1% scaledDarkest 0.1% scaledto display deviceto display device
Reinhart OperatorReinhart Operator
![Page 102: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/102.jpg)
Wh t d ?What do we see?
Vs.
Richard Szeliski Computational Photography 102
![Page 103: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/103.jpg)
Wh t d th ?What does the eye sees?
The eye has a huge dynamic range
Richard Szeliski Computational Photography 103
Do we see a true radiance map?
![Page 104: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/104.jpg)
M tMetamores
Richard Szeliski Computational Photography 104
Can we use this for range compression?
![Page 105: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/105.jpg)
Fast bilateral filtering for the display of high-dynamic-rangedisplay of high-dynamic-range images
Frédo Durand and Julie Dorsey SIGGRAPH 2002.
![Page 106: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/106.jpg)
N ï G iNaïve: Gamma compressionX Xcolors are washed-out Why?X X colors are washed-out. Why?
Input Gamma
Richard Szeliski Computational Photography 106
![Page 107: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/107.jpg)
G i i t itGamma compression on intensityColors are OK details are blurredColors are OK, details are blurred
Gamma on intensityIntensity
Color
Richard Szeliski Computational Photography 107
![Page 108: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/108.jpg)
O h i 1968 Chi t l 1993Oppenheim 1968, Chiu et al. 1993Reduce contrast of low frequencies keep highReduce contrast of low-frequencies, keep high
Reduce low frequencyLow-freq.
i h fHigh-freq.
Color
Richard Szeliski Computational Photography 108
![Page 109: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/109.jpg)
H lHalosStrong edges contain high frequencyStrong edges contain high frequency
Reduce low frequencyLow-freq.
i h fHigh-freq.
Color
Richard Szeliski Computational Photography 109
![Page 110: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/110.jpg)
O hOur approachDo not blur across edges: non linear filteringDo not blur across edges: non-linear filtering
OutputLarge-scale
ilDetail
Color
Richard Szeliski Computational Photography 110
![Page 111: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/111.jpg)
Bil t l filtBilateral filter
Tomasi and Mand ci 1998Tomasi and Manduci 1998http://www.cse.ucsc.edu/~manduchi/Papers/ICCV98.pdf
Related toRelated to • SUSAN filter [Smith and Brady 95]
http://citeseer.ist.psu.edu/smith95susan.html• Digital-TV [Chan, Osher and Chen 2001]
http://citeseer.ist.psu.edu/chan01digital.htmli filt• sigma filter
http://www.geogr.ku.dk/CHIPS/Manual/f187.htm
Richard Szeliski Computational Photography 111
![Page 112: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/112.jpg)
St t ith G i filt iStart with Gaussian filtering
O tp t is bl rredOutput is blurred
J f IJ
Richard Szeliski Computational Photography 112
output input
![Page 113: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/113.jpg)
Bil t l filt i i liBilateral filtering is non-linear
The eights are different for each o tp t pi elThe weights are different for each output pixel
)(xJ ),( xf ))()(( xIIg )(I)(1xk )(xk
x xi
Richard Szeliski Computational Photography 113
output input
![Page 114: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/114.jpg)
Oth iOther view
The bilateral filter ses the 3D distanceThe bilateral filter uses the 3D distance
Richard Szeliski Computational Photography 114
![Page 115: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/115.jpg)
Contrast reductionOutputInput HDR image
Intensity Large scale Large scaleIntensity Large scaleReducecontrast
Large scale
DetailFastBilateral
DetailPreserve!
Color
Filter
Color
Richard Szeliski Computational Photography 115
![Page 116: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/116.jpg)
D i d tiDynamic range reduction
To red ce contrast of base la er• To reduce contrast of base layer• scale in the log domain exponent in linear
• Set a target range: log (5)• Set a target range: log10 (5)• Compute range in the log layer: (max-min)
D d i di i i• Deduce using division• Normalize so that the biggest value in the
(linear) base is 1 (0 in log):(linear) base is 1 (0 in log):• offset the compressed based by its max
Richard Szeliski Computational Photography 116
![Page 117: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/117.jpg)
S f hSummary of approachDo not blur base/gain layer: non linear filteringDo not blur base/gain layer: non-linear filtering
OutputLarge-scale
ilDetail
Color
Richard Szeliski Computational Photography 117
![Page 118: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/118.jpg)
Gradient domain high dynamic range compression
Raanan Fattal, Dani Lischinski, and Michael WermanSIGGRAPH 2002.
![Page 119: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/119.jpg)
G di t T M iGradient Tone Mapping
Richard Szeliski Computational Photography 119
Slide from Siggraph 2005 by Raskar (Graphs by Fattal et al.)
![Page 120: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/120.jpg)
G di t tt tiGradient attenuation
Richard Szeliski Computational Photography 120
From Fattal et al.
![Page 121: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/121.jpg)
Interactive Local Adjustment of Tonal Values
Dani LischinskiZeev Farbman
Matt UyttendaeleRick Szeliski
The Hebrew University Microsoft Research
SIGGRAPH 2006
![Page 122: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/122.jpg)
T l M i l tiTonal Manipulation•brightness•brightness•exposure•contrastcontrast•saturation•color temperature•…
Richard Szeliski Computational Photography 122
![Page 123: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/123.jpg)
Interpretation 1:Interpretation 1:
Richard Szeliski Computational Photography 123
![Page 124: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/124.jpg)
Interpretation 2:Interpretation 2:
Richard Szeliski Computational Photography 124
![Page 125: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/125.jpg)
Interpretation 3:Interpretation 3:
Richard Szeliski Computational Photography 125
![Page 126: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/126.jpg)
Thi W k i Ab tThis Work is About:
Ne tool for interacti e tonal manip lationNew tool for interactive tonal manipulation: developing negatives in the digital darkroom.
Target material:• HDR images: the ultimate digital negative• HDR images: the ultimate digital negative.• Camera RAW images: the most common digital
negative.g• Ordinary snapshots.
Richard Szeliski Computational Photography 126
![Page 127: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/127.jpg)
E i ti T lExisting Tools
A tomatic tone mapping algorithmsAutomatic tone mapping algorithms• Why do we need yet another tone mapping
approach?approach?• Why interactive rather than automatic?
Image manipulation and editing packages, e.g., Adobe Photoshop.
Richard Szeliski Computational Photography 127
![Page 128: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/128.jpg)
T R d ti O tTone Reproduction Operators
Bilateral Filtering Gradient Domain Photographic
Richard Szeliski Computational Photography 128Durand & Dorsey 2002 Fattal et al. 2002 Reinhard et al. 2002
![Page 129: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/129.jpg)
A t ti I t tiAutomatic vs. Interactive
Bilateral Filtering Interactive Tone Photographic
Richard Szeliski Computational Photography 129Durand & Dorsey 2002 Mapping Reinhard et al. 2002
![Page 130: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/130.jpg)
A t ti I t tiAutomatic vs. Interactive
E isting a tomatic TM operators areExisting automatic TM operators are“black boxes”• No direct control over the outcome• No direct control over the outcome• No local adjustment• Not suitable for creative/artistic work• Results do not always look “photographic”• Most operators not really automatic
Richard Szeliski Computational Photography 130
![Page 131: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/131.jpg)
B t Wh t Ab t Ph t h ?But What About Photoshop?
Yo can do j st abo t e er thingYou can do just about everything …Adjustment LayersL M kLayer Masks
• Select regions• Paint blending weights• Paint blending weights
but you need a lot of experience patience… but you need a lot of experience, patience, and time!
Richard Szeliski Computational Photography 131
![Page 132: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/132.jpg)
E lOur approach
Example15 minutes in Photoshop: 3 minutes:Our approach15 minutes in Photoshop: 3 minutes:
Richard Szeliski Computational Photography 132
![Page 133: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/133.jpg)
A hApproach
User indicates regions sing scribblesUser indicates regions using scribbles.User adjusts tonal values using sliders.
Scribbles + tonal values are interpreted as soft t i tconstraints.
Optimization framework “propagates” the constraints to the entire imageconstraints to the entire image.
Richard Szeliski Computational Photography 133
![Page 134: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/134.jpg)
U i t fUser interface
Richard Szeliski Computational Photography 134
![Page 135: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/135.jpg)
I t t i tInput: constraints+0 5 f t+0 5 f t+0.5 f-stops+0.5 f-stops
-1 0 f-stops-1 0 f-stops-1.0 f-stops-1.0 f-stops
+2.0 f-stops+2.0 f-stops
+1.2 f-stops+1.2 f-stops
Richard Szeliski Computational Photography 135
![Page 136: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/136.jpg)
R lt dj t tResult: adjustment map
Richard Szeliski Computational Photography 136
![Page 137: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/137.jpg)
C t i t P tiApproximate constraints with a function whose
Constraint PropagationApproximate constraints with a function whose
smoothness is determined by underlying image:
data termsmoothness
term
Our smoothness term:
data term term
Richard Szeliski Computational Photography 137
![Page 138: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/138.jpg)
I fl F tiInfluence Functions
Richard Szeliski Computational Photography 138
![Page 139: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/139.jpg)
I fl F tiInfluence Functions
Richard Szeliski Computational Photography 139
![Page 140: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/140.jpg)
A t ti I iti li tiAutomatic Initialization
Inspired b Ansel Adams’ “Zone S stem”Inspired by Ansel Adams’ “Zone System”.• Segment image (very crudely) into brightness
“zones”zones• Determine the desired exposure for each
zonezone• Let the image-guided optimization produce a
piecewise smooth exposure mappiecewise smooth exposure map
Richard Szeliski Computational Photography 140
![Page 141: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/141.jpg)
R lt A t ti dResults – Automatic mode
Richard Szeliski Computational Photography 141
![Page 142: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/142.jpg)
R lt A t ti M dResults – Automatic Mode
Richard Szeliski Computational Photography 142
![Page 143: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/143.jpg)
R lt A t ti dResults – Automatic mode
Richard Szeliski Computational Photography 143
![Page 144: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/144.jpg)
Richard Szeliski Computational Photography 144
![Page 145: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/145.jpg)
Richard Szeliski Computational Photography 145
![Page 146: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/146.jpg)
Richard Szeliski Computational Photography 146
![Page 147: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/147.jpg)
S h t E h tSnapshot Enhancement
Richard Szeliski Computational Photography 147
![Page 148: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/148.jpg)
S h t E h tSnapshot Enhancement
Richard Szeliski Computational Photography 148
![Page 149: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/149.jpg)
S ti ll V i t Whit B lSpatially Variant White Balance
Richard Szeliski Computational Photography 149
![Page 150: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/150.jpg)
C i f tComparison of tone mappers
D rand and Dorse Fast bilateral filtering for theDurand and Dorsey. Fast bilateral filtering for the display of high-dynamic-range images. SIGGRAPH 2002SIGGRAPH 2002.
Fattal, Lischinski, and Werman. Gradient domain high dynamic range compression. SIGGRAPHhigh dynamic range compression. SIGGRAPH 2002.
Li, Sharan, and Adelson. Compressing and , , p gCompanding High Dynamic Range Images with Subband Architectures. SIGGRAPH 2005.
Richard Szeliski Computational Photography 150
![Page 151: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/151.jpg)
Fattal et al. 2002Li et al. 2005
Lischinski et al. 2006
Reinhard et al. 2002 Durand & Dorsey 2002
Richard Szeliski Computational Photography 151
![Page 152: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/152.jpg)
Merging flash andnon-flash images
Georg Petschnigg, Maneesh Agrawala, Hugues Hoppe, Rick Szeliski,
Mi h l C h K t TMichael Cohen, Kentaro Toyama[SIGGRAPH’2004]
![Page 153: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/153.jpg)
Flash + non-flash imagesFlash non flash images
Flash photos ha e less noise more detailFlash photos have less noise, more detailNon-flash photos have better colorId th t thIdea: merge them together
• But how?
+ =
Richard Szeliski Computational Photography 153
non-flash flash merged
![Page 154: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/154.jpg)
Flash + non-flash imagesFlash non flash images
Smooth non flash photo sing flash photo’sSmooth non-flash photo using flash photo’s edge information
Add high frequency details from flash imageAdd high-frequency details from flash image
+ =
Richard Szeliski Computational Photography 154
non-flash flash merged
![Page 155: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/155.jpg)
J i t bil t l filtJoint bilateral filter
Richard Szeliski Computational Photography 155
![Page 156: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/156.jpg)
Bil t l d t il filtBilateral detail filter
Richard Szeliski Computational Photography 156
![Page 157: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/157.jpg)
Final resultFinal result
flashnon-flashmerged
Richard Szeliski Computational Photography 157
flashnon flashmerged
![Page 158: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/158.jpg)
![Page 159: C280, Computer Vision - Peopletrevor/CS280Notes... · • Biologically inspired filtering approaches (Bergen & Adelson, ... the primary engine for parsing an image. 11. ... Sensor](https://reader035.fdocuments.us/reader035/viewer/2022070916/5fb631d5fe2822508c7c1868/html5/thumbnails/159.jpg)
CodaImage Formation
Recognition Intro.
Topic ModelsImage Formation
Color
Filt
Topic Models
Recognition KernelsFilters
PyramidsVoting and Parts
ContextLocal Features
TextureArticulated Recognition
Photometric StereoTexture
Alignment
Fl
Photometric Stereo
TrackingFlow
StereoMRFS
SegmentationSFM
g
Comp. Photography