Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of...
-
Upload
kory-harrison -
Category
Documents
-
view
213 -
download
0
Transcript of Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of...
![Page 1: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/1.jpg)
Learning to Perceive Learning to Perceive Transparency from the Transparency from the
Statistics of Natural ScenesStatistics of Natural Scenes
Anat LevinSchool of Computer Science and Engineering
The Hebrew University of Jerusalem
Joint work with Assaf Zomet and Yair Weiss
![Page 2: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/2.jpg)
21 :layers two III 0 :layer one 1 II
Transparency
![Page 3: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/3.jpg)
),(),(),( 21 yxIyxIyxI
How does our visual system choose the right decomposition??
•Why not “simpler” one layer solution?
•Which two layers out of infinite possibilities?
![Page 4: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/4.jpg)
Talk Outlines
•Motivation and previous work
•Our approach
•Results and future work
![Page 5: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/5.jpg)
Transparency in the real world
“Fashion Planet's photographers have spent the last five years working to bring you clean photographs of the windows on New York especially without the reflections that usually occur in such photography”
http://www.fashionplanet.com/sept98/features/reflections/home
![Page 6: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/6.jpg)
Transparency and shading
),(),(),( 21 yxIyxIyxI ),(),(),( yxRyxLyxI
![Page 7: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/7.jpg)
Transparency in human vision
• Metelli's conditions (Metelli 74)
•T-junctions, X-junctions, doubly reversing junctions (Adelson and Anandan 90, Anderson 99)
Two layersOne layer
Not obvious how to apply “junction catalogs” to real images.
![Page 8: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/8.jpg)
Transparency from multiple frames
•Two frames with polarizer using ICA (Farid and Adelson 99, Zibulevsky 02)
•Multiple frames with specific motions (Irani et al. 94, Szeliski et al. 00, Weiss 01)
![Page 9: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/9.jpg)
Shading from a single frame
),(),(),( 21 yxIyxIyxI ),(),(),( yxRyxLyxI
•Retinex (Land and McCann 71).
•Color (Drew, Finlayson Hordley 02)
•Learning approach (Tappen, Freeman Adelson 02)
![Page 10: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/10.jpg)
Talk Outlines
•Motivation and previous work
•Our approach
•Results and future work
![Page 11: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/11.jpg)
Our Approach
Ill-posed problem.
Assume probability distribution Pr(I1), Pr(I2)
and search for most probable solution.
(ICA with a single microphone)
),(),(),( 21 yxIyxIyxI
![Page 12: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/12.jpg)
Statistics of natural scenes
Input image dx histogram dx Log histogram
![Page 13: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/13.jpg)
1 ,)( / sxexp
Statistics of derivative filters
Log histogram
Generalized Gaussian distribution (Mallat 89, Simoncelli 95)
Gaussian –x2
–x 1/2
-1
0
Log P
robab
ility
Laplacian –|x|
![Page 14: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/14.jpg)
Is sparsity enough?
= +
= +Or:
![Page 15: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/15.jpg)
Exactly the same derivatives exist in the single layer solution as in the two layers solution.
Is sparsity enough?
= +
= +Or:
![Page 16: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/16.jpg)
Beyond sparseness
• Higher order statistics of filter outputs (e.g. Portilla and Simoncelli 2000).
•Marginals of more complicated feature detectors (e.g. Zhu and Mumford 97, Della Pietra Della Pietra and Lafferty 96).
![Page 17: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/17.jpg)
Corners and transparency
•In typical images, edges are sparse.
•Adding typical images is expected to increase the number of corners.
•Not true for white noise
= +
![Page 18: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/18.jpg)
Harris-like operator
),(),(),(
),(),(),(),(det),( 2
2
00 yxIyxIyxI
yxIyxIyxIyxwyxc
yyx
yxx
![Page 19: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/19.jpg)
Derivative Filter Corner Operator
Corner histograms
![Page 20: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/20.jpg)
Fitting: 1/)( sxexp
0.7 0.2
Derivative Filter Corner Operator
Typical exponents for
natural images:
2/)( sxexp
1/ 21 ss
![Page 21: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/21.jpg)
Simple prior for transparency prediction
),(),(log),(log,
yxcyxIZyxPyx
The probability of a decomposition 21 III
),(log),(log),(log 2211yxyxyx IIPIIPIIP
1/ ,2.0 0.7, 21 ss
![Page 22: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/22.jpg)
),(),(log),(log,
yxcyxIZyxPyx
Does this predict transparency?
1II 1II
![Page 23: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/23.jpg)
How important are the statistics?
Is it important that the statistics are non Gaussian? Would any cost that penalized high gradients and corners work?
),(),(log),(log,
yxcyxIZyxPyx
1 ,2.0 0.7,
![Page 24: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/24.jpg)
The importance of being non Gaussian
2.0 0.7, 2 2,
1II 1II
),(),(log),(log,
yxcyxIZyxPyx
![Page 25: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/25.jpg)
The “scalar transparency” problem
Consider a prior over positive scalars
For which priors is the MAP solution sparse?
0,0th wi,1 baba
xexp )(
![Page 26: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/26.jpg)
The “scalar transparency” problem
0,0 with ,1 baba
Observation:
The MAP solution is obtained with a=0, b=1 or a=1, b=0 if and only if f(x)=log P(x) is convex.
0 10 10.5 0.5
)2
(ba
f
2
)()( bfaf 2
)()( bfaf
)2
(ba
f
MAP solution: a=0, b=1 MAP solution: a=0.5, b=0.5
![Page 27: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/27.jpg)
2.0 0.7, 2 2,
0 10.5
1II 1II
0 10.5
The importance of being non Gaussian
![Page 28: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/28.jpg)
1II 1II
Can we perform a global optimization??
),(),(log),(log,
yxcyxIZyxPyx
![Page 29: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/29.jpg)
Conversion to discrete MRF
Local Potential- derivative filters:
Pairwise Potential- pairwise approximation to the corner operator:
-Enforcing integrability
ii fgii eg )(
)det()det(, ),(
Tjj
Tii
Tjj
Tii ffffgggg
jiji egg
),,(,, kjikji ggg
1g 2g 3g 4g 5g
10g9g8g7g6g
11g 13g 14g 15g12g
For the decomposition:
,21 III
iiyixi
iyixi
gIIf
IIIg
),(
),,( 111 gradient at location i
![Page 30: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/30.jpg)
Conversion to discrete MRF
Local Potential- derivative filters:
Pairwise Potential- pairwise approximation to the corner operator:
-Integrability enforcing
For the decomposition: ,21 III iiyixiiyixi gIIfiIIIg ),( .location at gradient oftion discretiza ),,( 1
11
),,(),()(1
)(,,
,,,
, kjikji
kjijiji
jiii
i ggggggZ
gP
ii fgii eg )(
)det()det(, ),(
Tjj
Tii
Tjj
Tii ffffgggg
jiji egg
),,(,, kjikji ggg
![Page 31: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/31.jpg)
possible assignments.
Solution: use max-product belief propagation.
The MRF has many cycles but BP works in similar problems (Freeman and Pasztor 99, Frey et al 2001. Sun et al 2002).
Converges to strong local minimum (Weiss and Freeman 2001)
Optimizing discrete MRF ,21 III
.location at gradient oftion discretiza ,),(
.location at gradient oftion discretiza ),,(
2
111
iIgIIf
iIIIg
iiyixi
iyixi
),,(),()(1
)(,,
,,,
, kjikji
kjijiji
jiii
i ggggggZ
gP
Ng
![Page 32: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/32.jpg)
Drawbacks of BP for this problem
•Large memory and time complexity.
•Convergence depends on update order.
•Discretization artifacts
![Page 33: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/33.jpg)
Talk Outlines
•Motivation and previous work
•Our approach
•Results and future work
![Page 34: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/34.jpg)
Results
input Output layer 1 Output layer 2
![Page 35: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/35.jpg)
Results
input Output layer 1 Output layer 2
![Page 36: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/36.jpg)
Future Work
Original Non linear filter
•Dealing with a more complex texture
+ =
![Page 37: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/37.jpg)
Future Work •Dealing with a more complex texture:
•Use application specific priors (e.g. Manhattan World)
•Extend to shading and illumination.
•Applying other optimization methods.
•Learn discriminative features automatically•A coarse qualitative separation.
![Page 38: Learning to Perceive Transparency from the Statistics of Natural Scenes Anat Levin School of Computer Science and Engineering The Hebrew University of.](https://reader035.fdocuments.us/reader035/viewer/2022070402/56649f255503460f94c3bc3e/html5/thumbnails/38.jpg)
Conclusions•Natural scene statistics predict perception of transparency.
•First algorithm that can decompose a single image into the sum of two images.