Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · ·...
Transcript of Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · ·...
![Page 1: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/1.jpg)
Natural Image Statistics
Siwei Lyu
![Page 2: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/2.jpg)
vision and image
2
Vision is a process that produces from images of the external world a description that is useful to the viewer.
[Marr, 1982]
![Page 3: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/3.jpg)
vision and image
65×65 8-bit gray-scale images: 25665×65 ∼ 10105
seconds since big bang: ~ 1017
atoms in the universe: ∼10802
Vision is a process that produces from images of the external world a description that is useful to the viewer.
[Marr, 1982]
![Page 4: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/4.jpg)
... ...
3
![Page 5: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/5.jpg)
... ...
3
“natural image”
![Page 6: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/6.jpg)
... ... The distribution of natural images is complicated. Perhaps it is something like beer foam, which is mostly empty but contains a thin mesh-work of fluid which fills the space and occupies almost no volume. The fluid region represents those images which are natural in character.
[Ruderman, 1996]
3
“natural image”
![Page 7: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/7.jpg)
dependency in natural images
[Kersten, 1987]
4
1% deleted 40% deleted 100% deleted
structure = predictivity = redundancy = statistical dependency
![Page 8: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/8.jpg)
natural image statistics
■ natural images are a small subset in the image space
■ natural images have non-random structures that reflect regularities in the physical world
■ natural images as an ensemble can be studied by their common regular statistical properties
5
![Page 9: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/9.jpg)
6
biological vision“the [neurally] encoded image is a very partial representation of the light that arrives at the eye: there is only a narrow region of high visual acuity in the fovea; the dynamic range of the sensors is very small; and the representation of wavelet is very coarse. You would never buy a camera with such poor optics and coarse spatial encoding. Yet, the visual algorithms can interpret the properties of objects from this poor encoding”
- Brian Wandell, Foundation of Vision, 1995
![Page 10: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/10.jpg)
Retina
LGN
Visual!
cortex
biological vision
optic nerve
![Page 11: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/11.jpg)
Retina
LGN
Visual!
cortex
biological vision
optic nerve
![Page 12: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/12.jpg)
Retina
LGN
Visual!
cortex
biological vision
optic nerve
![Page 13: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/13.jpg)
Retina
LGN
Visual!
cortex
biological vision
optic nerve
![Page 14: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/14.jpg)
= ? + noise
(c)
(a)
(b)
!
image restoration
8
![Page 15: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/15.jpg)
= ? + noise
(c)
(a)
(b)
!
image restoration
8
![Page 16: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/16.jpg)
= ? + noise
(c)
(a)
(b)
!
image restoration
8
![Page 17: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/17.jpg)
surface perception
9
high skewness low skewness
[Motoyoshi etal., 2007]
![Page 18: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/18.jpg)
engineering applications• image compression
‣ e.g., JPEG, JPEG 2000
• noise and blur removal, inpainting, super-resolution‣ e.g., [Freeman etal. 2000; Roth & Black, 2005; Levin etal, 2009]
• texture synthesis ‣ e.g., [Heeger & Bergen, 1995; Zhu, Wu & Mumford, 2001; Portilla & Simoncelli,
2003]
• visual saliency ‣ e.g., [Itti etal, 2003; Gao & Vasconcelos, 2009]
• low level features for object/scene recognition ‣ e.g., [Oliva & Torrolba, 2001; Kouh & Poggio, 2009]
• and many more … …
10
![Page 19: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/19.jpg)
scope
■ statistical approach to the study of natural images• gray-scale static images
■ focus on concepts and their relations, but not on• specific mathematical/computational details• specific applications in biology/engineering
■ follow one particular theme of developments• statistical properties observed on ensembles of natural images• probabilistic models that capture such properties• image representations that simplify such properties
11
![Page 20: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/20.jpg)
scope
■ statistical approach to the study of natural images• gray-scale static images
■ focus on concepts and their relations, but not on• specific mathematical/computational details• specific applications in biology/engineering
■ follow one particular theme of developments• statistical properties observed on ensembles of natural images• probabilistic models that capture such properties• image representations that simplify such properties
12
![Page 21: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/21.jpg)
how natural images can be studied■ step 1: collect an image database
- find a lot of nice-looking images
13
[van Hateren & van der Schaaf, 1998]
![Page 22: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/22.jpg)
how natural images can be studied■ step 1: collect an image database
- find a lot of nice-looking images
■ step 2: choose an image representation- a language describe and a tool to probe these images
14
![Page 23: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/23.jpg)
image representations
■ encoder/decoder: information bottleneck• preservation of essential and relevant structures• special case: perfect reconstruction
15
imagerepresentation
encodertransform
decodertransform
![Page 24: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/24.jpg)
why representation matters?
16
![Page 25: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/25.jpg)
why representation matters?
■ example: numbers• Arabic: 123
• Roman: MCXXIII
• binary: 1111011
• English: one hundred and twenty three
• Japanese: 百二十三
16
![Page 26: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/26.jpg)
why representation matters?
■ example: numbers• Arabic: 123
• Roman: MCXXIII
• binary: 1111011
• English: one hundred and twenty three
• Japanese: 百二十三
■ operations• multiply by 10
• multiply by 4
16
![Page 27: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/27.jpg)
pixel representation
Computational Vision & Neuroscience Group
/73
Linear models of natural images
14
Pixel basis
= s1· + s2· + s3· + . . .
figure courtesy of M. Bethge17
![Page 28: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/28.jpg)
desiderata"
■ simplicity of the encoder/decoder transforms• linear transform is preferred
■ simplicity of the representation• e.g., reveal lower intrinsic dimension
18!"#$%&'(#)%"*#%*+,##-$"*.",/01)1
2)(,$&%*3'#)-$$-4561'",
7-8,1*.9:*;")<-$1)#0
=>
+?.*-8,@A/-*BCD
! !"#$%&'()*+&%$",-$%(%*.,/&01#,2%,3%(%1".&&45#)&2$#,2(+%5(1(%$&1%#21,%1-,%5#)&2$#,2$" !"#$#%&&'()*+,*-$)./0)$1*)*&/"2%$#/")/.)$1*),&/34()$1*0*)#5)"/)%--%0*"$)5$03,$30*)#")$1*)5*$)/.)-/#"$5
" 61//5#"2)%")%--0/-0#%$*)0/$%$#/")%&&/75)35)$/)3"8*#&)$1*)3"4*0&'#"2)5$03,$30*9):;/3),%")$1#"<)/.)$1#5)0/$%$#/")%5)=7%&<#"2)%0/3"4=)$1*)$10**>4#?*"5#/"%&)5*$()&//<#"2)./0)$1*)@*5$)8#*7-/#"$A
! 678%0(2%"&+*%3#25%$90"%925&.+:#2;%$1.9019.&<%=1%$&+&01$%(%.,1(1#,2%$90"%1"(1%),$1%,3%1"&%>(.#(?#+#1:%-#1"#2%1"&%5(1(%$&1%#$%.&*.&$&21&5%#2%1"&%3#.$1%3&-%5#)&2$#,2$%,3%1"&%.,1(1&5%5(1(
" !")/30)$10**>4#?*"5#/"%&),%5*()$1#5)?%')5**?)/.)&#$$&*)35*
" B/7*8*0()71*")$1*)4%$%)#5)1#21&')?3&$#4#?*"5#/"%&):CDE5)/.)4#?*"5#/"5A()$1#5)%"%&'5#5)#5)F3#$*)-/7*0.3&
![Page 29: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/29.jpg)
how natural images can be studied■ step 1: collect an image database
- find a lot of nice-looking images
■ step 2: choose an image representation- a language describe and a tool to probe these images
■ step 3: make observations of statistical properties- find something interesting and unexpected
19
![Page 30: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/30.jpg)
statistical observations■ pixel representation
- second-order pixel correlations
- scale invariance
■ frequency representation- power law distribution of power
■ band-pass filtered representation- heavy-tail non-Gaussian marginals
- sparsity of representationss
- strong higher-order dependency of nearby representationss
- decay of dependency with distance
■ many more ………….
20
![Page 31: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/31.jpg)
how natural images can be studied■ step 1: collect an image database
- find a lot of nice-looking images
■ step 2: choose an image representation- a language describe and a tool to probe these images
■ step 3: make observations of statistical properties- find something interesting and unexpected
■ step 4: devise a mathematical model for these observations
- give a concise description and/or an (formal) explanation why natural images have such properties
21
![Page 32: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/32.jpg)
how to construct model
all possible images
naturalimages
22
onion peeling
![Page 33: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/33.jpg)
how to construct model
images of same marginal stats
all possible images
naturalimages
22
onion peeling
![Page 34: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/34.jpg)
how to construct model
images of same marginal stats
images of same second order stats
all possible images
naturalimages
22
onion peeling
![Page 35: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/35.jpg)
how to construct model
images of same marginal stats
images of same second order stats
images of same higher order stats
all possible images
naturalimages
22
onion peeling
![Page 36: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/36.jpg)
from statistics to model
■ principle of maximum entropy [Jaynes, 1954]
• given a set of statistical constraints on data
• choose a probabilistic model with maximum entropy
• solution
λ is determined by c
23
E(f(x)) = c
p� = argmaxp
H(p)
p�(x) ∝ exp(−λf(x))
![Page 37: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/37.jpg)
maxEnt examples
■ constraint on range -> uniform■ matching mean -> exponential■ matching covariance -> Gaussian■ matching all singleton marginals -> factorial model
■ matching all clique marginals -> Markov random field
24
∀i, pi(xi) = qi(xi) ⇒ p�(�x) =�
i
qi(xi)
∀clique c, pc(�xc) = qc(�xc) ⇒ p�(�x) ∝ exp(−�
c
λc(�xc))
[Schneidman etal., 2003]
![Page 38: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/38.jpg)
■ maximum a posterior (MAP)
■ minimum mean squares error (MMSE)
Bayesian inference
25
xMAP = argmaxx
p(x|y) = argmaxx
p(y|x)p(x)
xMMSE = argminx�
�
x�x− x��2p(x|y)dx
=�x xp(y|x)p(x)dx�x p(y|x)p(x)dx
= E(x|y)
![Page 39: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/39.jpg)
how natural images can be studied■ step 1: collect an image database
- find a lot of nice-looking images
■ step 2: choose an image representation- a language describe and a tool to probe these images
■ step 3: make observations of statistical properties- find something interesting and unexpected
■ step 4: devise a mathematical model for these observations
- give a concise description and/or an (formal) explanation why natural images have such properties
■ step 5: improve the representation, go back to step 3
26
![Page 40: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/40.jpg)
desiderata"
■ simplicity of the encoder/decoder transforms• linear transform is preferred
■ simplicity of the representation• lower intrinsic dimension • simplified statistical structure
- reduce statistical dependency
27
![Page 41: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/41.jpg)
measure statistical dependency■ multi-information
- [Studeny and Vejnarova, 1998]
• non-negative with any density over x
• zero when p(x) is factorial
- elements of x are mutually independent
- justifies factorial models have maximum entropy with constraints on singleton marginal densities
28
I(�x) = DKL
�p(�x)
������
k
p(xk)
�
=�
k
H(xk)−H(�x)
![Page 42: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/42.jpg)
biology: efficient coding
29
Neural characterization
Ingredients:
• stimuli• response model• estimation method
Transform
retinaopticnerve
optic nerve has a channel capacity C
[Attneave, 1954; Barlow, 1961]
![Page 43: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/43.jpg)
biology: efficient coding
29
Neural characterization
Ingredients:
• stimuli• response model• estimation method
Transform
retinaopticnerve
coding efficiency [Attick, 1991]
E =H(�x)
C=
�i H(xi)
C
H(�x)�i H(xi)
=
�i H(xi)
C
�i H(xi)− I(�x)�
i H(xi)
optic nerve has a channel capacity C
[Attneave, 1954; Barlow, 1961]
![Page 44: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/44.jpg)
biology: efficient coding
29
Neural characterization
Ingredients:
• stimuli• response model• estimation method
Transform
retinaopticnerve
coding efficiency [Attick, 1991]
E =H(�x)
C=
�i H(xi)
C
H(�x)�i H(xi)
=
�i H(xi)
C
�i H(xi)− I(�x)�
i H(xi)
optic nerve has a channel capacity C
channel usageefficiency
[Attneave, 1954; Barlow, 1961]
![Page 45: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/45.jpg)
biology: efficient coding
29
Neural characterization
Ingredients:
• stimuli• response model• estimation method
Transform
retinaopticnerve
coding efficiency [Attick, 1991]
E =H(�x)
C=
�i H(xi)
C
H(�x)�i H(xi)
=
�i H(xi)
C
�i H(xi)− I(�x)�
i H(xi)
optic nerve has a channel capacity C
channel usageefficiency
codeefficiency
[Attneave, 1954; Barlow, 1961]
![Page 46: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/46.jpg)
biology: efficient coding
29
Neural characterization
Ingredients:
• stimuli• response model• estimation method
Transform
retinaopticnerve
coding efficiency [Attick, 1991]
E =H(�x)
C=
�i H(xi)
C
H(�x)�i H(xi)
=
�i H(xi)
C
�i H(xi)− I(�x)�
i H(xi)
optic nerve has a channel capacity C
channel usageefficiency
codeefficiency
efficient code- match channel marginals- independent
[Attneave, 1954; Barlow, 1961]
![Page 47: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/47.jpg)
dependency reduction
■ simplify modeling• if components of x are independent, the joint density of x can be
expressed as the product of marginals on each component• dimensionality reduction in the parameter space
■ parallel manipulation• if components of x are independent, each component can be
processed independently
■ parallel sampling
30
![Page 48: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/48.jpg)
closed loop
31
representations
statisticaldependencies
models
key question: where to put the complexity?
![Page 49: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/49.jpg)
closed loop
31
representations
statisticaldependencies
models
empiric
al observatio
n
key question: where to put the complexity?
![Page 50: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/50.jpg)
closed loop
31
representations
statisticaldependencies
models
maximum entropyem
pirical obser
vation
key question: where to put the complexity?
![Page 51: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/51.jpg)
closed loop
31
representations
statisticaldependencies
models
maximum entropyem
pirical obser
vation
dependency reduction
key question: where to put the complexity?
![Page 52: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/52.jpg)
pixel - marginal distributions
32
![Page 53: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/53.jpg)
I(x,y)
I(x+1
,y)
I(x,y)I(x
+2,y
)I(x,y)
I(x+4
,y)
10 20 30 400
1
Spatial separation (pixels)
Corre
lation
a. b.
I(x,y)
I(x+1
,y)
I(x,y)
I(x+2
,y)
I(x,y)
I(x+4
,y)
10 20 30 400
1
Spatial separation (pixels)
Corre
lation
a. b.
pixel - second order correlation
33
second-ordercorrelation
![Page 54: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/54.jpg)
Gaussian model
■ assume zero mean and match second order statistics• covariance matrix
■ maximum entropic model is Gaussian
■ extension: Gaussian Markov randomfield for large images- specified by the inverse covariance (precision/structure) matrix
Σ = E(�x�xT )
p(�x) ∝ exp�−1
2�xT Σ−1�x
�
34
![Page 55: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/55.jpg)
Bayesian denoising
■ additive white Gaussian noise• likelihood
■ prior model
■ posterior density (another Gaussian)
■ solution: Wiener filter
p(�x) ∝ exp�−1
2�xT Σ−1�x
�
p(�x|�x) ∝ exp�−1
2�xT Σ−1�x− ��x− �y�2
2σ2w
�
�xMAP = �xMMSE = Σ(Σ + σ2wI)−1�y
�y = �x + �w
p(�y|�x) ∝ exp[−��y − �x�2/2σ2w]
35
![Page 56: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/56.jpg)
PCA representation
■ Gaussians only have second-order dependency
■ minimum (independent) when Σ is diagonal - Hadamard’s inequality
■ a transform that diagonalizes Σ can eliminate all dependencies (second-order)
■ result: principal component analysis (PCA)
I(�x) ∝d�
i=1
log(Σ)ii − log det(Σ)
36
![Page 57: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/57.jpg)
PCA■ eigen-decomposition of covariance
- U: orthonormal matrix (rotation)- Λ: diagonal matrix of eigenvalues
- covariance becomes diagonal
- independent Gaussian, if x is Gaussian- no correlation, if x is from arbitrary source
Σ = UΛUT
37
�x
�xpca
E{�xpca�xTpca}
= UTE{�x�xT }U= UTUΛUTU = Λ
�xpca = UT�x
![Page 58: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/58.jpg)
38
PCA basis from image patches
U
![Page 59: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/59.jpg)
whitening
■ making the PCA representation isotropicin variances
- V is an orthonormal matrix (rotation)
- isotropic Gaussian, if x is Gaussian- whitened, if x is from arbitrary source- whitening transform is not unique
39
�x
�xpca
�xwht
E{�xwht�xTwht}
= V Λ−1/2UTE{�x�xT }UΛ−1/2V T
= V Λ−1/2UTUΛUTUΛ−1/2V T = I
�xwht = V Λ− 12 �xpca = V Λ− 1
2UT�x
![Page 60: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/60.jpg)
ZCA whitening
■ zero-phase component analysis [Bell & Sejnowski, 1996]
• choose V = U, the result is a symmetric linear transform• minimizing squared distortion between data and representation
- minimum wiring length principle [Vincent & Baddeley, 2003]
• similar to the center-surround receptive fields for retina gangalion cells
40
�xzca = UΛ− 12UT�x
![Page 61: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/61.jpg)
fixed transform
■ translation invariance
- circular boundary handling■ covariance matrix Σ is a circulant matrix■ example:
41
cov(I(x, y), I(x +�x, y +�y)) = cov(I(0, 0), I(�x,�y))
1 2 3 4 55 1 2 3 44 5 1 2 33 4 5 1 22 3 4 5 0
![Page 62: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/62.jpg)
Fourier representation
■ Fourier transform diagonalizes the circulant covariance matrix • discrete Fourier transform basis are eigenvectors• Fourier transform of the circulant kernel are the eigenvalues
■ DFT is the eigen-system for translational invariant ensembles of images with circular boundary condition• question: why complex-valued?
42
Σ = circ(�v) = F diag(F∗�v)F∗
![Page 63: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/63.jpg)
Fourier - marginal■ spectral power
[Field, 1994]
figure from [Simoncelli, 2005]
43
F (sω) = spF (ω)
F (ω) =A
ωγ
![Page 64: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/64.jpg)
scale invariance of image variance
44
E( )=1/4 E( )
Spectral power
Structural:
F (sω) = spF (ω)
F (ω) ∝ 1ωp
[Ritterman 52; DeRiugin 56; Field 87; Tolhurst 92; Ruderman/Bialek 94; ...]
Assume scale-invariance:
then:
0 1 2 30
1
2
3
4
5
6
Log10
spatialfrequency (cycles/image)
Lo
g 10 p
ow
er
Empirical:
![Page 65: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/65.jpg)
applications
■ denoising (Wiener filter in frequency domain)■ JPEG compression■ Dolby noise reduction
F (ω) =A
ωγ
45
signal whiten noisychannel
noisereduction
unwhiten
![Page 66: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/66.jpg)
not sufficient
46
not natural image not independent noise
[Simoncelli and Olshausen, 2001]
sample from power law Gaussian sample
natural image after whitening
![Page 67: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/67.jpg)
structures in phases
[Oppenheim & Lim, 1981]
magnitudes
phases
47
![Page 68: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/68.jpg)
dependency
48
I(�x) =d�
k=1
log(Σkk)− log |Σ|
+ DKL (p(�x) � G(�x) )−d�
k=1
DKL (p(xk) � G(xk) )
second-orderdependency
higher-order dependency
![Page 69: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/69.jpg)
summary
■ pixel domain matching second-order statistics leads to Gaussian image models
■ eliminating dependencies in Gaussian models leads to PCA/whitening based representations
■ extending PCA to global image domain leads to frequency domain representations
■ Gaussian model + PCA representations are not sufficient to model natural images• higher-order statistical dependencies not being captured
49
![Page 70: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/70.jpg)
band pass filters
■ localize in space and frequency■ reduce low-frequency components
50
PCA ZCA random
![Page 71: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/71.jpg)
bandpass filter domain
⊗ =
51
band-passfilter
![Page 72: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/72.jpg)
■ marginal density
[Burt&Adelson 82; Field 87; Mallat 89; Daugman 89, ...]
band-pass filter domain
log
p(x)
0
52
Gaussian
natural image
![Page 73: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/73.jpg)
marginal model
■ well fit with generalized Gaussian
Marginal densities
P (x) ! exp"|x/s|p
[Mallat 89; Simoncelli&Adelson 96; Moulin&Liu 99; ...]
Well-fit by a generalized Gaussian:
Wavelet coefficient value
log
(Pro
ba
bili
ty)
p = 0.46
!H/H = 0.0031
Wavelet coefficient value
log
(Pro
ba
bili
ty)
p = 0.58
!H/H = 0.0011
Wavelet coefficient valuelo
g(P
rob
ab
ility
)
p = 0.48
!H/H = 0.0014
Wavelet coefficient value
log
(Pro
ba
bili
ty)
p = 0.59
!H/H = 0.0012
Fig. 4. Log histograms of a single wavelet subband of four example images (see Fig. 1 for image description). For eachhistogram, tails are truncated so as to show 99.8% of the distribution. Also shown (dashed lines) are fitted model densitiescorresponding to equation (3). Text indicates the maximum-likelihood value of p used for the fitted model density, andthe relative entropy (Kullback-Leibler divergence) of the model and histogram, as a fraction of the total entropy of thehistogram.
non-Gaussian than others. By the mid 1990s, a numberof authors had developed methods of optimizing a ba-sis of filters in order to to maximize the non-Gaussianityof the responses [e.g., 36, 4]. Often these methods oper-ate by optimizing a higher-order statistic such as kurto-sis (the fourth moment divided by the squared variance).The resulting basis sets contain oriented filters of differentsizes with frequency bandwidths of roughly one octave.Figure 5 shows an example basis set, obtained by opti-mizing kurtosis of the marginal responses to an ensembleof 12 ! 12 pixel blocks drawn from a large ensemble ofnatural images. In parallel with these statistical develop-ments, authors from a variety of communities were devel-oping multi-scale orthonormal bases for signal and imageanalysis, now generically known as “wavelets” (see chap-ter 4.2 in this volume). These provide a good approxima-tion to optimized bases such as that shown in Fig. 5.
Once we’ve transformed the image to a multi-scalewavelet representation, what statistical model can we useto characterize the the coefficients? The statistical moti-vation for the choice of basis came from the shape of themarginals, and thus it would seem natural to assume thatthe coefficients within a subband are independent andidentically distributed. With this assumption, the modelis completely determined by the marginal statistics of thecoefficients, which can be examined empirically as in theexamples of Fig. 4. For natural images, these histogramsare surprisingly well described by a two-parameter gen-eralized Gaussian (also known as a stretched, or generalizedexponential) distribution [e.g., 31, 47, 34]:
Pc(c; s, p) =exp("|c/s|p)
Z(s, p), (3)
where the normalization constant is Z(s, p) = 2 sp!( 1
p ).An exponent of p = 2 corresponds to a Gaussian den-sity, and p = 1 corresponds to the Laplacian density. In
Fig. 5. Example basis functions derived by optimizing amarginal kurtosis criterion [see 35].
5
[Mallat 89; Simoncelli&Adelson 96; Moulin&Liu 99; …]
p(s) ∝ exp�− |s|p
σ
�
53
![Page 74: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/74.jpg)
Gaussian scale mixtures
- u: zero mean Gaussian with unit variance- z: positive random variable- different p(z)
generalized Gaussian, Student’s t, Bessel’s K, Cauchy, α-stable, etc
p(x)
x
p(x)
x
[Andrews & Mallows 74, Wainwright & Simoncelli, 99]
x = u√
z
54
![Page 75: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/75.jpg)
factorial model
enforce consistency on singleton marginal densities, i.e., p(xi) = qi(xi), maximum entropic density is the factorial density
p(�x) =�d
i=1 qi(xi)
55
H(�x) =�
i H(xi)− I(�x)
maximum entropy
![Page 76: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/76.jpg)
Bayesian denoising - coringII. BLS for non-Gaussian prior
• Assume marginal distribution [Mallat ‘89]:
• Then Bayes estimator is generally nonlinear:
P (x) ! exp"|x/s|p
p = 2.0 p = 1.0 p = 0.5
[Simoncelli & Adelson, ‘96]56
![Page 77: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/77.jpg)
[Simoncelli & Adelson, 1996]
original image noise (SNR = 9dB)
Wiener filter (11.88dB) coring (13.82dB)
57
![Page 78: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/78.jpg)
dependencies
■ band-pass filtered representationss of natural images are not independent
58
pyramidspyramids
![Page 79: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/79.jpg)
LTF model• linearly transformed factorial (LTF)• each component in x is a linear mixing of independent
super-Gaussian sources, so they are not independent
• A is an invertible linear transform (basis), A-1 are the encoding transform
p(�s) =�d
i=1 p(si)
�x = A�s =
| · · · |
�a1 · · · �ad
| · · · |
s1...sd
= s1�a1 + · · · + sd�ad
59
![Page 80: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/80.jpg)
LTF model - generative view■ SVD of matrix A:
- U,V: orthonormal matrices (rotation) - Λ: diagonal matrix (Λii)1/2 ≥ 0 -- singular value
s x = U Λ1/2VTs
rotation scale rotation
A = UΛ1/2V T
VTs Λ1/2VTs
60
![Page 81: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/81.jpg)
representation■ independent component analysis (ICA)
[Comon 94; Cardoso 96; Bell/Sejnowski 97; …]
• many different implementations - JADE, InfoMax, FastICA, etc.
■ interpretation using SVD
• U and Λ obtained from PCA
61
�xica = A−1�x = V Λ−1/2UT�x
E{�x�xT } = AE{�xica�xTica}AT
= UΛ1/2V T IV Λ1/2UT
= UΛUT
independentcomponents
are decorrelated
![Page 82: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/82.jpg)
ICA■ ICA can be seen as a whitening operation
■ how to find the last rotation V
rotation scale rotation
62
�xica = A−1�x = V Λ−1/2UT�x
PCA?? whitening
![Page 83: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/83.jpg)
■ minimizing multi-information
• for super-Gaussian densities, lower kurtosis suggests lower entropy
Computational Vision & Neuroscience Group
/73
Higher-order redundancy reduction:Independent Component Analysis (ICA)
Find the most non-Gaussian directions:
48
search for the last rotation in ICA
63
I(�x) =�
k
H(xk)−H(�x)
not changed by rotation
minimize singletonentropy
![Page 84: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/84.jpg)
PCA/whiteningICA/whitening
�x
�xwht = Λ−12 UT �x
64
�xpca = UT�x
�xica = V Λ− 12UT�x
![Page 85: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/85.jpg)
similar to the receptive field of V1 simple cells [Olshausen & Field 1996, Bell & Sejnowski 1997]
65
ICA basis from image patches
![Page 86: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/86.jpg)
ICA basis
■ approximated by Gabor functions• localized in space/frequency• orientation preference
■ connection with wavelet66
![Page 87: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/87.jpg)
linear representations
67
spatialdomain
frequencydomain
pixel Fourier Gabor
![Page 88: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/88.jpg)
wavelet
■ developed in parallel with the ICA methodology- [Burt & Adelson, 1981; Mallat, 1989]
■ data independent • implemented as filter banks
■ wavelet filters are similar to those found by ICA• localized in space/time• orientation selective
■ applicable to whole image• incorporate scale invariance with multi-scale pyramid structure
68
![Page 89: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/89.jpg)
pyramid
69
figure courtesy of Jeremy Freeman
![Page 90: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/90.jpg)
pyramidspyramid
69
figure courtesy of Jeremy Freeman
![Page 91: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/91.jpg)
pyramidspyramidspyramid
69
figure courtesy of Jeremy Freeman
![Page 92: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/92.jpg)
pyramidspyramids pyramidspyramid
69
figure courtesy of Jeremy Freeman
![Page 93: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/93.jpg)
application
■ ICA and wavelet methodology brings forth a revolutionary breakthrough for image processing and computer vision, for every application there is a significant improvement in performance• compression • denoising• image features• texture synthesis• … ...
70
![Page 94: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/94.jpg)
LTF also a weak model...
Sample Gaussianized
Sample ICA-transformed
and Gaussianized
figure courtesy of Eero Simoncelli
sample from LTF model+ wavelet representation
natural images after filtered with ICA basis
71
not sufficient
not natural image not independent noise
![Page 95: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/95.jpg)
problems with LTF/ICA
■ any band-pass filter will lead to heavy tail marginals• even random ones
■ according to LTF model, random projection (filtering) should look like Gaussian• central limit theorem
72
![Page 96: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/96.jpg)
[Bethge, 06; Lyu & Simoncelli, 09]
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits
/co
eff)
Separation
raw
pca/ica
73
dependency reduction of ICAICA reduces less than 5% of statistical dependency
compared to PCA on natural images
![Page 97: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/97.jpg)
summary
■ in band-pass filter domain, natural images have• non-Gaussian marginal distributions• higher-order dependency
■ statistical properties lead to LTF model ■ LTF model leads to ICA/wavelet representations■ not sufficient to describe natural images
74
![Page 98: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/98.jpg)
pyramids
joint density of natural image band-pass filter representationss
with separation of 2 pixels
75
pyramidsproblem - joint density
[Wegmann & Zetzsche, 1990; Baddeley, 1996; Simoncelli, 1997]
![Page 99: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/99.jpg)
elliptically symmetric density
pesd(�x) =1
α|Σ| 12f
�−1
2�xT Σ−1�x
�pssd(�x) =
1α
f
�−1
2�xT �x
�
spherically symmetric density
whitening
(Fang et.al. 1990)
![Page 100: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/100.jpg)
![Page 101: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/101.jpg)
![Page 102: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/102.jpg)
d = 2 d = 16 d = 32
raw
ica
ss
fac
kurt
4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !
Figure 4: Contour plots of joint histograms of pairs of bandpass filter responses and theirtransforms from the “boats” image with di!erent spatial separations (given in units ofpixels). See text for details.
16
d = 2 d = 16 d = 32
raw
ica
ss
fac
kurt
4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !
Figure 4: Contour plots of joint histograms of pairs of bandpass filter responses and theirtransforms from the “boats” image with di!erent spatial separations (given in units ofpixels). See text for details.
16
d = 2 d = 16 d = 32
raw
ica
ss
fac
kurt
4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !
Figure 4: Contour plots of joint histograms of pairs of bandpass filter responses and theirtransforms from the “boats” image with di!erent spatial separations (given in units ofpixels). See text for details.
16
d = 2 d = 16 d = 32
raw
ica
ss
fac
kurt
4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !
Figure 4: Contour plots of joint histograms of pairs of bandpass filter responses and theirtransforms from the “boats” image with di!erent spatial separations (given in units ofpixels). See text for details.
16
d = 2 d = 16 d = 32
raw
ica
ss
fac
kurt
4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !4
6
8
10
12
0 !/4 !/2 3!/4 !
Figure 4: Contour plots of joint histograms of pairs of bandpass filter responses and theirtransforms from the “boats” image with di!erent spatial separations (given in units ofpixels). See text for details.
16
raw pairs whitened sphericalized factorialized
kurt
osis
![Page 103: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/103.jpg)
3 6 9 12 15 18 200
0.05
0.1
0.15
0.2
kurtosis
blk size = 3x3
blksphericalfactorial
Spherical vs LTF
3x3 7x7 15x15
3 6 9 12 15 18 200
0.05
0.1
0.15
0.2
kurtosis
blk size = 7x7
blksphericalfactorial
3 6 9 12 15 18 200
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
kurtosis
blk size = 11x11
blksphericalfactorial
data (ICA’d): factorialized:sphericalized:
• Histograms, kurtosis of projections of image blocks onto random
unit-norm basis functions.
• These imply data are closer to spherical than factorial
[Lyu & Simoncelli 08]
![Page 104: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/104.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
![Page 105: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/105.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
![Page 106: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/106.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
![Page 107: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/107.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
![Page 108: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/108.jpg)
Gaussian is the only density that can be both factorial and spherically symmetric [Nash and Klamkin 1976]
p(�x) =1�
(2π)dexp
�−�xT �x
2
�
=1�
(2π)dexp
�−1
2
d�
i=1
x2i
�
=d�
i=1
1√2π
exp�−1
2x2
i
�
=d�
i=1
p(xi)
85
![Page 109: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/109.jpg)
PCA/whitening
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
86
![Page 110: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/110.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
PCA/whitening
ICA
87
![Page 111: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/111.jpg)
EllipticalLinearly
transformed factorial
Factorial Gaussian Spherical
PCA/whitening
???ICA
88
![Page 112: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/112.jpg)
PCAICA
![Page 113: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/113.jpg)
PCAICA
![Page 114: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/114.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representation
■ one signal corresponds to one representation
■ one representation corresponds to one signal
■ representationss are mutually independent
91
�x = A�s
![Page 115: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/115.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representation
■ one signal corresponds to one representation
■ one representation corresponds to one signal
■ representationss are mutually independent
91
completelinear sys
�x = A�s
![Page 116: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/116.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representation
■ one signal corresponds to one representation
■ one representation corresponds to one signal
■ representationss are mutually independent• explicitly modeling dependencies in representation
92
![Page 117: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/117.jpg)
complete representation
■ independent subspace analysis and topographic ICA- [Hyvarinen & Hoyer, 2000; Hyvarinen, Hoyer & Inki, 2000]
■ hierarchical models- e.g., [Karklin & Lewicki, 2003, 2009; Ranzato & Hinton, 2010; … … ]
■ joint GSM model for wavelet coefficients- [Wainwright & Simoncelli, 1999; Portilla etal., 2003]
■ MRF models for wavelet coefficients- e.g., [Crouse etal, 1999; Lyu & Simoncelli, 2008; Lyu, 2009; … … ]
■ tree dependent component analysis- [Zoran & Weiss, 2009]
93
![Page 118: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/118.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representation
■ one signal corresponds to one representation
■ one representation corresponds to one signal• multiple representation can lead to one signal
■ representation are mutually independent
94
![Page 119: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/119.jpg)
■ achieving sparsity can be a driving principle itself• over-complete sparse coding (nonlin encoding, lin decoding)
‣ [Olshausen & Field, 1996]
• compressed sensing (lin encoding, nonlin decoding)‣ [Candes & Donoho, 2003]
• PCA/whitening/ICA (lin encoding, lin decoding)
95
![Page 120: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/120.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representation
■ one signal corresponds to one representation• focusing on the analysis
■ one representation corresponds to one signal
■ representations are mutually independent
96
�s = B�x
![Page 121: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/121.jpg)
maximum entropy models
■ use representationss as constraints to build statistical models• patch models
- product of experts [Teh etal., 2003]
- product of edgeperts [Gehler and Welling, 2006]
• image models- FRAME [Zhu, Wu & Mumford, 2001]
- field of experts [Roth & Black, 2005]
97
![Page 122: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/122.jpg)
assumptions of LTF/ICA
■ linear transform between signal and representations • find nonlinear encoding/decoding transforms
■ one signal corresponds to one representations
■ one representations corresponds to one signal
■ representationss are mutually independent
98
![Page 123: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/123.jpg)
PCAICA
?
99
![Page 124: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/124.jpg)
radial Gaussianization (RG)
[Lyu & Simoncelli, 2009]100
![Page 125: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/125.jpg)
101
radial Gaussianization (RG)
[Lyu & Simoncelli, 2009]
![Page 126: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/126.jpg)
pχ(r) ∝ r exp(−r2/2)
pr(r) ∝ rf(−r2/2)
102
radial Gaussianization (RG)
[Lyu & Simoncelli, 2009]
![Page 127: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/127.jpg)
g(r) = F−1χ Fr(r)
pχ(r) ∝ r exp(−r2/2)
pr(r) ∝ rf(−r2/2)
103
radial Gaussianization (RG)
[Lyu & Simoncelli, 2009]
![Page 128: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/128.jpg)
�xrg =g(��xwht�)��xwht�
�xwht
g(r) = F−1χ Fr(r)
pχ(r) ∝ r exp(−r2/2)
pr(r) ∝ rf(−r2/2)
104
radial Gaussianization (RG)
[Lyu & Simoncelli, 2009]
![Page 129: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/129.jpg)
105
![Page 130: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/130.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 131: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/131.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 132: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/132.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 133: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/133.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 134: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/134.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 135: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/135.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 136: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/136.jpg)
1 2 4 8 16 320
0.1
0.2
0.3
0.4
0.5
MI
(bits/c
oe
ff)
Separation
!
raw
pca/ica
rg
106
![Page 137: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/137.jpg)
0.2 0.3 0.4 0.5 0.6
0.2
0.3
0.4
0.5
0.6
0.7blk size = 3x3
0.6 0.7 0.8 0.9 1 1.1
0.6
0.7
0.8
0.9
1
1.1
1.2
1.3 blk size = 15x15
IPCA − IRAW IPCA − IRAW
blocks of local mean removed pixel blocks of natural images
(+)IRG − IRAW
(◦)IICA − IRAW
107
![Page 138: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/138.jpg)
PCAICA RG
unification asGaussianization?
108
![Page 139: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/139.jpg)
p(x)
x
p(y)
y
109
marginal Gaussianization
![Page 140: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/140.jpg)
110
divisive normalization
■ nonlinear transform
Traditional V1 models
Simple cell
Complex cell +
Traditional V1 models
Simple cell
Complex cell +stim
uli
... ...
Traditional V1 models
Simple cell
Complex cell +
+
... ...
Traditional V1 models
Simple cell
Complex cell +
/
resp
onse
s
�s =��x��
α+ ��x�2�x
��x�
radialfunction
![Page 141: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/141.jpg)
divisive normalization
■ observation• visual cortex [Heeger, 1991]• retina/LGN [Caradini etal. 2008]• auditory [Schwartz & Simoncelli, 1999]• olfactory [Wilson etal, 2010]
■ underlying principle• dynamic gain control• dependency reduction [Schwartz & Simoncelli, 2001]
111
![Page 142: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/142.jpg)
112
divisive normalization
■ nonlinear transform
Traditional V1 models
Simple cell
Complex cell +
Traditional V1 models
Simple cell
Complex cell +stim
uli
... ...
Traditional V1 models
Simple cell
Complex cell +
+
... ...
Traditional V1 models
Simple cell
Complex cell +
/
resp
onse
s
�s =��x��
α+ ��x�2�x
��x�
radialfunction
![Page 143: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/143.jpg)
divisive normalization
■ comparing the two radial transforms
113
0 5 10 150
1
2
3
4
5
IDN = 0.1741 bits/coeff IRG = 0.2614 bits/coeff
RGDN
![Page 144: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/144.jpg)
114
summary
■ in band-pass filter domain, we observe• non-Gaussian marginal densities• elliptically symmetric joint densities
■ observations lead to elliptically symmetric models■ ESD models lead to nonlinear radial Gaussianization■ extended to Lp elliptically symmetric models
- [Sinz & Bethge, 2009]
■ not sufficient• not effective for longer-range dependencies
![Page 145: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/145.jpg)
next step: building hierarchies
■ hierarchical representations• iterative Gaussianization/hierarchical ICA/bio-inspired
- [Chen & Gopinath, 2000; Shan etal, 2007; Karklin & Lewicki, 2003; Serre & Poggio, 2006]
■ hierarchical model• DBN type models, convolutional net
115
![Page 146: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/146.jpg)
summary
■ natural images are special in the space of all possible images and have regular statistical properties
■ these properties can be captured using representation and statistical models• dependency reduction• maximum entropy with constraints
■ key question: where to put the complexity
116
![Page 147: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/147.jpg)
afterthoughts
■ are the observed properties real or results of “artifacts of the lens through which we view the data”
117
![Page 148: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/148.jpg)
we believe but cannot prove ...
■ there is a probabilistic model over natural images in the space of all images of a give size
118
p(x is a natural image) = 0.87
p(x)
![Page 149: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/149.jpg)
we believe but cannot prove ...
■ there is a probability measure over natural images in the space of all images of a give size
■ this probability measure has invariance• translation invariance (a.k.a., stationary, homogeneous)
- marginal densities haveno dependency with spatial locations
119
…….
![Page 150: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/150.jpg)
we believe but cannot prove ...
■ there is a probability measure over natural images in the space of all images of a give size
■ this probability measure has invariance• translation invariance (a.k.a., stationary, homogeneous)
- marginal densities haveno dependency with spatial locations
- joint densities have nodependency with spatiallocations
120
…….
![Page 151: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/151.jpg)
we believe but cannot prove ...
■ there is a probability measure over natural images in the space of all images of a give size
■ this probability measure has invariance• translation invariance (a.k.a., stationary, homogeneous)
- marginal densities haveno dependency with spatial locations
- joint densities have nodependency with spatiallocations
- practical issue: proper boundary handling
121
…….
![Page 152: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/152.jpg)
we believe but cannot prove ...
■ there is a probability measure over natural images in the space of all images of a give size
■ this probability measure has invariance• translation invariance (a.k.a., stationary, homogeneous)• (empirical) ergodic
- ensemble average = spatialaverage
- ensemble marginal = spatialmarginal
122
…….
![Page 153: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/153.jpg)
we believe but cannot prove ...
■ there is a probability measure over natural images in the space of all images of a give size
■ this probability measure has invariance• translation invariance (a.k.a., stationary, homogeneous)• (empirical) ergodic
- ensemble average = spatialaverage
- ensemble joint = spatialjoint
123
…….
![Page 154: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/154.jpg)
afterthoughts
124
!15 !10 !5 0 5 10
0
0.1
0.2
0.3
0.4
!15 !10 !5 0 5 10
0
0.1
0.2
0.3
0.4
sd=8
B)
sd=2
sd=1
sd=4
C)A)
[Baddeley 1996]
![Page 155: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/155.jpg)
afterthoughts
■ image specific model• CRF image models for denoising, directly model p(x|y)
- [Tappen etal., 2007; 2009]
• primary sketch model - [Guo, Zhu and Wu, 2007]
125
(a) input image I (b) sketch graph Ssk (c) sketchable image IΛsk
(d) texture regions SΛnsk (e) synthesized textures IΛnsk (f) synthesized image Isyn
Figure 2: An example of the primal sketch model. (a) An input image I. (b) The sketch graph Ssk
computed from the image I. Each vertex in the graph correspond to an image primitive shown in Figure 3.
These primitives are occluding patches rather than linear additive bases. (c) The sketchable part of the
image by aligning the primitives to the graph. (d) The remaining non-sketchable portion is segmented
into a small number of homogeneous texture regions. (e) Synthesized textures on these regions. (f) The
final synthesized image integrating seamlessly the sketchable and non-sketchable parts.
we synthesize a partial image IΛsk in (c) for the sketchable part of the image, where Λsk collects
the sketchable pixels. Clearly this corresponds to the structural part of the image. The remaining
textural part is said to be non-sketchable and is segmented into a small number of homogeneous
texture regions. Each region is shown by a grey level in (c) and statistics (histograms of responses
from 5-7 small filters) are extracted as the statistical summary. Then we synthesize textures
on these regions by simulating the Markov random field (MRF) models which reproduce the
statistical summaries in these regions. The MRF models interpolate the sketchable part of the
image. The non-sketchable part of the image is denoted as IΛnsk , where Λnsk collects the non-
sketchable pixels. The final synthesized image is shown in (f) which integrates seamlessly the
sketchable and non-sketchable parts.
A set of image primitives are constructed for modeling the structures in natural images.
4
![Page 156: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/156.jpg)
big question marks
■ what are natural images, anyway?
■
ironically, white noises are “natural” as they are the result of cosmic radiations
■ naturalness is subjective126
![Page 157: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/157.jpg)
natural!
image!
statistics
math
statisticsbiology
computer!
science
image!
processing
machine!
learning
computer!
vision
optimization
perception
neuro-!
science
signal!
processing
127
![Page 158: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/158.jpg)
want to know more?• D. L. Ruderman. The statistics of natural images. Network: Computation
in Neural Systems, 5:517–548, 1996.• E. P. Simoncelli and B. Olshausen. Natural image statistics and neural
representation. Annual Review of Neuroscience, 24:1193–1216, 2001.• S.-C. Zhu. Statistical modeling and conceptualization of visual patterns.
IEEE Trans. PAMI, 25(6), 2003.• A. Srivastava, A. B. Lee, E. P. Simoncelli, and S.-C. Zhu. On advances in
statistical modeling of natural images. J. Math. Imaging and Vision, 18(1):17–33, 2003.
• E. P. Simoncelli. Statistical modeling of photographic images. Handbook of Image and Video Processing. Academic Press, 2005.
• A. Hyvärinen, J. Hurri, and P. O. Hoyer. Natural Image Statistics: A probabilistic approach to early computational vision. Springer, 2009.
128
![Page 159: Natural Image Statistics - University of Torontoasamir/cifar/lyu_08_11_10_slides.… · · 2010-08-11character. [Ruderman, 1996] 3 “natural image” dependency in ... • gray-scale](https://reader031.fdocuments.us/reader031/viewer/2022030416/5aa1be6d7f8b9ac67a8c3109/html5/thumbnails/159.jpg)
129
thank you