NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah...
-
date post
21-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of NlogN Entropy Optimization Sarit Shwartz Yoav Y. Schechner Michael Zibulevsky Sponsors: ISF, Dvorah...
NlogN Entropy Optimization
Sarit Shwartz
Yoav Y. Schechner
Michael Zibulevsky
Sponsors: ISF, Dvorah Foundation
1
Kernel Estimators: Parzen Windows
Data
True PDF
Estimated PDF
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
49
Previous Work
• Parametric PDF:Hyvärinen 98, Bell; Sejnowski 95, Pham; Garrat 97.
• Cumulants:Cardoso ; Souloumiac 93.
Not accurate
• Order statistics:Vasicek 76, Learned-Miller; Fisher 03.
• KD trees:Gray; Moore 03.
Previous Work
Not differentiable
Entropy Estimation
• Kernel Estimators: reduced complexityPham, 03, .Erdogmus; Principe; Hild, 03, Morejon; Principe 04,
Schraudolph 04, (Stochastic gradient).
Source Range: Continuous
Parzen Windows Estimator
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
50
• Minimization of Mutual Information
• Differentiable
• Computationally efficient
- Currently O( K N )
Independent Component Analysis
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
online code
(see website)
51
2 2
Convolution
Sampling
Parzen Windows as a Convolution
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
Wish it was … Discrete convolution
52
Efficient Kernel Estimator
A. Samples of
estimated sourcesA
PDF estimation Fan; Marron 94, Silverman 82.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
A. Samples of
estimated sources
B. Interpolation to
uniform grid
(histogram)
A
B
Efficient Kernel Estimator
PDF estimation Fan; Marron 94, Silverman 82.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
C
• Samples of estimated sources
• Interpolation to uniform grid(histogram)
• Discrete convolution with Parzen window
A
B
PDF estimation Fan; Marron 94, Silverman 82.
Efficient Kernel Estimator
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
53
D
Efficient Entropy Estimator
C
• Interpolation to original values
A
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
54
Can it be Used for Optimization?
Wseparate
• Iterations exploiting derivatives of .
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
55
Can it be Used for Optimization?
Wseparate
• Binning fluctuations of .
• Fluctuations amplified by differentiation.
• Fluctuations slow convergence, false minima.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
56
Function
Quantized function
Quantization and Optimization
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
57
Function
Quantized function
Function with a quantized derivative
Quantization and Optimization
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
57
Analytic Entropy Gradient
Accurate derivative
Efficient calculation
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
Complexity
Analytic Entropy Gradient
K- number of sources, N-data length.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
Entropy Gradient by Convolutions
Convolution
Convolution
Convolution
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
58
• Calculation of using convolutions.
• Approximation of convolutions with
complexity.
• Distinct quantization of the derivative.
Not differentiation of a quantized
function.
Entropy Gradient by Convolutions
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
59
K=6 random sources, N= 3000 samples.
AlgorithmSignal to Interference ratio [dB]
Time
Basic Non-param ICA
18 4760 min.
Our algorithm22 31.2 min.
Jade7 40.2 sec.
Infomax8 41.6 sec.
Fast ICA5 31.9 sec.
Super ICA performance
Parametric
algorithms.
Non-parametric
algorithms.
Shwartz, Schechner & Zibulevsky, NlogN entropy optimization
60