NlogN Entropy Optimization

Sarit Shwartz

Yoav Y. Schechner

Michael Zibulevsky

Sponsors: ISF, Dvorah Foundation

1

Kernel Estimators: Parzen Windows

Data

True PDF

Estimated PDF

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

49

Previous Work

• Parametric PDF:Hyvärinen 98, Bell; Sejnowski 95, Pham; Garrat 97.

• Cumulants:Cardoso ; Souloumiac 93.

Not accurate

• Order statistics:Vasicek 76, Learned-Miller; Fisher 03.

• KD trees:Gray; Moore 03.

Previous Work

Not differentiable

Entropy Estimation

• Kernel Estimators: reduced complexityPham, 03, .Erdogmus; Principe; Hild, 03, Morejon; Principe 04,

Schraudolph 04, (Stochastic gradient).

Source Range: Continuous

Parzen Windows Estimator

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

50

• Minimization of Mutual Information

• Differentiable

• Computationally efficient

- Currently O( K N )

Independent Component Analysis

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

online code

(see website)

51

2 2

Convolution

Sampling

Parzen Windows as a Convolution

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

Wish it was … Discrete convolution

52

Efficient Kernel Estimator

A. Samples of

estimated sourcesA

PDF estimation Fan; Marron 94, Silverman 82.

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

53

A. Samples of

estimated sources

B. Interpolation to

uniform grid

(histogram)

A

B

Efficient Kernel Estimator

PDF estimation Fan; Marron 94, Silverman 82.

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

53

C

• Samples of estimated sources

• Interpolation to uniform grid(histogram)

• Discrete convolution with Parzen window

A

B

PDF estimation Fan; Marron 94, Silverman 82.

Efficient Kernel Estimator

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

53

D

Efficient Entropy Estimator

C

• Interpolation to original values

A

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

54

Can it be Used for Optimization?

Wseparate

• Iterations exploiting derivatives of .

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

55

Can it be Used for Optimization?

Wseparate

• Binning fluctuations of .

• Fluctuations amplified by differentiation.

• Fluctuations slow convergence, false minima.

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

56

Function

Quantized function

Quantization and Optimization

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

57

Function

Quantized function

Function with a quantized derivative

Quantization and Optimization

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

57

Analytic Entropy Gradient

Accurate derivative

Efficient calculation

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

Complexity

Analytic Entropy Gradient

K- number of sources, N-data length.

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

Entropy Gradient by Convolutions

Convolution

Convolution

Convolution

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

58

• Calculation of using convolutions.

• Approximation of convolutions with

complexity.

• Distinct quantization of the derivative.

Not differentiation of a quantized

function.

Entropy Gradient by Convolutions

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

59

K=6 random sources, N= 3000 samples.

AlgorithmSignal to Interference ratio [dB]

Time

Basic Non-param ICA

18 4760 min.

Our algorithm22 31.2 min.

Jade7 40.2 sec.

Infomax8 41.6 sec.

Fast ICA5 31.9 sec.

Super ICA performance

Parametric

algorithms.

Non-parametric

algorithms.

Shwartz, Schechner & Zibulevsky, NlogN entropy optimization

60