A Weighted Average of Sparse Representations is Better than the Sparsest One Alone

Michael Elad and Irad Yavneh

SIAM Conference on Imaging Science ’08

Presented by Dehong Liu

ECE, Duke University

July 24, 2009

Outline

• Motivation• A mixture of sparse representations• Experiments and results• Analysis • Conclusion

Motivation

• Noise removal problem y=x+v, in which y is a measurement signal, x is the clean signal, v

is assumed to be zero mean iid Gaussian.

• Sparse representationx=D, in which DRnm, n<m, is a sparse vector.

• Compressive sensing problem

• Orthogonal Matching Pursuit (OMP)

Sparsest representation

• Question:“Does this mean that other competitive and slightly inferior sparse r

epresentations are meaningless?”

A mixture of sparse representations

• How to generate a set of sparse representations?– Randomized OMP

• How to fuse these sparse representations? – A plain averaging

OMP algorithm

Randomized OMP

Experiments and results

Model:

• y=x+v=D+v• D: 100x200 random dictionary with entries dra

wn from N(0,1), and then with columns normalized;

: a random representations with k=10 non-zeros chosen at random and with values drawn from N(0,1);

• v: white Gaussian noise with entries drawn from N(0,1);

• Noise threshold in OMP algorithm T=100(??);• Run the OMP once, and the RandOMP 1000 t

imes.

Observations

0 10 20 30 400

50

100

150

Candinality

His

togra

m

Random-OMP cardinalitiesOMP cardinality

85 90 95 100 1050

50

100

150

200

250

300

350

Representation Error

His

togra

m

Random-OMP errorOMP error

0 0.1 0.2 0.3 0.40

50

100

150

200

250

300

Noise Attenuation

His

togra

m

Random-OMP denoisingOMP denoising

0 5 10 15 200.05

0.1

0.15

0.2

0.25

0.3

0.35

Cardinality

No

ise

Att

enu

ation

Random-OMP denoisingOMP denoising

Sparse vector reconstruction

0 50 100 150 200-3

-2

-1

0

1

2

3

index

valu

e

Averaged Rep.Original Rep.OMP Rep.

The average representation over 1000 RandOMP representations is not sparse at all.

Denoising factor based on 1000 experiments

0 0.1 0.2 0.3 0.4 0.50

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

OMP Denoising Factor

Ra

ndO

MP

Den

oisi

ng F

acto

r

OMP versus RandOMP resultsMean Point

Denoising factor=

Run RandOMP 100 times for each experiment.

Performance with different parameters

Analysis

The RandOMP is an approximation of the Minimum-Mean-Squared-Error (MMSE) estimate.

“

”

The above results correspond to a 20x30 dictionary. Parameters: True support=3, x=1, Averaged over 1000 experiments.

0.5 1 1.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5R

ela

tive

Me

an

-Sq

ua

red

-Err

or

20

1. Emp. Oracle2. Theor. Oracle3. Emp. MMSE4. Theor. MMSE5. Emp. MAP6. Theor. MAP7. OMP8. RandOMP

Comparison

Conclusion

• The paper shows that averaging several sparse representations for a signal lead to better denoising, as it approximates the MMSE estimator.