Aalborg Universitet

When 'exact recovery' is exact recovery in compressed sensing simulation

Sturm, Bob L.

Published in:Proceedings of the European Signal Processing Conference

Publication date:2012

Document VersionEarly version, also known as pre-print

Link to publication from Aalborg University

Citation for published version (APA):Sturm, B. L. (2012). When 'exact recovery' is exact recovery in compressed sensing simulation. Proceedings ofthe European Signal Processing Conference, 2012, 979-983. http://www.scopus.com/inward/record.url?eid=2-s2.0-84869754566&partnerID=40&md5=7acd2d5ff6239a41f89f0083ebcdc6a4

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

? Users may download and print one copy of any publication from the public portal for the purpose of private study or research. ? You may not further distribute the material or use it for any profit-making activity or commercial gain ? You may freely distribute the URL identifying the publication in the public portal ?

Take down policyIf you believe that this document breaches copyright please contact us at vbn@aub.aau.dk providing details, and we will remove access tothe work immediately and investigate your claim.

Downloaded from vbn.aau.dk on: August 19, 2021

When “Exact Recovery” is Exact Recoveryin Compressed Sensing Simulation

Bob L. Sturm1

Department of Architecture, Design and Media TechnologyAalborg University Copenhagen

A.C. Meyers Vænge 15, DK-2450, Denmark

August 27, 2012

1B. L. Sturm is supported in part by Independent Postdoc Grant 11-105218from Det Frie Forskningsrad.

1 / 25

Introduction

Setup

Measurements u come from sensing x by sensing matrix Φ: u = Φx + n.We use a recovery algorithm to build x given u and Φ, e.g., OMP, BP.

Exact Recovery

In theory, we have no trouble asking x?= x.

In practice, we must use a different criterion.

At least two different criteria have been used in the simulationof compressed sensing recovery algorithms.

2 / 25

Introduction

One exact recovery criterion in CS simulation: Support

Let Ω index the columns of Φ, and define the support of x as

S(x) := i ∈ Ω : xi 6= 0.

x is exactly recovered with respect to support if

S(x) = S(x). (SC)

This has been used in simulations of CS recovery in, e.g.,

E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signalreconstruction from highly incomplete frequency information,” IEEE Trans. Info.Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.

J. Tropp and A. C. Gilbert, “Signal recovery from random measurements viaorthogonal matching pursuit,” IEEE Trans. Info. Theory, vol. 53, no. 12, pp.4655-4666, Dec. 2007.

A. K. Fletcher, S. Rangan, and V. K. Goyal, “Necessary and sufficient conditionsfor sparsity pattern recovery,” IEEE Trans. Info. Theory, vol. 55, no. 12, pp.5758-5772, Dec. 2009.

3 / 25

Introduction

One exact recovery criterion in CS simulation: Support

E. Candes, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signalreconstruction from highly incomplete frequency information,” IEEE Trans. Info.Theory, vol. 52, no. 2, pp. 489-509, Feb. 2006.

For N = 512. (a) Empirical prob. exact recovery as fun. of M (ord.), K/M (abs.).

White is 1.0. (b) Empirical prob. of exact recovery for M = 64 as function of K/M .

4 / 25

Introduction

One exact recovery criterion in CS simulation: Support

J. Tropp and A. C. Gilbert, “Signal recovery from random measurements via orthogonalmatching pursuit,” IEEE Trans. Info. Theory, vol. 53, no. 12, pp. 4655-4666, Dec 2007

5 / 25

Introduction

One exact recovery criterion in CS simulation: Support

A. K. Fletcher, S. Rangan, and V. K. Goyal, “Necessary and sufficient conditions forsparsity pattern recovery,” IEEE Trans. Info. Theory, vol. 55, no. 12, pp. 5758-5772,Dec. 2009.

6 / 25

Introduction

Another exact recovery criterion: Normalized `2-norm Error

Define a 0 ≤ ε2 < 1.

x is exactly recovered with respect to normalized squared error if

‖x− x‖22‖x‖22

≤ ε2 (ε2C)

This has been used in simulations of CS recovery in, e.g.,

A. Maleki and D. L. Donoho, “Optimally tuned iterative reconstruction algorithmsfor compressed sensing,” IEEE J. Sel. Topics Signal Process., vol. 4, no. 2, pp.330-341, Apr. 2010.

J. Vila and P. Schniter, “Expectation-maximization Bernoulli-Gaussianapproximate message passing,” in Proc. Asilomar Conf. Signals, Syst., Comput.,Pacific Grove, CA, Nov. 2011.

Y. Wang and W. Yin, “Sparse signal reconstruction via iterative supportdetection,” SIAM J. Imaging Sciences, vol. 3, no. 3, pp. 462-491, 2010.

7 / 25

Introduction

Another exact recovery criterion: Normalized `2-norm Error

A. Maleki and D. L. Donoho, “Optimally tuned iterative reconstructionalgorithms for compressed sensing,” IEEE J. Sel. Topics Signal Process.,vol. 4, no. 2, pp. 330-341, Apr. 2010. (ε = 0.01.)

8 / 25

Introduction

Another exact recovery criterion: Normalized `2-norm Error

J. Vila and P. Schniter, “Expectation-maximization Bernoulli-Gaussianapproximate message passing,” in Proc. Asilomar Conf. Signals, Syst.,Comput., Pacific Grove, CA, Nov. 2011. (ε = 0.01.)

9 / 25

Introduction

Another exact recovery criterion: Normalized `2-norm Error

Y. Wang and W. Yin, “Sparse signal reconstruction via iterative supportdetection,” SIAM J. Imaging Sci., vol. 3, no. 3, 2010. (ε =?.)

10 / 25

Introduction

Two Criteria for Exact Recovery

1 x is exactly recovered with respect to support if

S(x) = S(x) (SC)

2 x is exactly recovered with respect to normalized squared error if

‖x− x‖22‖x‖22

≤ ε2 (ε2C)

One does not necessarily imply the other. There are instances, however,when one must be true if the other is true.

My Aims

With regards to running and comparing simulations of CS recovery:

Given a pair (x,x), when does “exact recovery” occur with respect toonly one or both criteria?

What is the role of ε2, and how should we define it?

11 / 25

Introduction Presentation Outline

Presentation Outline

1 Noiseless Case

x ∼ Bernoulli-Rademacher sparse signalsx ∼ Bernoulli-Gaussian sparse signalsSimulations

2 Noisy Case

x ∼ Bernoulli-Rademacher sparse signalsSimulations

3 Conclusions

12 / 25

Noiseless Case Setup

Noiseless Case

Measurements u come from sensing x by the sensing matrix Φ, ‖n‖ = 0:

u = Φx + n.

Given x, the weights minimizing the measurement modeling error are

yls := arg miny′‖u−ΦS(x)y

′‖22 = Φ†S(x)u.

With x composed of yls, if (SC) then for any ε2 ∈ [0, 1] (ε2C).

If, however, (ε2C) for ε2 = 0 then necessarily (SC).

Now we analyze the behavior of these criteria for signals distributedBernoulli-Rademacher, Gaussian, and empirically in other ways.

13 / 25

Noiseless Case Setup

Noiseless Case

Consider the best case scenario for sparsity s

S(x) = 1, 2, . . . , s;x lacks the first 0 < k < s elements, i.e., for n ∈ 1, . . . , k(xn = 0);

x has all the others, i.e., n ∈ Ω\1, . . . , k(xn = xn).

This means that

S(x) ⊂ S(x), i.e., x has no false detections;

the missed detections do not influence our estimation of the values ofthe recovered support.

In this case, (ε2C) and not (SC) becomes for 1 ≤ k ≤ s

1

‖x‖22

k∑n=1

x2n ≤ ε2. (1)

14 / 25

Noiseless Case Bernoulli-Rademacher Signals

Bernoulli-Rademacher Signals

If x ∼ Bernoulli-Rademacher, its non-zero elements are iid equiprobable in−1, 1. In this case, ‖x‖22 = s, so

P(ε2C) ∧ ¬(SC) =

1, k/s ≤ ε2

0, else.(2)

For Bernoulli-Rademacher sparse signals in the best case scenario:

The parameter ε2 limits the number of missed detections k for a sparsity s.

As long as s < ε−2 for x ∼ Bernoulli-Rademacher, (ε2C) → (SC).

In Maleki et al. 2010, where s < 800 and ε2 = 10−4, (ε2C) → (SC).However, if for this ε2 the sparsity s > 10000, then the two conditionsare no longer equivalent.

15 / 25

Noiseless Case Bernoulli-Gaussian Signals

Bernoulli-Gaussian Signals

Let the s non-zero elements of x ∼ N (0, σ2y) with variance σ2y > 0.Define the independent chi-squared rvs

Yk :=

k∑n=1

[xn/σy]2 ∼ χ2(k), Zs−k :=

s∑n=k+1

[xn/σy]2 ∼ χ2(s− k)

Since Yk and Zs−k are independent, Fk,s−k := [Yk/k]/[Zs−k/(s− k)]∼ F(k, s− k). Thus, in the best case scenario

P(ε2C) ∧ ¬(SC) = P

Fk,s−k <

ε2

1− ε21− k/sk/s

. (3)

If k/s > ε2, then, for s ≥ 2k, P Fk,s−k < 1 + δ > 0.5 for δ > 0.

For Bernoulli-Gaussian signals in the best case scenario:

The parameter ε2 limits the number of missed detections k before((ε2C) ∧¬ (SC)) is false in a majority sense.

16 / 25

Noiseless Case Experiments

Experiments for several ε2 (labeled) & sparsities (legend)

(a) Zero-mean Gaussian (theoretical)

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio Missing Elements (100 k/s)

Pro

b.

Exa

ct

Re

co

ve

ry w

rt (

e2C

)

0.250.11e−2

100

400

2000

(b) Laplacian (empirical)

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio Missing Elements (100 k/s)

Pro

b.

Exa

ct

Re

co

ve

ry w

rt (

e2C

)

0.250.11e−2

100

400

2000

(c) Uniform (empirical)

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio Missing Elements (100 k/s)

Pro

b.

Exa

ct

Re

co

ve

ry w

rt (

e2C

)

0.250.11e−2

100

400

2000

(d) Bimodal Gaussian (empirical)

0 5 10 15 20 25 30 35 400

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Ratio Missing Elements (100 k/s)

Pro

b.

Exa

ct

Re

co

ve

ry w

rt (

e2C

)

0.250.11e−2

100

400

2000

17 / 25

Noisy Case Setup

Noisy Case (assuming (SC))

Measurements u come from sensing x by the sensing matrix Φ, ‖n‖ > 0:

u = Φx + n.

Assume (SC), and x is built from Φ†S(x)u. The weights in real solution are

y := arg miny′‖u− n−ΦS(x)y

′‖22 = Φ†S(x)(u− n).

Then, (ε2C) becomes

‖y −Φ†S(x)u‖22

‖y‖22=‖Φ†S(x)(u− n)−Φ†S(x)u‖

22

‖y‖22=‖Φ†S(x)n‖

22

‖y‖22≤ ε2. (4)

Hence, for any ε2 ∈ (0, 1] we can find an n such that ((SC) ∧¬ (ε2C)).This is different from noiseless case.

18 / 25

Noisy Case Bernoulli-Rademacher Signals Given (SC)

Bernoulli-Rademacher Signals Given (SC)

Define v := Φ†S(x)n, and assume its |S(x)| elements are iid N (0, σ2v) andindependent of y. Define the chi-squared-distributed rv

Vs :=

s∑n=1

[vn/σv]2 ∼ χ2(s). (5)

If s elements of x ∼ Rademacher, the probability of (ε2C) given (SC)

P(ε2C)|(SC) = P

Vs <

ε2s

σ2v

. (6)

Note P Vs < s+ δ > 0.5 for δ > 0.

For Bernoulli-Rademacher signals, in the best case scenario:

Given (SC), if ε2 ≥ σ2v then (ε2C) in a majority sense.

19 / 25

Noisy Case Bernoulli-Gaussian Signals Given (SC)

Bernoulli-Gaussian Signals Given (SC)

Assume s non-zero elements of x ∼ N (0, σ2y), independent of v. Define

Xs :=

s∑n=1

[xn/σy]2 ∼ χ2(s). (7)

The ratio Vs/Xs is an F-distributed rv Ws,s := Vs/Xs ∼ F(s, s).Thus, the probability of (ε2C) given (SC) is

P(ε2C)|(SC) = P

Ws,s <

σ2yσ2vε2

. (8)

Note P Ws,s < 1 + δ > 0.5 for δ > 0.

For Bernoulli-Gaussian signals, in the best case scenario:

Given (SC), if ε2 ≥ σ2v/σ2y then (ε2C) in a majority sense.

20 / 25

Noisy Case Experiments

Experiments for several SNR (legend) given (SC)

(a) Rademacher (theoretical)

0 0.05 0.1 0.150

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

b.

Exa

ct

Re

co

ve

ry b

y (

e2C

) &

(S

C)

100ǫ2

60 dB

33 dB

30 dB

(b) Zero-mean Gaussian (theoretical)

0 0.05 0.1 0.150

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

b.

Exa

ct

Re

co

ve

ry b

y (

e2C

) &

(S

C)

100ǫ2

60 dB

33 dB

30 dB

(c) Zero-mean Laplacian (empirical)

0 0.05 0.1 0.150

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

b.

Exa

ct

Re

co

ve

ry b

y (

e2C

) &

(S

C)

100ǫ2

60 dB

33 dB

30 dB

(d) Zero-mean Uniform (empirical)

0 0.05 0.1 0.150

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Pro

b.

Exa

ct

Re

co

ve

ry b

y (

e2C

) &

(S

C)

100ǫ2

60 dB

33 dB

30 dB

21 / 25

Noisy Case Setup

Noisy Case (assuming not (SC))

Consider (ε2C) is true but not (SC), and best case scenario for sparsity s:

S(x) = 1, 2, . . . , s;x lacks the first 0 < k < s elements, i.e., for n ∈ 1, . . . , k(xn = 0);

x has the others perturbed by v: n ∈ Ω\1, . . . , k(xn = xn + vn).

This means that:

S(x) ⊂ S(x), i.e., x has no false detections;

missed detections do not influence estimations of support recovered;

values of true detections perturbed only by the noise.

Assume x and v are independent, (ε2C) given not (SC) becomes

1

‖x‖22

[k∑

n=1

x2n +

s−k∑n=1

v2n

]≤ ε2. (9)

22 / 25

Noisy Case Bernoulli-Rademacher Signals

Bernoulli-Rademacher Signals (assuming not (SC))

Define the rv

Gs−k :=

s−k∑n=1

[vn/σv]2 ∼ χ2(s− k). (10)

When the non-zero elements of x are distributed Rademacher, andvn ∼ N (0, σ2v), (ε2C) given not (SC) becomes

P(ε2C) ∧ ¬(SC) = P

Gs−k <

ε2s− kσ2v

. (11)

Note PGs−k < s− k + δ > 0.5 for δ > 0.

For Bernoulli-Rademacher signals in the best case scenario:

Ifε2s− kσ2v

< s− k, then (ε2C) is false in a majority sense.

23 / 25

Noisy Case Experiments

Experiments for several ε2 (labeled) & SNR (legend)

(a) Rademacher (theoretical) (b) Zero-mean Gaussian (empirical)

24 / 25

Summary and Conclusion

Summary and Conclusion

In theory, we can test for exact recovery with x?= x.

In practice (finite precision), we must use a different criterion.

In the simulation of compressed sensing recovery algorithms, twodifferent exact recovery criteria have been used:

1 x is exactly recovered with respect to support if

S(x) = S(x) (SC)

2 x is exactly recovered with respect to normalized squared error if

‖x− x‖22‖x‖22

≤ ε2. (ε2C)

We have shown that

each does not necessarily imply the other

ε2 limits the acceptable number of missed detections.

See the paper for more useful tips!

25 / 25