Masatake Fujita Shihan’s 8 Powerful Techniques Perfomed by Attila Pivony
POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data...
Transcript of POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data...
![Page 1: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/1.jpg)
16 March 2020
POLITECNICO DI TORINORepository ISTITUZIONALE
Compressed sensing: basics and beyond (tutorial) / Fosson, Sophie; Magli, Enrico. - (2015). ((Intervento presentato alconvegno Fifteenth International Conference on Computer Aided Systems Theory (EUROCAST 2015) tenutosi a LasPalmas de Gran Canaria, Spain nel Feb 8-13, 2015.
Original
Compressed sensing: basics and beyond (tutorial)
Publisher:
PublishedDOI:
Terms of use:openAccess
Publisher copyright
(Article begins on next page)
This article is made available under terms and conditions as specified in the corresponding bibliographic description inthe repository
Availability:This version is available at: 11583/2624990 since: 2015-12-05T15:07:46Z
![Page 2: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/2.jpg)
Compressed SensingBasics and Beyond
. EUROCAST 2015..
. Feb 12, 2015
S.M. Fosson E. Magli
www.crisp-erc.eu
![Page 3: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/3.jpg)
Outline
..1 Mathematical problem
..2 Applications
..3 Recovery
..4 Distributed compressed sensing
![Page 4: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/4.jpg)
Outline
..1 Mathematical problem
..2 Applications
..3 Recovery
..4 Distributed compressed sensing
![Page 5: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/5.jpg)
Mathematical problem
Compressed sensing (compressed sampling, compressive sensing... CS)deals with.Underdetermined linear systems .....
.
Ax = yx ∈ Rn (unknown), y ∈ Rm (measurements), A ∈ Rm×n, m < n
Within the infinite set of solutions, CS looks for the sparsest one.... with sparsity assumptions...x is k-sparse, i.e., it has k non-zero components, where k ≪ n
S.M. Fosson COMPRESSED SENSING 4/33
![Page 6: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/6.jpg)
Mathematical problem
S.M. Fosson COMPRESSED SENSING 5/33
![Page 7: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/7.jpg)
Questions
Ax = y, x ∈ Rn(sparse), y ∈ Rm,m < n
..1 Is the problem well-posed (= is the solution unique)?
..2 Are there feasible algorithms to find the solution?
..3 Which applications motivate this study?
Answers..1 Yes, under some conditions..2 A number of recovery algorithms have been proposed..3 ▶ Sparsity is ubiquitous: many signals are sparse in some basis
(y = Aϕx where ϕ is the sparsifying basis, e.g., DCT, wavelets,Fourier... )
▶ Applications where data acquisition is difficult/expensive, andone aims to move the computational load to the receiver
S.M. Fosson COMPRESSED SENSING 6/33
![Page 8: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/8.jpg)
Outline
..1 Mathematical problem
..2 Applications
..3 Recovery
..4 Distributed compressed sensing
![Page 9: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/9.jpg)
Medical Imaging
Magnetic Resonance Imaging (MRI): acquisition is slow[Lustig (2012)]
→ sense the compressed information directly
S.M. Fosson COMPRESSED SENSING 8/33
![Page 10: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/10.jpg)
Compression and sampling
Ax = y, x ∈ Rn(sparse), y ∈ Rm,m < n
• Sampling: Nyquist-Shannon sampling theorem states given asignal bandlimited in (B,B), to represent it over a timeinterval T, we need at least 2BT samples
• CS indicates a way to merge compression and sampling, andsample at a sub-Nyquist rate [Tropp et al. (2009)]
S.M. Fosson COMPRESSED SENSING 9/33
![Page 11: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/11.jpg)
Compression and sampling
sensing
ADCcompression
?
S.M. Fosson COMPRESSED SENSING 10/33
![Page 12: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/12.jpg)
Wideband spectrum sensing
Modulated wideband converter (MWC) [Mishali and Eldar (2010)]
• Sub-Nyquist sampling for signals sparse in the frequency domain• Realized in hardware (with commercial devices)
S.M. Fosson COMPRESSED SENSING 11/33
![Page 13: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/13.jpg)
Single-pixel camera
Boufonos et al., ICASSP 2008Key ingredient: a microarray consisting of a large number of smallmirrors that can be turned on or off individuallyLight from the image is reflected on this microarray and a lenscombines all the reflected beams in one sensor, the single pixel ofthe camera
S.M. Fosson COMPRESSED SENSING 12/33
![Page 14: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/14.jpg)
Outline
..1 Mathematical problem
..2 Applications
..3 Recovery
..4 Distributed compressed sensing
![Page 15: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/15.jpg)
Mathematical formulation
.ℓ0-norm...∥x∥0 := number of nonzeros entries of x ∈ Rn
Natural formulation of the CS problem:.
.
P0 : minx∈Rn
∥x∥0 subject to Ax = y
• Is the solution unique?• P0 is NP-hard!
S.M. Fosson COMPRESSED SENSING 14/33
![Page 16: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/16.jpg)
Uniqueness of the solution
.Spark..
.spark(A) := minimum number of columns of A that are linearlydependent.Theorem [D. Donoho, M. Elad (2003)]..
.For any vector y ∈ Rm, there exists at most one k-sparse signalx ∈ Rn such that y = Ax if and only if spark(A) > 2k.
S.M. Fosson COMPRESSED SENSING 15/33
![Page 17: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/17.jpg)
Uniqueness of the solution
.Coherence..
.µ(A) := maxi ̸=j|AT
i Aj|∥Ai∥2∥Aj∥2
(Ai = ith column of A)
.Theorem [D. Donoho, M. Elad (2003)]..
.
Ifk <
12
(1 +
1µ(A)
)y ∈ Rm, there exists at most one k-sparse signal x ∈ Rn such thaty = Ax.
S.M. Fosson COMPRESSED SENSING 16/33
![Page 18: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/18.jpg)
Basis Pursuit (BP)
Possible solution: convex relation.Basis Pursuit..
.
P1 : minx∈Rn
∥x∥1 subject to Ax = y
• P1 is convex; can be solved by linear programming• Are P0 and P1 equivalent?
S.M. Fosson COMPRESSED SENSING 17/33
![Page 19: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/19.jpg)
Coherence
.Coherence..
.µ(A) := maxi ̸=j|AT
i Aj|∥Ai∥2∥Aj∥2
(Ai = ith column of A)
.Theorem [Elad and Bruckstein (2002)]..
.
If for the sparset solution x⋆ we have
∥x⋆∥0 <
√2 − 1
2µ(A)
then the solution of P1 is equal to the solution of P0.
S.M. Fosson COMPRESSED SENSING 18/33
![Page 20: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/20.jpg)
Restricted Isometry Property (RIP)
.RIP..
.
Matrix A satisties the RIP of order k if there exists δk ∈ (0, 1) suchthat the following relation holds for any k-sparse x:
(1 − δk) ∥x∥22 ≤ ∥Ax∥2
2 ≤ (1 + δk) ∥x∥22
.Theorem [Candès (2008)]..
.If δk <
√2 − 1, then for all k-sparse x ∈ Rn such that Ax = y, the
solution of P1 is equal to the solution of P0.
S.M. Fosson COMPRESSED SENSING 19/33
![Page 21: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/21.jpg)
Which matrices?
• Coherence, spark, RIP: not easy to compute• Random matrices A with i.i.d. entries drawn from continuous
distributions have spark(A) = m + 1 with probability one.• Gaussian, Bernoulli matrices: given δ ∈ (0, 1) there exist c1, c2
depending on δ such that G. and B. matrices satisfy the RIPwith constant δ and any m ≥ c1k log(n/k) with probability≥ 1 − 2e−c2m [Baraniuk (2008)]
• Structured matrices: circulant matrices, partial Fouriermatrices
S.M. Fosson COMPRESSED SENSING 20/33
![Page 22: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/22.jpg)
Orthogonal Matching Pursuit (OMP)
• “When we talk about BP, we often say that the linearprogram can be solved in polynomial time with standardscientific software, and we cite books on convex programming[...]. This line of talk is misleading because it may take a longtime to solve the linear program, even for signals of moderatelength” [Tropp and Gilbert (2007)]
• Possible solution: greedy algorithm, fast, easy to implement→ OMP
S.M. Fosson COMPRESSED SENSING 21/33
![Page 23: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/23.jpg)
Orthogonal Matching Pursuit (OMP)
..1 Initialize r0 = y, Λ0 = ∅
..2 For t = 1, . . . ,Tmax
..3 λt = argmaxj=1,...,n
|ATj rt−1|
..4 Λt = Λt−1 ∪ {λt}
..5 x̂t = argminx∈Rn
∥y − AΛtx∥2
..6 rt = y − AΛt x̂t
• Tmax ≈ k• OMP requires the knowledge of k!
S.M. Fosson COMPRESSED SENSING 22/33
![Page 24: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/24.jpg)
Variants of BP
.Basis Pursuit Denoise (BPDN)..
.
P1 : minx∈Rn
∥x∥1 subject to ∥Ax = y∥2 ≤ ε
Unconstrained version of BPDN.Lasso...minx∈Rn
(∥Ax − y∥2
2 + λ ∥x∥1)
For some λ > 0, Lasso and BPDN have the same solution (thechoice of λ is tricky!)
S.M. Fosson COMPRESSED SENSING 23/33
![Page 25: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/25.jpg)
Iterative soft thresholding (IST)
..1 x̂0 = 0
..2 For t = 1, . . . ,Tmax
..3 x̂t = Sλ(x̂t−1 + τAT(y − A ∗ x̂t−1))
where the operator Sλ is defined entry by entry asSλ(x) = sgn(|x| − λ) if |x| > λ, 0 otherwise
• IST achieves a minimum of the Lasso [Fornasier (2010)], andin many common situations such minimum is unique[Tibshirani (2012)]
• Faster method to get a minimum of the Lasso: alternatingdirection method of multipliers (ADMM)
S.M. Fosson COMPRESSED SENSING 24/33
![Page 26: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/26.jpg)
Iterative hard thresholding
..1 x̂0 = 0
..2 For t = 1, . . . ,Tmax
..3 x̂t = Hk(x̂t−1 + AT(y − Ax̂t−1))
where the operator Hk(x) is the non-linear operator that sets allbut the largest (in magnitude) k elements of x to zero [Blumensath(2008)]
S.M. Fosson COMPRESSED SENSING 25/33
![Page 27: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/27.jpg)
Outline
..1 Mathematical problem
..2 Applications
..3 Recovery
..4 Distributed compressed sensing
![Page 28: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/28.jpg)
Distributed compressed sensing (DCS)
• Data acquisition is perfomed by a network of sensors
yv = Avxv v ∈ V = { sensors }
• First works: recovery is performed by a fusion center thatgathers information from the network (sensing matrices,measurements)
• New: in-network recovery, exploiting local communication andconsensus procedures
• We need iterative algorithms
S.M. Fosson COMPRESSED SENSING 27/33
![Page 29: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/29.jpg)
Distributed Compressed Sensing (DCS)
S.M. Fosson COMPRESSED SENSING 28/33
![Page 30: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/30.jpg)
Distributed Compressed Sensing (DCS)
S.M. Fosson COMPRESSED SENSING 29/33
![Page 31: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/31.jpg)
Distributed Compressed Sensing (DCS)
S.M. Fosson COMPRESSED SENSING 30/33
![Page 32: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/32.jpg)
Distributed Compressed Sensing (DCS)
S.M. Fosson COMPRESSED SENSING 31/33
![Page 33: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/33.jpg)
DCS: in-network recovery
..1 C. Ravazzi, S.M. Fosson, E. Magli, E., Energy-saving GossipAlgorithm for Compressed Sensing in Multi-agent Systems,ICASSP, 2014
..2 S.M. Fosson, J. Matamoros, C. Antón-Haro , E. Magli,Distributed Support Detection of Jointly Sparse Signals,ICASSP, 2014
..3 J. Matamoros, S.M. Fosson, E. Magli, C. Antón-Haro,Distributed ADMM for in-network reconstruction of sparsesignals with innovations, IEEE GlobalSIP, 2014
..4 C. Ravazzi, S.M. Fosson, E. Magli, Distributed iterativethresholding for ℓ0/ℓ1-regularized linear inverse problems,IEEE Trans. Inf. Theory, 2015.
S.M. Fosson COMPRESSED SENSING 32/33
![Page 34: POLITECNICO DI TORINO Repository ISTITUZIONALE...Distributedcompressedsensing(DCS) • Data acquisition is perfomed by a network of sensors yv = Avxv v 2 V = f sensors g • First](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6f06fbe80adb2ff32c4e98/html5/thumbnails/34.jpg)
CS references
..1 http://dsp.rice.edu/cs
..2 E. Candès, J. Romberg, and T. Tao, Robust uncertaintyprinciples: Exact signal reconstruction from highly incompletefrequency information, IEEE Trans. Inf. Theory, Feb. 2006
..3 D. Donoho, Compressed sensing, IEEE Trans. Inf. Theory,Apr. 2006
..4 A mathematical Introduction to Compressive Sensing, editedby S. Foucart and H. Rauhut, 2013
..5 Compressed Sensing: Theory and Applications, edited by Y.C. Eldar and G. Kutyniok, 2012
..6 Theoretical Foundations and Numerical Methods for SparseRecovery, edited by M. Fornasier, 2010
S.M. Fosson COMPRESSED SENSING 33/33