Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali],...
-
Upload
dangnguyet -
Category
Documents
-
view
230 -
download
1
Transcript of Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali],...
![Page 1: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/1.jpg)
Model-based Compressive Compressive Sensing
Richard Baraniuk
Sensing
Richard BaraniukRice University
![Page 2: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/2.jpg)
Chinmay HegdeChinmay Hegde
Volkan Cevher
Marco Duarte
![Page 3: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/3.jpg)
Compressive Sensing
• Sensing via randomized dimensionality reduction
random sparsemeasurements signal
nonzeroentries
• Recovery: solve an ill-posed inverse problem
exploit the geometrical structure of sparse/compressible signals
![Page 4: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/4.jpg)
Restricted Isometry Property (RIP)• Preserve the structure of sparse/compressible signals
K-dimensional subspaces
![Page 5: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/5.jpg)
Restricted Isometry Property (RIP)• Preserve the structure of sparse/compressible signals
• RIP of order 2K implies: for all K-sparse x1 and x2
K-dimensional subspaces
![Page 6: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/6.jpg)
Beyond Sparse Models • Sparse/compressible signal model captures
simplistic primary structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
![Page 7: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/7.jpg)
Beyond Sparse Models • Sparse/compressible signal model captures
simplistic primary structure
• Modern compression/processing algorithms capture richer secondary coefficient structure
wavelets:natural images
Gabor atoms:chirps/tones
pixels:background subtracted
images
![Page 8: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/8.jpg)
Sparse Signals• Defn: K-sparse signals comprise a
particular set of K-dim canonical subspaces
![Page 9: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/9.jpg)
Model-Sparse Signals• Defn: A K-sparse signal model comprises a
particular (reduced) set of K-dim canonical subspaces [Blumensath and Davies]subspaces [Blumensath and Davies]
• Fewer subspaces <> relaxed RIP <> relaxed RIP <> stable recovery using
fewer measurements M
![Page 10: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/10.jpg)
Model-based CS
Running Example:
Tree-Sparse Signals
![Page 11: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/11.jpg)
Wavelet Sparse
• Typical of wavelet transformsof natural signals of natural signals and images (piecewise smooth)
![Page 12: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/12.jpg)
Tree-Sparse
• Model: K-sparse coefficients + significant coefficients
lie on a rooted subtreelie on a rooted subtree
• Typical of wavelet transformsof natural signals of natural signals and images (piecewise smooth)
![Page 13: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/13.jpg)
Tree-Sparse
• Model: K-sparse coefficients + significant coefficients + significant coefficients
lie on a rooted subtree
• Sparse approx: find best set of coefficients– sortingsorting– hard thresholding
T fi d b d b• Tree-sparse approx: find best rooted subtreeof coefficients
– CSSA [B][ ]
– dynamic programming [Donoho]
![Page 14: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/14.jpg)
Wavelet Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtreelie on a rooted subtree
• RIP: stable embedding
K-planes
![Page 15: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/15.jpg)
Tree-Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtreelie on a rooted subtree
• Tree-RIP: stable embedding [Blumensath and Davies][ ]
K-planes
![Page 16: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/16.jpg)
Tree-Sparse• Model: K-sparse coefficients
+ significant coefficients lie on a rooted subtreelie on a rooted subtree
• Tree-RIP: stable embedding [Blumensath and Davies]
R i j t t i t • Recovery: inject tree-sparse approx into IHT/CoSaMP
![Page 17: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/17.jpg)
Recall: Iterated Thresholding
[Nowak, Figueiredo; Kingsbury, Reeves; Daubechies, Defrise, De Mol; Blumensath, Davies; ]
update signal estimate
Davies; …]
update signal estimate
prune signal estimate(b t K t )(best K-term approx)
update residual
![Page 18: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/18.jpg)
Iterated Model Thresholding
update signal estimateupdate signal estimate
prune signal estimate(best K-term model approx)
update residual
![Page 19: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/19.jpg)
Tree-Sparse Signal Recovery
target signal CoSaMP, (MSE=1.12)
N=1024M=80
L1-minimization(MSE=0.751)
Tree-sparse CoSaMP (MSE=0.037)
![Page 20: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/20.jpg)
Compressible Signals
• Real-world signals are compressible, not sparse
• Compressible <> approximable by sparse
– compressible signals lie close to a union of subspacesl d i ffi i t– power-law decay in coefficients
– ie: approximation error decays rapidly as
• If has RIP, thenboth sparse andboth sparse andcompressible signalsare stably recoverablei LP d l via LP or greedy alg
sorted index
![Page 21: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/21.jpg)
Model-Compressible Signals
• Model-compressible <> approximable by model-sparsey p
– model-compressible signals lie close to a reduced union of subspaces
– ie: model-approx error decays rapidly asie: model approx error decays rapidly as
![Page 22: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/22.jpg)
Model-Compressible Signals
• Model-compressible <> approximable by model-sparsey p
– model-compressible signals lie close to a reduced union of subspaces
– ie: model-approx error decays rapidly asie: model approx error decays rapidly as
![Page 23: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/23.jpg)
Model-Compressible Signals
• Model-compressible <> approximable by model-sparsey p
– model-compressible signals lie close to a reduced union of subspaces
– ie: model-approx error decays rapidly asie: model approx error decays rapidly as
![Page 24: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/24.jpg)
Model-Compressible Signals
• Model-compressible <> approximable by model-sparsey p
– model-compressible signals lie close to a reduced union of subspaces
– ie: model-approx error decays rapidly asie: model approx error decays rapidly as
N lt hil d l RIP • New result: while model-RIP enables stable model-sparse recovery, model-RIP is y,not sufficient for stable model-compressible recovery!
![Page 25: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/25.jpg)
Stable Recovery• Result: Stable model-compressible signal recovery
requires that have both:– RIP + Restricted Amplification Property– RIP + Restricted Amplification Property
• RAmP: controls nonisometry of in the i i ’ id l bapproximation’s residual subspaces
optimal K-term optimal 2K-term residual subspacemodel recovery(error controlled
by RIP)
pmodel recovery(error controlled
by RIP)
p(error not controlled
by RIP)
![Page 26: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/26.jpg)
Tree-RIP, Tree-RAmPTheorem: An MxN iid subgaussian random matrix has the Tree(K)-RIP if
Theorem: An MxN iid subgaussian random matrix has the Tree(K)-RAmP if
![Page 27: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/27.jpg)
Performance
• Using model-based IHT, CoSaMP with RIP+RAmP
• Model-sparse signals– noise-free measurements: exact recovery
i t t bl – noisy measurements: stable recovery
• Model-compressible signalsp g– recovery as good as K-model-sparse approximation
CS recoveryerror
signal model K-termapprox error
noisesignal model K-termapprox error
![Page 28: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/28.jpg)
Simulation
• Number samples for correct recovery
• Piecewise cubic signals +
lwavelets
• Models/algorithms:• Models/algorithms:– sparse (CoSaMP)– tree-sparse
( C S )(tree-CoSaMP)
![Page 29: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/29.jpg)
Simulation
• Recovery performance (MSE) vs. number of measurements
• Piecewise cubic i l signals +
wavelets
• Models/algorithms:– sparse (CoSaMP)– tree-sparse
(tree-CoSaMP)
![Page 30: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/30.jpg)
Other Useful Models
• When the model-based framework makes sense:– model with
N t d i ti t (NAP)Nested approximation property (NAP)fast approximation algorithm
– sensing matrix with sensing matrix with model-RIPmodel-RAmP
• Ex: block sparsity / signal ensembles[Tropp, Gilbert, Strauss], [Stojnic, Parvaresh, Hassibi], [Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde]
• Ex: clustered signalsEx: clustered signals[C, Duarte, Hegde, B], [C, Indyk, Hegde, Duarte, B]
![Page 31: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/31.jpg)
Block-Sparse Signal
target CoSaMP (MSE = 0.723)
N = 4096K 6 active blocksK = 6 active blocksJ = block length = 64M = 2.5JK = 960 msnts
block-sparse model recovery (MSE=0.015)
![Page 32: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/32.jpg)
Block-Compressible Signal
target CoSaMP (MSE=0.711)
block-sparse recovery (MSE=0.195)
best 5-block approximation (MSE=0.116 )
![Page 33: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/33.jpg)
Clustered Sparsity
• (K,C) sparse signals (1-D)– K-sparse within at most C clusters
• For stable recovery (model-RIP + RAmP)( )
• Model approximation using dynamic programmingprogramming
[Cevher, Indyk, Hedge, B; Sampta 2009]
![Page 34: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/34.jpg)
Clustered Sparsity
• Model clustering of significant pixels in space domain using graphical model (Ising MRF)graphical model (Ising MRF)
• Ising model approximation i h tvia graph cuts
[Cevher, Duarte, Hedge, B; NIPS 2008]
target Ising-modelrecovery
CoSaMPrecovery
LP (FPC)recovery
![Page 35: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/35.jpg)
Summary
• Why CS works: stable embedding for signals with concise geometric structure
• Sparse signals >> model-sparse signalsC ibl i l >> d l ibl • Compressible signals >> model-compressible
signals
• Greedy model-based signal recovery algorithms
upshot: provably fewer measurementsmore stable recovery
new concept: RIP >> RAmPnew concept: RIP >> RAmP
![Page 36: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/36.jpg)
New Directions
• Diverse data types demand new models– deterministic models– deterministic models– probabilistic/Bayesian/graphical models [Carin et al]
– manifold models for signal ensembles [Wakin et al]
• New model-based recovery algorithms
• Can we weaken RAmP?
• Relate to results in coding/info theory literature [Nowak et al, …]
dsp.rice.edu/cs
![Page 37: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/37.jpg)
Open Positions
open postdoc positions in sparsity / compressive sensing at Rice University
dsp.rice.edui hb@ i [email protected]
![Page 38: Model-based Compressive Sensing - …people.ee.duke.edu/~lcarin/baraniuk.pdf[Eldar, Mishali], [Baron, Duarte et al], [B, C, Duarte, Hegde] • Ex: clustered signals ... sparsity](https://reader031.fdocuments.us/reader031/viewer/2022020214/5b0607667f8b9ad1768c3ce6/html5/thumbnails/38.jpg)
dsp.rice.edu/csdsp.rice.edu/cs