Sparse Reconstruction via the Reed-Muller Sieve
Sina JafarpourPrinceton UniversityGame Theory Meets Compressed Sensing
Based on joint work with:
VolkanCevherRobertCalderbankRobSchapire
Compressed SensingMain tasks:Design a sensing matrixDesign a reconstruction algorithmObjectives:Reconstruct a k-sparse signal from efficientlyThe algorithm should recover without knowing its supportSuccessful reconstruction from as few number of measurements as possible!Robustness against noise Model selection and sparse approximation
Model Selection and Sparse ApproximationMeasurement vector: : data-domain noise : measurement noiseModel Selection Goal:Given successfully recover the support of Sparse Approximation Goal: Given find a vector with
ApplicationsData streamingPacket routingSparsity in canonical basis
Biomedical imagingMRISparsity in Fourier basis
Digital communicationMultiuser DetectionSparsity in canonical basis
DestinationSource
Restricted Isometry PropertyDefinition: A matrix satisfies if for every -sparse vector :
Tractable Compressed Sensing: [CRT06/Don06] If is , then for the solution of
satisfies the guarantee
Basis Pursuit optimizationProblems with RIP Matricesiid Gaussian/Bernoulli matrices:No efficient algorithm for verifying RIP memory requirementInefficient computation (matrix-vector multiplication)Partial Fourier/Hadamard ensembles:No efficient algorithm for verifying RIPSub-optimal number of measurements:Algebraic structures:Sub-optimal number of measurements:
DeterministicCompressed SensingviaExpander GraphsExpander GraphsSensing matrix: Normalized adjacency matrix of an expander graph Existence: A random bipartite graph with is expander.
Explicit construction of expander graphs also exist [GUV08]. Expander-Based Compressed Sensing:Message Passing AlgorithmsTheorem [JXHC09] : If is a normalized expander then givena message passing algorithm after iterations recovers .Decoding in error-free case: Based on expander codes [Sipser and Spielman96][BIR09] (SSMP): A similar message passing algorithm with running time and guarantee Decoding when error exists: Pros: Decoding time is almost linear in Message-passing methods are easy to implement Cons: Poor practical performance Basis Pursuit Algorithm[BGIKS08]: Let be the normalized adjacency of a expander graph, then the solution of the basis pursuit optimization
Satisfies the guarantee
Pros: Better practical performance Cons: Decoding time is cubic in Sub-gradient methods are hard to implementGame Theory Meets Compressed SensingGoal: an efficient approximation algorithm for BPTo solve BP, it is sufficient to be able to efficiently solve
How? By doing binary search overPlan: approximately solve efficiently By reformulating it as a zero-sum gameApproximately solving the game-value
Recall BP Optimization:Game-Theoretic Reformulation of the Problem1: Consider the following 1-dimensional example2: Fix (say ) and observe that MaximumIntervalDiscrete set(2 elements) Bregman Divergence Examples: Euclidean distance: with Relative-entropy: with
GAME Algorithm:
Alice: Updates Bob: Updates Finds a withFinds an s.t.Greedy Alice:Conservative Alice:Gradient Projection Alice:Greedy Bob: Finds an s.t.Lemma [JCS11] is 1-sparse and can be computed in
Game-value Approximation GuaranteeGAME can be viewed as a repurposing of the Multiplicative Update Algorithm [Freund&Schapire99]
After iterations finds a -sparse vector with
Tradeoff between sparsity and approximation errorAs :
In expander-based CS we are only interested in the approximation error (Basis Pursuit Theorem) Empirical PerformanceMPBPEmpirical Convergence Rate
33Slope: -1Comparison with Nesterov OptimizationNesterov: A general iterative approximation algorithm for solving non-smooth optimization problemsNesterov requires iterations (similar to GAME)Each iteration requires solving 3 smooth optimization problemMuch more complicated than one adjoint operation (GAME) Nesterovs approximation guarantee:
GAMEs theoretical guarantee
Experimental ComparisonGAMERandom vs. Expander-Based Compressed SensingMatrixNumber ofMeasurementsEncodingTimeRecoveryTimeRecoveryGuaranteeExplicit ConstructionGaussianNoFourierNoExpander(MesPas)YesExpander(GAME)YesFairBestGoodOurContribution
Take-Home Message:Random vs. DeterministicGerhard Richter
The value of a random matrix:US $ 3,703,500
The value of a deterministic matrix:US $ 27,191,525
Piet Mondrian
Thank You!
M k " N
:M N
f =
f and
fand
p = O (edq + emq)
(!p/!q guarantee)
f = (+ ed) + em
ed RN
em RM
232
232
(1 )22 22 (1 + )22
2/1
(2k, 0.4) RIP
minimize 1
s.t. f2 em2
2 = Oed1
k+ em1
(k, ) RIP
k
O(MN)
M = O(k log5N)
M = O(k2)
N
N (S)
(1 !)d|S|
|S| k |N (S)| (1 )d|S|
M = Ok logN/k
2
M
d
S
|S| k
O(N logN/k)
1/1
1 = O(ed1 + eM1)
N
2k
(2k, 1/8)
f =
1/1
1 = O (ed1 + em1)
N
minimize1
s.t. f1 em1
(2k, 1/8)
s.t. f1 em1
minimize f1
s.t. 1
minimize1
= maxP{1,1}
P
x = 0
P = 1
P = 1
min|x|1
maxP[1,1]
P (2x 1) = min|x|1
maxP{1,1}
P (2x 1)
min|x|1
|2x 1| = min|x|1
max {2x 1, 1 2x}
maxP{1,1}M
P( f)
= min1
min1
f1 = min1
maxP[1,1]M
= min|x|1
maxP{1,1}
P (2x 1)
x
maxP[1,1]
P
R(P )
R(Q) + (P Q)!R(Q)
B(P,Q)
R(P ) = P2
B(P,Q) = P Q2
P
R(P ) =
i
P (i) lnP (i)
B(P,Q) =
i
Q(i)
i
P (i) +
i
P (i) lnQ(i)P (i)
B(P,Q) .= R(P )R(Q) (P Q)!R(Q)
Q
R
R(Q)
R(P0) = 0
P0
P0
0
0
Pt.= arg max
P[1,1]P(t1 f)
Pt.= arg min
P[1,1]MB(P, Pt1) P(t1 f)
R(Qt) = R(Pt1) + (t1 f)
Pt.= arg min
P[1,1]MB(P,Qt)
Pt
t
t.= arg min
1Pt (
f)
0.= arg min
1P0 (
f)
t
P
O(N logN/k)
t
Output :
.=1T
Ti=1
i
min1
maxP[1,1]M
P( f)
Objective :
T
f1 = min1 f1 +O
M
T
T
f1 min1 f1
T
104 103 102 101103
102
101
100
101
Measurement noi se std (m)
Rec
onstru
ctionerro
r( 1
/
1)
Reconstruction Error: N=5000, M=500, k=35
l1magicSSMPeGAME
proven theoretical convergence rate is O(T0.5)
empirical convergence rate is O(T1)
Measurement error: f1
Data error: 1
T = O(M)
f1 eM1 +O(M
T
)
f1 eM1 +O(
M
T
)
101 102 103102
101
100
101
Conv
erge
nce
rate
Number of iterations (T)
Convergence of BPP Algorithm: N=5000, M=500, k=35
BPP: MSMT ErrorBPP:Data ErrorNest: Data ErrorNest: MSMT Error
k log5N
N logN
N logN log R
!1/!1
BP
k log (N/k)
BP
k log(N/k)
k log(N/k)
N log(N/k)
N log(N/k)
N log(N/k)
kN log2(N/k)R2
R = SNR .=2e2
kN log (N/k)
kN log (N/k) log R
!2/!1
Top Related