xD
Sparse & Redundant Signal Representation, and its Role
in Image Processing
Michael Elad The CS Department The Technion – Israel Institute of
technology Haifa 32000, Israel
Wave 2006 – Wavelet and Applications
Ecole Polytechnique Federale de Lausanne (EPFL)
July 10th–14th, 2006
Sparse and Redundant Signal Representation, and Its Role in Image Processing
2xD
Today’s Talk is About
Sparsity and
RedundancyWe will try to show today that
• Sparsity & Redundancy can be used to design new/renewed & powerful signal/image processing tools (e.g., transforms, priors, models, compression, …),
• The obtained machinery works very well – we will show these ideas deployed to image processing applications.
Sparse and Redundant Signal Representation, and Its Role in Image Processing
3xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
Welcome to Sparsela
nd
Sparse and Redundant Signal Representation, and Its Role in Image Processing
4xD
Generating Signals in Sparseland
MK
N
DA fixed Dictionary
•Every column in D (dictionary) is a prototype signal (Atom).
•The vector is generated randomly with few non-zeros in random locations and random values.
A sparse & random vector
αx
N
Sparse and Redundant Signal Representation, and Its Role in Image Processing
5xD
• Simple: Every generated signal is built as a linear combination of few atoms from our dictionary D
• Rich: A general model: the obtained signals are a special type mixture-of-Gaussians (or Laplacians).
Sparseland Signals Are Special
Multiply by D
αDx
Mα
Sparse and Redundant Signal Representation, and Its Role in Image Processing
6xD
D
Transforms in Sparseland ?
Nx
MαD
M• Assume that x is known to emerge from . .
• We desire simplicity, independence, and expressiveness.
M• How about “Given x, find the α that generated it in ” ?
KαT
Sparse and Redundant Signal Representation, and Its Role in Image Processing
7xD
Known
So, In Order to Transform …
We need to solve an under-determined linear system of equations:
xαD
• We will measure sparsity using the L0 norm:
0α
• Among all (infinitely many) possible solutions we want the sparsest !!
Sparse and Redundant Signal Representation, and Its Role in Image Processing
8xD
-1 +1
1
pxxf
x
Measure of Sparsity?
k
1j
ppp xx
11
22
1p
pp
0p
pp
As p 0 we get a count of the non-zeros in the vector
0α
Sparse and Redundant Signal Representation, and Its Role in Image Processing
9xD
α̂• Are there practical ways to get ?
Signal’s Transform in Sparseland
α
αα
Dx.t.s
Min0 α̂Multipl
y by D
αDx
A sparse &
random vector
α
4 Major Questions
αα ˆ• Is ? Under which conditions?
• How effective are those ways?
• How would we get D?
Sparse and Redundant Signal Representation, and Its Role in Image Processing
10xD
Inverse Problems in Sparseland ?
• Assume that x is known to emerge from . M
MαD
Nx
vxy H• Suppose we observe , a “blurred” and noisy version of x with . How will we recover x?
ε2
v
xHMy
Noise
• How about “find the α that generated the x …” again?
Kˆ αQ
Sparse and Redundant Signal Representation, and Its Role in Image Processing
11xD
Inverse Problems in Sparseland ?
Multiply by D
αDxA sparse &
random vector
α “blur” by H
vxy H
v
4 Major Questions (again!)
• How would we get D?
α̂• Are there practical ways to get ?• How effective are those ways?
αα ˆ• Is ? How far can it go?
εα
αα
2
0
y
.t.sMin
HDα̂
Sparse and Redundant Signal Representation, and Its Role in Image Processing
12xD
Back Home … Any Lessons?
Several recent trends worth looking at:
JPEG to JPEG2000 - From (L2-norm) KLT to wavelet and non-linear approximation
From Wiener to robust restoration – From L2-norm (Fourier) to L1. (e.g., TV, Beltrami, wavelet shrinkage …) From unitary to richer representations – Frames, shift-invariance, steerable wavelet, contourlet, curvelet
Approximation theory – Non-linear approximation
ICA and related models
Sparsity.
Overcompleteness.
Sparsity.
Sparsity & Overcompleteness.
Independence and Sparsity.
Sparsela
nd is
HERE
Sparse and Redundant Signal Representation, and Its Role in Image Processing
13xD
To Summarize so far …
We do! this model is relevant to us
just as well.Who
cares?
So, what are the
implications?
Sparsity? looks
complicated(a) Practical solvers? (b) How will we get D?(c) Applications?
The Sparseland model for signals is
interesting since it is rich and yet every signal has
a simple description
M
We need to solve (or approximate the
solution of) a linear system with more
unknowns than equations, while finding
the sparsest possible solution
Sparse and Redundant Signal Representation, and Its Role in Image Processing
14xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
T
Q
Sparse and Redundant Signal Representation, and Its Role in Image Processing
15xD
Lets Start with the Transform …
known
ααα
Dx.t.sMin:P00
Put formally,
Our dream for Now: Find the sparsest solution of
Known
xαD
Sparse and Redundant Signal Representation, and Its Role in Image Processing
16xD
Multiply by D
αDx
α
M Question 1 – Uniqueness?
α
αα
Dx.t.s
Min0 α̂
Suppose we can
solve this exactly
αα ˆWhy should we necessarily get ?
It might happen that eventually .00ˆ αα
Sparse and Redundant Signal Representation, and Its Role in Image Processing
17xD
Matrix “Spark”
Rank = 4Rank = 4
Spark = 3Spark = 3
1 0 0 0 1
0 1 0 0 1
0 0 1 0 0
0 0 0 1 0
Example:
Donoho & E. (‘02)
Definition: Given a matrix D, =Spark{D} is the smallest and and number of columns that are linearly dependent.
* In tensor decomposition, Kruskal defined something similar already in 1989.
*
Sparse and Redundant Signal Representation, and Its Role in Image Processing
18xD
Uniqueness Rule
ααα
Dx.t.sMin:P00
Suppose this problem has been solved somehow
This result implies that if generates
signals using “sparse enough” , the solution of the above will find it exactly.
M
If we found a representation that satisfy
Then necessarily it is unique (the sparsest).
02α
σ
Uniqueness
Donoho & E. (‘02)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
19xD
α̂Are there reasonable ways to find ?
M
Question 2 – Practical P0 Solver?
α
αα
Dx.t.s
Min0 α̂Multipl
y by D
αDx
α
Sparse and Redundant Signal Representation, and Its Role in Image Processing
20xD
Matching Pursuit (MP)
Mallat & Zhang (1993)
• Next steps: given the previously found atoms, find the next one to best fit …
• The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients after each round.
• The MP is a greedy algorithm that finds one atom at a time.
• Step 1: find the one atom that best matches the signal.
Sparse and Redundant Signal Representation, and Its Role in Image Processing
21xD
Basis Pursuit (BP)
ααα
Dx.t.sMin0
Instead of solving
ααα
Dx.t.sMin1
Solve Instead
Chen, Donoho, & Saunders (1995)
• The newly defined problem is convex (linear programming).
• Very efficient solvers can be deployed: Interior point methods [Chen, Donoho, & Saunders (`95)] ,
Sequential shrinkage for union of ortho-bases [Bruce et.al. (`98)],
Iterated shrinkage [Figuerido & Nowak (`03), Daubechies, Defrise, &
Demole (‘04), E. (`05), E., Matalon, & Zibulevsky (`06)].
Sparse and Redundant Signal Representation, and Its Role in Image Processing
22xD
Multiply by D
αDx
α
M Question 3 – Approx. Quality?
α
αα
Dx.t.s
Min0 α̂
α̂How effective are the MP/BP in finding ?
Sparse and Redundant Signal Representation, and Its Role in Image Processing
23xD
Evaluating the “Spark”
• The Mutual Coherence M is the largest off-diagonal entry in absolute value.
DT
=D
DTD
• Compute
• The Mutual Coherence is a property of the dictionary (just like the “Spark”). In fact, the following relation can be shown:
M1
1
Assume normalized
columns
Sparse and Redundant Signal Representation, and Its Role in Image Processing
24xD
BP and MP Equivalence
Given a signal x with a representation ,
Assuming that , BP and MP are
Guaranteed to find the sparsest solution.
αDx
M115.00
αDonoho & E. (‘02) Gribonval & Nielsen
(‘03)Tropp (‘03)
Temlyakov (‘03)
Equivalence
MP and BP are different in general (hard to say which is better).
The above result corresponds to the worst-case.
Average performance results are available too, showing much better bounds [Donoho (`04), Candes et.al. (`04), Tanner et.al. (`05),
Tropp et.al. (`06)].
Sparse and Redundant Signal Representation, and Its Role in Image Processing
25xD
What About Inverse Problems?
εα
αα
2
0
y
.t.sMin
HDα̂“blur”
by H
vxy H
v
• We had similar questions regarding uniqueness, practical solvers, and their efficiency.
• It turns out that similar answers are applicable here due to several recent works [Donoho, E. and Temlyakov (`04), Tropp (`04),
Fuchs (`04), Gribonval et. al. (`05)].
Multiply by D
αDxA sparse &
random vector
α
Sparse and Redundant Signal Representation, and Its Role in Image Processing
26xD
To Summarize so far …
The Sparseland model for signals is
relevant to us. We can design transforms and
priors based on it
Use pursuit Algorithms
Find the sparsest solution?
A sequence of works during the
past 3-4 years gives theoretic justifications for
these tools behavior
Why works so well?
What next?(a) How shall we find
D?
(b) Will this work for applications? Which?
Sparse and Redundant Signal Representation, and Its Role in Image Processing
27xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
Sparse and Redundant Signal Representation, and Its Role in Image Processing
28xD
Problem Setting
Multiply by D
αDx
Mα
L0
α
Given these P examples and a fixed size [NK] dictionary D:
1. Is D unique?
2. How would we find D?
P1jjX
Sparse and Redundant Signal Representation, and Its Role in Image Processing
29xD
Uniqueness?
If is rich enough* and if
then D is unique.
2
SparkL
D
Uniqueness
Aharon, E., & Bruckstein (`05)
Comments:
M
L
K• “Rich Enough”: The signals from could be clustered to groups that share the same support. At least L+1 examples per each are needed.• This result is proved constructively, but the number of examples needed to pull this off is huge – we will show a far better method next. • A parallel result that takes into account noise is yet to be constructed.
P1jjx
Sparse and Redundant Signal Representation, and Its Role in Image Processing
30xD
Each example is a linear combination of atoms from
D
Practical Approach – Objective
DX A
Each example has a sparse representation with no more than L
atoms
L,j.t.sxMin0j
P
1j
2
2jj,
DAD
Field & Olshausen (96’)Engan et. al. (99’)
Lewicki & Sejnowski (00’)Cotter et. al. (03’)
Gribonval et. al. (04’)(n,K,L are assumed known, D has norm. columns)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
31xD
K–Means For Clustering
DInitialize D
Sparse Coding
Nearest Neighbor
Dictionary Update
Column-by-Column by Mean computation over the relevant
examples
XT
Clustering: An extreme sparse coding
Sparse and Redundant Signal Representation, and Its Role in Image Processing
32xD
The K–SVD Algorithm – General
DInitialize D
Sparse CodingUse MP or BP
Dictionary Update
Column-by-Column by SVD computation
Aharon, E., & Bruckstein (`04)
XT
Sparse and Redundant Signal Representation, and Its Role in Image Processing
33xD
K–SVD Sparse Coding Stage
D
X
L,j.t.sxMin0j
P
1j
2
2jj
ααDA
For the jth example we solve
L.t.sxMin0
2
2j ααα
D
Pursuit Problem !!!
T
Sparse and Redundant Signal Representation, and Its Role in Image Processing
34xD
K–SVD Dictionary Update Stage
DGk : The examples in and that use the column dk.
P
1jjX
?dk
The content of dk influences only the examples in Gk.
Let us fix all A apart from the kth column and seek both dk and the kth column to better fit the residual!
Sparse and Redundant Signal Representation, and Its Role in Image Processing
35xD
dk is obtained by SVD on the examples’ residual in Gk.
K–SVD Dictionary Update Stage
D?dk Residu
alE
kα
We should solve:
2
FkTk,d
dMinkk
Eαα
Sparse and Redundant Signal Representation, and Its Role in Image Processing
36xD
K–SVD: A Synthetic Experiment
D
Create A 2030 random dictionary with normalized
columns
Generate 2000 signal examples with 3 atoms per each and add
noise
0 5 10 15 20 25 30 35 40 45 500
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Iteration
Rel
ativ
e A
tom
s F
ound
MOD performance
K-SVD performance
Results
D
Train a dictionary using the KSVD and MOD and compare
Sparse and Redundant Signal Representation, and Its Role in Image Processing
37xD
To Summarize so far …
The Sparseland model for signals is
relevant to us. In order to use it effectively we
need to know D
Use the K-SVD algorithm
How D can be found?
We have established a
uniqueness result and shown how to practically train D using the K-SVD
Will it work well ?
What next?(a) Deploy to applications
(b) Generalize in various ways: multiscale, non-negative factorization, speed-up, …
(c) Performance guarantees?
Sparse and Redundant Signal Representation, and Its Role in Image Processing
38xD
Agenda
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
Sparse and Redundant Signal Representation, and Its Role in Image Processing
39xD
Application 1: Image Inpainting Assume: the signal x has been created
by x=Dα0 with very sparse α0.
Missing values in x imply missing rows in this linear system.
By removing these rows, we get .
Now solve
If α0 was sparse enough, it will be the solution of the
above problem! Thus, computing Dα0 recovers x perfectly.
x0 D
=x~~
D
D~
x~.t.s0Min
Sparse and Redundant Signal Representation, and Its Role in Image Processing
40xD
Application 1: The Practice
2
2pp
yArgMinˆ
D
Given a noisy image y, we can clean it using the Maximum A-posteriori Probability estimator by solving
What if some of the pixels in that image are missing (filled with zeros)? Define a mask operator as the diagonal matrix W, and now solve instead
ˆx̂ D
2
2pp
yArgMinˆ
WD ˆx̂ D
When handling general images, there is a need to concatenate two dictionaries to get an effective treatment of both texture and cartoon contents – This leads to separation [E., Starck, & Donoho (’05)].
Sparse and Redundant Signal Representation, and Its Role in Image Processing
41xD
Inpainting Results
Source
Outcome
Predetermined dictionary: Curvelet (cartoon) + Overlapped DCT (texture)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
42xD
Inpainting Results
Source
Outcome
predetermined dictionary: Curvelet (cartoon) + Overlapped DCT (texture)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
43xD
Inpainting Results20%
50%
80%
Sparse and Redundant Signal Representation, and Its Role in Image Processing
44xD
The solution: Force shift-invariant sparsity - on each patch of size N-by-N (e.g., N=8) in the image, including overlaps.
Application 2: Image Denoising
Given a noisy image y, we have already mentioned the ability to clean it by solving
When using the K-SVD, it cannot train a dictionary for large support images – How do we go from local treatment of patches to a global prior?
2
2pp
yArgMinˆ
D ˆx̂ D
Sparse and Redundant Signal Representation, and Its Role in Image Processing
45xD
Our prior
Extracts a patch in the ij
location
L.t.s
xyx21
ArgMinx̂
0
0ij
ij
2
2ijij2
2}{,x ijij
DR
Application 2: Image Denoising
2
2pp
yArgMinˆ
D ˆx̂ D
Our MAP penalty
becomes
Sparse and Redundant Signal Representation, and Its Role in Image Processing
46xD
K-SVD
L.t.sxyxArgMinx̂0
0ijij
2
2ijij222
1
,}{,x ijij
DRD
x=y and D known
L.t.s
xMin
00
2
2ijij
DR
Compute ij per patch
using the matching pursuit
x and ij known
ij
2
2ijxMin DR
Compute D to minimize
using SVD, updating one column at a time
D and ij known
ijij
Tij
1
ijij
Tij yIx DRRR
Compute x by
which is a simple averaging of shifted
patches
Application 2: Image Denoising
Sparse and Redundant Signal Representation, and Its Role in Image Processing
47xD
Sparse-code every patch of 8-by-8 pixels
Application 2: The Algorithm
Update the dictionary based
on the above codes
D
Compute the output image x
Complexity of this algorithm: O(N2×L×Iterations) per pixel. For N=8, L=1, and 10 iterations, we need 640 operations per pixel.
Sparse and Redundant Signal Representation, and Its Role in Image Processing
48xD
Initial dictionary (overcomplete DCT)
64×256
Denoising ResultsSource
Result 30.829dB
The obtained dictionary after 10 iterations
Noisy image
20
The results of this algorithm compete favorably with the state-of-the-art (e.g., GSM+steerable wavelets [Portilla, Strela,
Wainwright, & Simoncelli (‘03)] - giving ~0.5-1dB better results)
Sparse and Redundant Signal Representation, and Its Role in Image Processing
49xD
Application 3: Compression
The problem: Compressing photo-ID images.
General purpose methods (JPEG, JPEG2000) do not take into account the specific family.
By adapting to the image-content (PCA/K-SVD), better results could be obtained.
For these techniques to operate well, train dictionaries locally (per patch) using a training set of images is required.
In PCA, only the (quantized) coefficients are stored, whereas the K-SVD requires storage of the indices as well.
Geometric alignment of the image is very helpful and should be done.
Sparse and Redundant Signal Representation, and Its Role in Image Processing
50xD
Application 3: The Algorithm
Training set (2500 images)
Detect main features and warp the images to a
common reference (20 parameters)
On
the tra
inin
g se
t
Divide the image into disjoint 15-by-15 patches.
For each compute mean and dictionary
Per each patch find the operating parameters (number of atoms L,
quantization Q)
Warp, remove the mean from each patch, sparse code using L atoms, and
apply Q)
On the test image
Sparse and Redundant Signal Representation, and Its Role in Image Processing
51xD
Compression Results
Results for 820
Bytes per
each file
11.99
10.83
10.93
10.49
8.92
8.71
8.81
7.89
8.61
5.56
4.82
5.58
Sparse and Redundant Signal Representation, and Its Role in Image Processing
52xD
Compression Results
Results for 550
Bytes per
each file
15.81
14.67
15.30
13.89
12.41
12.57
10.66
9.44
10.27
6.60
5.49
6.36
Sparse and Redundant Signal Representation, and Its Role in Image Processing
53xD
Compression Results
Results for 400
Bytes per
each file
18.62
16.12
16.81
12.30
11.38
12.54
7.61
6.31
7.20
?
?
?
Sparse and Redundant Signal Representation, and Its Role in Image Processing
54xD
Today We Have Discussed
1. A Visit to Sparseland Motivating Sparsity & Overcompleteness
2. Problem 1: Transforms & Regularizations How & why should this work?
3. Problem 2: What About D?The quest for the origin of signals
4. Problem 3: ApplicationsImage filling in, denoising, separation, compression, …
Sparse and Redundant Signal Representation, and Its Role in Image Processing
55xD
Summary
Sparsity and Over- completeness are
important ideas that can be used in designing
better tools in signal/image processing
We are working on resolving those difficulties:• Performance of pursuit alg. • Speedup of those methods,• Training the dictionary,• Demonstrating applications,• …
There are difficulties in using
them!
Future transforms and regularizations will be data-driven, non-linear, overcomplete, and promoting sparsity.
The dream?
Top Related