Sparse and Overcomplete...
Transcript of Sparse and Overcomplete...
![Page 1: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/1.jpg)
Machine Learning for Signal Processing
Sparse and Overcomplete Representations
Bhiksha Raj (slides from Sourish Chaudhuri)
Oct 22, 2013
1
![Page 2: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/2.jpg)
Key Topics in this Lecture
• Basics – Component-based representations
– Overcomplete and Sparse Representations,
– Dictionaries
• Pursuit Algorithms
• How to learn a dictionary
• Why is an overcomplete representation powerful?
Sparse and Overcomplete Representations 2
![Page 3: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/3.jpg)
Representing Data
Dictionary (codebook)
3 Sparse and Overcomplete Representations
![Page 4: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/4.jpg)
Representing Data
Atoms Dictionary
4 Sparse and Overcomplete Representations
![Page 5: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/5.jpg)
Representing Data
Atoms Dictionary
Each atom is a basic unit that can be used to “compose” larger units.
5 Sparse and Overcomplete Representations
![Page 6: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/6.jpg)
Representing Data
Atoms
6 Sparse and Overcomplete Representations
![Page 7: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/7.jpg)
Representing Data
Atoms
7 Sparse and Overcomplete Representations
![Page 8: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/8.jpg)
Representing Data
Atoms
8 Sparse and Overcomplete Representations
![Page 9: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/9.jpg)
Representing Data
9 Sparse and Overcomplete Representations
![Page 10: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/10.jpg)
Representing Data
10 Sparse and Overcomplete Representations
![Page 11: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/11.jpg)
Representing Data
11 Sparse and Overcomplete Representations
![Page 12: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/12.jpg)
Representing Data
6.44
0.004
12.19
4.00
9.12
0.02
0.01
…………. 12 Sparse and Overcomplete Representations
![Page 13: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/13.jpg)
Representing Data
w1
………….
w6
w4
w3
w2
w5
w7
13 Sparse and Overcomplete Representations
![Page 14: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/14.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
N x 4096
14 Sparse and Overcomplete Representations
![Page 15: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/15.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
N x 4096
???
15 Sparse and Overcomplete Representations
![Page 16: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/16.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
N x 4096
VERY LARGE!!!
16 Sparse and Overcomplete Representations
![Page 17: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/17.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
N x 4096
VERY LARGE!!!
If N > 4096 (as it likely is) we have an overcomplete representation
17 Sparse and Overcomplete Representations
![Page 18: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/18.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
N x 4096
VERY LARGE!!!
More generally:
If #(basis vectors) > dimensions of input
we have an overcomplete representation
18 Sparse and Overcomplete Representations
![Page 19: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/19.jpg)
Representing Data
=
19 Sparse and Overcomplete Representations
![Page 20: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/20.jpg)
Representing Data
=
20 Sparse and Overcomplete Representations
![Page 21: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/21.jpg)
Representing Data
= D α X
Input 21 Sparse and Overcomplete Representations
![Page 22: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/22.jpg)
Representing Data
= D α X
Unknown
Input 22 Sparse and Overcomplete Representations
![Page 23: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/23.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
D.α = X
Basis Vectors
Input data
Unknowns
23 Sparse and Overcomplete Representations
![Page 24: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/24.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
Basis Vectors
Input data
Unknowns
When can we solve for α?
D.α = X
24 Sparse and Overcomplete Representations
![Page 25: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/25.jpg)
Quick Linear Algebra Refresher
• When #(Basis Vectors) = dim(Input Data), we have a unique solution
• When #(Basis Vectors) < dim(Input Data), we may have no solution
• When #(Basis Vectors) > dim(Input Data), we have infinitely many solutions
D.α = X
25 Sparse and Overcomplete Representations
![Page 26: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/26.jpg)
Quick Linear Algebra Refresher
• When #(Basis Vectors) = dim(Input Data), we have a unique solution
• When #(Basis Vectors) < dim(Input Data), we may have no solution
• When #(Basis Vectors) > dim(Input Data), we have infinitely many solutions
D.α = X
26 Sparse and Overcomplete Representations
Our Case
![Page 27: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/27.jpg)
Overcomplete Representations
#(Basis Vectors) > dimensions of the input
27 Sparse and Overcomplete Representations
![Page 28: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/28.jpg)
Overcomplete Representation
= D α X
Unknown
#(Basis Vectors) > dimensions of the input 28 Sparse and Overcomplete Representations
![Page 29: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/29.jpg)
Overcomplete Representations
• Why do we use them?
• How do we learn them?
29 Sparse and Overcomplete Representations
![Page 30: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/30.jpg)
Overcomplete Representations
• Why do we use them?
– A more natural representation of the real world
– More flexibility in matching data
– Can yield a better approximation of the statistical distribution of the data.
• How do we learn them?
30 Sparse and Overcomplete Representations
![Page 31: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/31.jpg)
Overcompleteness and Sparsity
• To solve an overcomplete system of the type:
• Make assumptions about the data.
• Suppose, we say that X is composed of no more than a fixed number (k) of “bases” from D (k ≤ dim(X)) – The term “bases” is an abuse of terminology..
• Now, we can find the set of k bases that best fit the data point, X.
D.α = X
31 Sparse and Overcomplete Representations
![Page 32: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/32.jpg)
Representing Data
Using bases that we know…
But no more than k=4 bases
32 Sparse and Overcomplete Representations
![Page 33: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/33.jpg)
Overcompleteness and Sparsity
Atoms
But no more than k=4 bases are “active”
33 Sparse and Overcomplete Representations
![Page 34: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/34.jpg)
Overcompleteness and Sparsity
Atoms
But no more than k=4 bases
34 Sparse and Overcomplete Representations
![Page 35: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/35.jpg)
No more than 4 bases
= D
α
X
35 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
![Page 36: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/36.jpg)
No more than 4 bases
= D
α
X
36 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY DICTIONARY ENTRIES ARE NON-ZERO
![Page 37: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/37.jpg)
No more than 4 bases
= D
α
X
37 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY DICTIONARY ENTRIES ARE NON-ZERO
MOST OF a IS ZERO!! a IS SPARSE
![Page 38: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/38.jpg)
Sparsity- Definition
• Sparse representations are representations that account for most or all information of a signal with a linear combination of a small number of atoms.
(from: www.see.ed.ac.uk/~tblumens/Sparse/Sparse.html)
38 Sparse and Overcomplete Representations
![Page 39: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/39.jpg)
The Sparsity Problem
• We don’t really know k
• You are given a signal X
• Assuming X was generated using the dictionary, can we find α that generated it?
39 Sparse and Overcomplete Representations
![Page 40: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/40.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
40 Sparse and Overcomplete Representations
![Page 41: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/41.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
41 Sparse and Overcomplete Representations
Counts the number of non-zero elements in α
![Page 42: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/42.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
How can we solve the above?
42 Sparse and Overcomplete Representations
![Page 43: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/43.jpg)
Obtaining Sparse Solutions
• We will look at 2 algorithms:
– Matching Pursuit (MP)
– Basis Pursuit (BP)
Sparse and Overcomplete Representations 43
![Page 44: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/44.jpg)
Matching Pursuit (MP)
• Greedy algorithm
• Finds an atom in the dictionary that best matches the input signal
• Remove the weighted value of this atom from the signal
• Again, find an atom in the dictionary that best matches the remaining signal.
• Continue till a defined stop condition is satisfied.
44 Sparse and Overcomplete Representations
![Page 45: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/45.jpg)
Matching Pursuit
• Find the dictionary atom that best matches the given signal.
Weight = w1
45 Sparse and Overcomplete Representations
![Page 46: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/46.jpg)
Matching Pursuit
• Remove weighted image to obtain updated signal
Find best match for this signal from the dictionary
46 Sparse and Overcomplete Representations
![Page 47: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/47.jpg)
Matching Pursuit
• Find best match for updated signal
47 Sparse and Overcomplete Representations
![Page 48: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/48.jpg)
Matching Pursuit
• Find best match for updated signal
Iterate till you reach a stopping condition, norm(ResidualInputSignal) < threshold
48 Sparse and Overcomplete Representations
![Page 49: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/49.jpg)
Matching Pursuit
From http://en.wikipedia.org/wiki/Matching_pursuit
49 Sparse and Overcomplete Representations
![Page 50: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/50.jpg)
Matching Pursuit
• Problems ???
50 Sparse and Overcomplete Representations
![Page 51: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/51.jpg)
Matching Pursuit
• Main Problem
– Computational complexity
– The entire dictionary has to be searched at every iteration
51 Sparse and Overcomplete Representations
![Page 52: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/52.jpg)
Comparing MP and BP
Matching Pursuit Basis Pursuit
Hard thresholding
Greedy optimization at each step
Weights obtained using greedy rules
(remember the equations)
52 Sparse and Overcomplete Representations
![Page 53: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/53.jpg)
Basis Pursuit (BP)
• Remember,
Mina
a0
s.t. X Da
53 Sparse and Overcomplete Representations
![Page 54: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/54.jpg)
Basis Pursuit
• Remember,
Mina
a0
s.t. X Da
In the general case, this is intractable
54 Sparse and Overcomplete Representations
![Page 55: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/55.jpg)
Basis Pursuit
• Remember,
Mina
a0
s.t. X Da
In the general case, this is intractable
55 Sparse and Overcomplete Representations
Requires combinatorial optimization
![Page 56: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/56.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
56 Sparse and Overcomplete Representations
![Page 57: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/57.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
57 Sparse and Overcomplete Representations
This holds when D obeys the Restricted Isometry Property.
![Page 58: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/58.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
58 Sparse and Overcomplete Representations
Constraint
Objective
![Page 59: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/59.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
59 Sparse and Overcomplete Representations
Constraint Objective
![Page 60: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/60.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
λ is a penalty term on the non-zero elements and promotes sparsity
60 Sparse and Overcomplete Representations
![Page 61: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/61.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
λ is a penalty term on the non-zero elements and promotes sparsity
Equivalent to LASSO; for more details, see this paper by Tibshirani
61 Sparse and Overcomplete Representations
![Page 62: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/62.jpg)
Basis Pursuit
• There are efficient ways to solve the LASSO formulation. [Link to Matlab code]
62 Sparse and Overcomplete Representations
![Page 63: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/63.jpg)
Comparing MP and BP
Matching Pursuit Basis Pursuit
Hard thresholding
Soft thresholding
Greedy optimization at each step
Global optimization
Weights obtained using greedy rules
Can force N-sparsity with appropriately
chosen weights
(remember the equations)
63 Sparse and Overcomplete Representations
![Page 64: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/64.jpg)
Many Other Methods..
• Iterative Hard Thresholding (IHT)
• CoSAMP
• OMP
• …
64 Sparse and Overcomplete Representations
![Page 65: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/65.jpg)
Applications of Sparse Representations
• Many many applications – Signal representation
– Statistical modelling
– ..
• Two extremely popular signal processing applications: – Compressive sensing
– Denoising
65 Sparse and Overcomplete Representations
![Page 66: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/66.jpg)
Compressive Sensing
• Recall the Nyquist criterion?
• To reconstruct a signal, you need to sample at twice the maximum frequency of the original signal
66 Sparse and Overcomplete Representations
![Page 67: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/67.jpg)
Compressive Sensing
• Recall the Nyquist criterion?
• To reconstruct a signal, you need to sample at twice the frequency of the original signal
• Is it possible to reconstruct signals when they have not been sampled so as to satisfy the Nyquist criterion?
67 Sparse and Overcomplete Representations
![Page 68: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/68.jpg)
Compressive Sensing
• Recall the Nyquist criterion?
• To reconstruct a signal, you need to sample at twice the frequency of the original signal
• Is it possible to reconstruct signals when they have not been sampled so as to satisfy the Nyquist criterion?
• Under specific criteria, yes!!!!
68 Sparse and Overcomplete Representations
![Page 69: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/69.jpg)
Compressive Sensing
• What criteria?
69 Sparse and Overcomplete Representations
![Page 70: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/70.jpg)
Compressive Sensing
• What criteria?
Sparsity!
70 Sparse and Overcomplete Representations
![Page 71: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/71.jpg)
Compressive Sensing
• What criteria?
• Exploit the structure of the data
• Most signals are sparse, in some domain
Sparsity!
71 Sparse and Overcomplete Representations
![Page 72: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/72.jpg)
Applications of Sparse Representations
• Two extremely popular applications:
– Compressive sensing
• You will hear more about this in Aswin’s class
– Denoising
72 Sparse and Overcomplete Representations
![Page 73: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/73.jpg)
Applications of Sparse Representations
• Two extremely popular applications:
– Compressive sensing
– Denoising
73 Sparse and Overcomplete Representations
![Page 74: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/74.jpg)
Denoising
• As the name suggests, remove noise!
74 Sparse and Overcomplete Representations
![Page 75: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/75.jpg)
Denoising
• As the name suggests, remove noise!
• We will look at image denoising as an example
75 Sparse and Overcomplete Representations
![Page 76: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/76.jpg)
Image Denoising
• Here’s what we want
76 Sparse and Overcomplete Representations
![Page 77: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/77.jpg)
Image Denoising
• Here’s what we want
77 Sparse and Overcomplete Representations
![Page 78: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/78.jpg)
Image Denoising
• Here’s what we want
78 Sparse and Overcomplete Representations
![Page 79: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/79.jpg)
Denoising
• As the name suggests, remove noise!
• We will look at image denoising as an example
A more general take-away: How to learn the dictionaries
79 Sparse and Overcomplete Representations
![Page 80: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/80.jpg)
The Image Denoising Problem
• Given an image
• Remove Gaussian additive noise from it
80 Sparse and Overcomplete Representations
![Page 81: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/81.jpg)
Image Denoising
Y = X + ε
Noisy Input Orig. Image
Gaussian Noise ε = Ν(0, σ)
81 Sparse and Overcomplete Representations
![Page 82: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/82.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible.
82 Sparse and Overcomplete Representations
![Page 83: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/83.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
83 Sparse and Overcomplete Representations
![Page 84: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/84.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
• Yes, we will learn the dictionaries
84 Sparse and Overcomplete Representations
![Page 85: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/85.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
• Yes, we will learn the dictionaries
• What data will we use? The corrupted image itself!
85 Sparse and Overcomplete Representations
![Page 86: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/86.jpg)
Image Denoising
• We use the data to be denoised to learn the dictionary.
• Training and denoising become an iterated process.
• We use image patches of size √n x √n pixels (i.e. if the image is 64x64, patches are 8x8)
86 Sparse and Overcomplete Representations
![Page 87: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/87.jpg)
Image Denoising
• The data dictionary D
– Size = n x k (k > n)
– This is known and fixed, to start with
– Every image patch can be sparsely represented using D
87 Sparse and Overcomplete Representations
![Page 88: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/88.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
88 Sparse and Overcomplete Representations
![Page 89: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/89.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
Can Matching Pursuit solve this?
89 Sparse and Overcomplete Representations
![Page 90: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/90.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
Can Matching Pursuit solve this?
90 Sparse and Overcomplete Representations
Yes
![Page 91: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/91.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
Can Matching Pursuit solve this?
91 Sparse and Overcomplete Representations
Yes
What constraints does it need?
![Page 92: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/92.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
Can Basis Pursuit solve this?
92 Sparse and Overcomplete Representations
![Page 93: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/93.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
But this is intractable!
93 Sparse and Overcomplete Representations
![Page 94: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/94.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
1}
Can Basis Pursuit solve this?
94 Sparse and Overcomplete Representations
![Page 95: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/95.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
1}
Can Basis Pursuit solve this? Yes
95 Sparse and Overcomplete Representations
![Page 96: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/96.jpg)
Image Denoising
• In the above, X is a patch.
Mina{ X Da
2 a
1}
96 Sparse and Overcomplete Representations
![Page 97: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/97.jpg)
Image Denoising
• In the above, X is a patch.
• If the larger image is fully expressed by the every patch in it, how can we go from patches to the image?
Mina{ X Da
2 a
1}
97 Sparse and Overcomplete Representations
![Page 98: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/98.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
98 Sparse and Overcomplete Representations
![Page 99: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/99.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
(X – Y) is the error between the input and denoised image. μ is a penalty on the error.
99 Sparse and Overcomplete Representations
![Page 100: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/100.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
Error bounding in each patch -what is Rij? -How many terms in the summation?
100 Sparse and Overcomplete Representations
![Page 101: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/101.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
λ forces sparsity
101 Sparse and Overcomplete Representations
![Page 102: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/102.jpg)
Image Denoising
• But, we don’t “know” our dictionary D.
• We want to estimate D as well.
102 Sparse and Overcomplete Representations
![Page 103: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/103.jpg)
Image Denoising
• But, we don’t “know” our dictionary D.
• We want to estimate D as well.
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
We can use the previous equation itself!!!
103 Sparse and Overcomplete Representations
![Page 104: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/104.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
104 Sparse and Overcomplete Representations
![Page 105: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/105.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
We cannot estimate them at the same time!
105 Sparse and Overcomplete Representations
![Page 106: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/106.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
Fix 2, and find the optimal 3rd.
106 Sparse and Overcomplete Representations
![Page 107: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/107.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Initialize X = Y
107 Sparse and Overcomplete Representations
![Page 108: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/108.jpg)
Image Denoising
Mina ij
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Initialize X = Y, initialize D
0
You know how to solve the remaining portion for α – MP, BP!
108 Sparse and Overcomplete Representations
![Page 109: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/109.jpg)
Image Denoising
• Now, update the dictionary D.
• Update D one column at a time, following the K-SVD algorithm
• K-SVD maintains the sparsity structure
109 Sparse and Overcomplete Representations
![Page 110: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/110.jpg)
Image Denoising
• Now, update the dictionary D.
• Update D one column at a time, following the K-SVD algorithm
• K-SVD maintains the sparsity structure
• Iteratively update α and D
110 Sparse and Overcomplete Representations
![Page 111: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/111.jpg)
K-SVD vs K-Means
• Kmeans: Given data Y
– Find D and a such that
– Error = ||Y – Da||2 is minimized, with constraint
– |ai|0 = 1
• K-SVD
– Find D and a such that
– Error = ||Y – Da||2 is minimized, with constraint
– |ai|0 < T
111 Sparse and Overcomplete Representations
![Page 112: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/112.jpg)
Image Denoising
• Updating D • For each basis vector, compute its contribution to the
image
Sparse and Overcomplete Representations 112
Ek Y D ja j
jk
![Page 113: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/113.jpg)
Image Denoising
• Updating D • For each basis vector, compute its contribution to the
image
• Eigen decomposition of Ek
Sparse and Overcomplete Representations 113
Ek UVT
![Page 114: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/114.jpg)
K-SVD
• Updating D • For each basis vector, compute its contribution to the
image
• Eigen decomposition of Ek
• Take the principal eigen vector as the updated basis vector.
• Update every entry in D
Sparse and Overcomplete Representations 114
Dk U1
![Page 115: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/115.jpg)
K-SVD
• Initialize D
• Estimate a
• Update every entry in D
• Iterate
Sparse and Overcomplete Representations 115
![Page 116: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/116.jpg)
Image Denoising
Sparse and Overcomplete Representations 116
Learned Dictionary for Face Image denoising
From: M. Elad and M. Aharon, Image denoising via learned dictionaries and sparse representation, CVPR, 2006.
![Page 117: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/117.jpg)
MinX{ X Y
2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
We know D and α
Const. wrt X
The quadratic term above has a closed-form solution
117 Sparse and Overcomplete Representations
![Page 118: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/118.jpg)
MinX{ X Y
2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
We know D and α
Const. wrt X
X (I RijTR)1(Y
ij
RijTDa ij )
ij
118 Sparse and Overcomplete Representations
![Page 119: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/119.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Sparse and Overcomplete Representations 119
![Page 120: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/120.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α
Dictionary D
Denoised Image X
Sparse and Overcomplete Representations 120
![Page 121: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/121.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α – Your favorite pursuit algorithm
Dictionary D – Using K-SVD
Denoised Image X
Sparse and Overcomplete Representations 121
![Page 122: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/122.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α – Your favorite pursuit algorithm
Dictionary D – Using K-SVD
Denoised Image X
Sparse and Overcomplete Representations 122
Iterating
![Page 123: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/123.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α
Dictionary D
Denoised Image X- Closed form solution
Sparse and Overcomplete Representations 123
![Page 124: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/124.jpg)
K-SVD algorithm (skip)
124 Sparse and Overcomplete Representations
![Page 125: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/125.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA of ICA Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
125 Sparse and Overcomplete Representations
Which is which?
![Page 126: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/126.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA ICA
126 Sparse and Overcomplete Representations
Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
![Page 127: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/127.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA ICA
Predicts data here
Doesn’t predict data here
Does pretty well
127 Sparse and Overcomplete Representations
Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
![Page 128: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/128.jpg)
Comparing to Other Techniques
Data still in 2-D space
ICA Overcomplete
Doesn’t capture the underlying representation, which Overcomplete representations can do…
128 Sparse and Overcomplete Representations
![Page 129: Sparse and Overcomplete Representationsmlsp.cs.cmu.edu/courses/fall2013/lectures/slides/class15.sparseove… · •Make assumptions about the data. •Suppose, we say that X is composed](https://reader036.fdocuments.us/reader036/viewer/2022081615/5fd25ba3ee37da607e18ab8f/html5/thumbnails/129.jpg)
Summary
• Overcomplete representations can be more powerful than component analysis techniques.
• Dictionary can be learned from data.
• Relative advantages and disadvantages of the pursuit algorithms.
129 Sparse and Overcomplete Representations