Design of Non-Linear Kernel Dictionaries for Object Recognition
description
Transcript of Design of Non-Linear Kernel Dictionaries for Object Recognition
1
DESIGN OF NON-LINEAR KERNEL DICTIONARIES FOR
OBJECT RECOGNITION
Murad MegjhaniMATH : 6397
2
Agenda• Sparse Coding• Dictionary Learning• Problem Formulation (Kernel)• Results and Discussions
3
MotivationGiven a 16x16(or nxn) image patch x, we can represent it using 256 real numbers(pixels).
Problem: Can we find or learn a better representation for this?
Given a set of images, learn a better way to represent image other than pixels.
4
What is Sparse Linear Modelsignal.abenRLet x
nxKK1 R],...,d[dD Let
D is “adapted” to x if it can represent it with a few basis vectors—that is, there exists a sparse vector γ in Rk such that x ≈Dγ. We call γ the sparse code.
D
x
be a set of normalized “basis vectors”.Lets call it Dictionary.
5
Sparse Coding Illustration Natural Images Bases [d1 , …, d64]
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
50 100 150 200 250 300 350 400 450 500
50
100
150
200
250
300
350
400
450
500
» 0.8 * + 0.3 * + 0.5 *
x » 0.8 * d36 + 0.3 * d42
+ 0.5 * d63 [0, 0, …, 0, 0.8, 0, …, 0, 0.3, 0, …, 0, 0.5, …]
= [γ1, …, γ64] (feature representation)
Test example
Compact & easilyinterpretable
6
Notation
nxKD RKR
nRx nxNX R
KxNR
n dimensional Signal Vector
Matrix of N signal vectors of n dimension
Over-complete basis -Dictionary of size K (K>>n)
Sparse representation for input signal xMatrix of Sparse Vectors
nd R Atom : Elementary signal representing template
.
≈.
… …ΓX DN21 x..................x,x K21 ddd ..........,
N21 ..................,
7
Sparse Coding Problem)(γ ||D|| j
22jjγ j
xmin
ψ induces sparsity in γj
• The l0 “pseudo norm”. ||γ||0 ≡ #{i s.t. γ[i] ≠ 0} (NP-hard).
• The l1 norm. ||γ||1 ≡ ∑ | γ[i] | (convex).
data fitting Sparsity inducing regularization
This is a selection problem.
When ψ is the l1 norm, the problem is called LASSO [1] or Basis Pursuit [2].
When ψ is the l0 norm, the problem is called Matching Pursuit [3] or Orthogonal Matching Pursuit [4].
8
Dictionary Learning Problem)(γ ||DX||
21
j2FD
,
min
data fitting Sparsity inducing regularization
Designed DictionaryD can be designed like in Haar [5], Curvelets [6], …
Learned Dictionary D can be created from data as in “Sparse coding with an over-complete basis set” [7] and K-SVD [8]
We will study two of these algorithms today and see how they can be Kernel-aized [9].
10
• Next steps: given the previously found
atoms, find the next one to best fit the residual.
• The MP is one of the greedy algorithms that finds one atom at a time [4].
• Step 1: find the one atom that best matches the signal.
• The Orthogonal MP (OMP) is an improved version that re-evaluates the coefficients by Least-Squares after each round.
D
x
Sparse Coding Algorithms Matching Pursuit
11
Orthogonal Matching PursuitT tosubject D 2
2
xmin
x r residual and 0; tscoefficien ; Set Active 0 S
for iter = 1…T doSelect the atom which most reduces the objective
rd 1-iiSi C
,maxargi
Update the active set: }{SS i
Update the residual Dri x
Update the coefficients
x
,ˆ
DDDD : OMP
rd]i[ : MPTS
1
STSSS
1-ii
end for
1.
2.
3.
4.
5.
6.
7.
Input x, Dictionary D and Sparsity Threshold T
MP : Updates only one coefficient corresponding to the selected atom
OMP : Updates coefficient of all the coefficients in the active set
Initialize
.
≈.
… …ΓX D
Initialize D Sparse Code Dictionary Update
12
Dictionary Learning Algorithms:
K-SVD
13
The K-SVD algorithm:Train an explicit dictionary from examples
.
…
Set of Examples
X.
…ΓD
The examples are linear combinations of the atoms
Each representation uses at most T atoms
The target function to minimize:
T ||γ|| i s.t. ||DX|| 0i2FD,Γ
Γmin
Input Output
Dictionary Learning Algorithms:
K-SVD
14
For the jth example
For sparse coding, use batch OMP, or any other sparse coding algorithm
.
≈.
… …ΓX D
The Sparse Coding Stage:
T ||γ|| i s.t. ||D-X 0i2F Γ
Γ||min
T ||γ|| i s.t. ||D- 0j22jj
x||min
j
Dictionary Learning Algorithms:
K-SVD
15
For the kth atom
(Residual)
.
≈.
… …ΓX D
T ||γ|| s.t. i ||γdγd X|| ||γdX|| 0i2F
Tkk
kj
Tjj
2F
K
1j
TjjD D
minmin
T || s.t. i ||γdE|| 0i2F
Tkkkdk
||min kj
TjjK γd X E
Dictionary Update Stage:
T ||γ|| i s.t. ||D-X 0i2F Γ
D||min
Dictionary Learning Algorithms:
K-SVD
16
We can do better!
.
≈.
… …Γ
sparsity?
X D
Dictionary Update Stage:
||γdE|| 2F
Tkkkdk
min ||γdE|| 2F
Tkkkd kk
,
min
Dictionary Learning Algorithms:
K-SVD
17
d ,γmin
Tk k
Only some of the examples use atom
dk
When updating γk, only re-compute
the coefficients for those examples
dkγk
T
Solve with SVD
We want to solve:
Ek2
F
~ ~~
Dictionary Update Stage:
Dictionary Learning Algorithms:
K-SVD
N1i T tosubject DX0i
2
2D
...min,
Summary:
Input :X, Sparsity Threshold T
Initialization : Set the random normalized dictionary matrix D(0) ∈ Rn×K . Set J = 1Repeat until convergence ( or for some fixed number of iterations)
Sparse Coding Stage: Use any Pursuit algorithm to compute γi for i = 1…N
T ||γ|| i s.t. ||D|| 0i22ii
i
xmin
Codebook Update Stage : for k = 1…K
• Define the group of examples that use dk, ωk = {i|1 ≤ i ≤ N, xi(k) ≠ 0}
• Compute
kj
TjjK γd-XE
• Restrict Ek by choosing only the columns corresponding to those elements that initially used dk in their representation, and obtain R
KE
• Apply SVD decomposition TRK VUE
Update : 1Rk1k ).v11(ud ,,
Set J = J+1
Output : D,Γ 19
Dictionary Learning Algorithms:
K-SVD
20
Non-Linear Dictionary Learning Problem Formulation
dictionarysought the is D
in XX nxN ΗSpaceFeature )(R
Goal is to learn the non-linear dictionary in the feature space H by solving
T ||γ|| i s.t. ||DX)(|| 0i2FD,Γ
Γmin
Proposition 1**: There exists an optimal solution D* to (1) that has the following form
NxK Asomefor X)A(D R*
(1)
**For proof refer Appendix VI of H. Van Nguyen, V. M. Patel, N. M. Nasrabadi, and R. Chellappa, “Design of non-linear kernel dictionaries for object recognition.,” IEEE Trans. Image Process., vol. 22, no. 12, pp. 5123–35, Dec. 2013
T ||γ|| i s.t. ||X)A(X)(|| 0i2FD,Γ
Γmin
Equation (1) now can be written as
21
2
F
2F ( ||(( ΓΓ- A-IX)X)AX) ||
tr
(( tr
ΓΓ
ΓΓ
A-IXXTA-I
A-IX)TX)TA-I
,K
Matrix. Kernel a is XX ,K
This gives the motivation to formulate the dictionary learning in functional spaceusing the kernel trick.
Non-Linear Dictionary Learning Problem Formulation
Kernel-Orthogonal Matching PursuitT tosubject A)X()(
0
2
2
zmin
; r residual and 0; tscoefficien ; Set Active zS
for iter = 1…T do
iiiT
SiaX d In |dr|
C)(Space Hilbertmaxargi
Update the active set: }{SS i
SSS A
Update the coefficients
T
S
1
STSS
SSTS
1
STSSS
X)A,(X)A(X,A
)( and (X)A D In DDDD
zKK
zzSpace Hilbertz
end for
1.2.3.
4.
5.
6.
7.
Input: z, kernel ‘K’, Sparsity Threshold T, Coefficient Matrix A
Select the atom which most reduces the objective
iTSii
Si
iTS
SSSiT
S
iT
SS
iT
iT
a) X)(X,- Y)(z,( where )(
a) X)(X,- Y)(z,( A where )aX()X(
)aX()AX( )aX(r dr
CKKmaxargi
KK)()((z)
)()((z))(
22
Initialize:
Output: γ
Kernel-Dictionary Learning Algorithms: K-SVD Summary:
Input: X, Sparsity Threshold T, Kernel function K
Initialization: Set the random normalized dictionary matrix A(0) ∈ Rn×K . Set J = 1Repeat until convergence ( or for some fixed number of iterations)
Sparse Coding Stage: Use any KOMP to compute γi for i = 1…N given A(J-1)
Codebook Update Stage: for each column ak
(J-1) in A(J-1) k = 1…K• Define the group of examples that use ak,
ωk = {i|1 ≤ i ≤ N, xi(k) ≠ 0}• Compute
)γ(aM and )γa-(I where(X)M(X)
)γ(X)(a)γa-(X)(I
γ(X)aγ(X)a-(X)
γ(X)a-(X)
as written becan (X)A-(X)error The
kkkjjkkk
kkjj
kkjj
jj
kj
2F
2F
kj
2F
kj
2F
K
1j
2F
EE ||||
||||
||||
||||
||||
• Restrict Ek by choosing only the columns corresponding to those elements that initially used ak in their representation, and obtain R
KE
• Apply SVD decomposition vEa updated choose
VV)X)(E(X,)(E
1RK
1(J)k
TRK
RK
K
Set J = J+1
Output: A,Γ
N1i T tosubject A)X()X(0i
2
2D
...min,
kj
E )γa-(I jjk
Where v1 is the first vector of V corresponding to the largest singular value σ1
2 in Δ23
25
Results and Discussion• Synthetic data ( not discussed)• Kernel Sparse Representation• Digit Recognition• Caltech-101 and Caltech-256 Object Recognition
Kernel Sparse Representation
Compares the mean-squared-error (MSE) of an image from the USPS dataset when approximated using the first m dominant kernel PCA components and m = [1, . . . , 20] kernel dictionary atoms (i.e. T0 = m).
“It is clearly seen that the MSE decays much faster for kernel KSVD than kernel PCA with respect to the number of selected bases. This observation implies that the image is nonlinearly sparse and learning a dictionary in the high dimensional feature space can provide a better representation of data.”
26
•Kernel Sparse Representation•Digit Recognition•Caltech-101 and Caltech-256 Object Recognition
Results and Discussion
Dataset • USPS handwritten digit database. • Image Size 16x16 making the dimension of the vector 256 and number
of classes = 10 ( number of digits).• Ntrain = 500 samples for training and Ntest = 200 samples for testing – for
each class.
Parameter Selection• The selection of parameters is done through a 5-fold cross-validation.• Parameters
o K (size of dictionary) = 300 atoms.o T0 (sparsity constraint)= 5.o maximum number of training iterations = 80.o Kernel Type : polynomial kernel of degree 4.
Digit Recognition
27
Results and DiscussionDigit Recognition
Approach 1 : Distributive Approach
X1 = [xi1,…,xiN] ∈ R256×500 i=1...
X10 = [xi1,…,xiN] ∈ R256×500 i=10
compute
Training Examples A separate Dictionary for each class was learned using Kernel-KSVD
A1 R∈ 500×300 ...
A10 R∈ 500×300
Testing : Given a test image z R∈ 256 , perform KOMP
z R∈ 256
γ1 R∈ 300 using A1 ...
γ10 R∈ 300 using A10
r1
.
.
.r10
)r( i101i ...
minargclass
[1,...,10]i )A(A)A()( )( iiii iiTiii
2
2iii YYY2AXr ,Kz,Kzz,K(z)
Pre-images of learned atoms : Since the dictionary is learned in the kernel space the atoms in the dictionary need to be converted to back to the Euclidean space to view the atom**.
** J. T.-Y. Kwok and I. W.-H. Tsang, “The pre-image problem in kernel methods,” IEEE Trans. Neural Netw., vol. 15, no. 6, pp. 1517–1525, Nov. 2004.
28
Results and Discussion
Testing : Given a test image z R∈ 256 , perform KOMP on joined Dictionary [A1,…,A10 ] Sparsity Constraint T = 10( in this case)
z R∈ 256 γ1 R∈ 3000 using [A1,…,A10]
r1
.
.
.r10
)r( i101i ...
minargclass
** J. T. Kwok and I. W. Tsang, “The pre-image problem in kernel methods.,” IEEE Trans. Neural Netw., vol. 15, no. 6, pp. 1517–25, Nov. 2004..
Digit Recognition
Approach 2 : Collective Approach
X1 = [xi1,…,xiN] ∈ R256×500 i=1...
X10 = [xi1,…,xiN] ∈ R256×500 i=10
compute
Training Examples A separate Dictionary for each class was learned using Kernel-KSVD
A1 R∈ 500×300 ...
A10 R∈ 500×300
Pre-images of learned atoms : Since the dictionary is learned in the kernel space the atoms in the dictionary need to be converted to back to the Euclidean space to view the atom**[10].
[1,...,10]i )A(A)A()( )( iiii iiTiii
2
2iii YYY2AXr ,Kz,Kzz,K(z)
29
Results and Discussion
Comparison of digit recognition accuracies for different methods in the presence of Gaussian noise and missing-pixel effects. Red color and orange color represent the distributive and collective classification approaches for kernel KSVD, respectively. In order to avoid clutter, we only report distributive approach for kernel MOD which gives better performance for this dataset. (a) Missing pixels. (b) Gaussian noise.
Digit Recognition
Performance Comparison – KSVD, Kernel PCA, Kernel K-SVD and Kernel MOD under different scenarios – missing pixels, different noise.
Kernel KSVD classification accuracy versus the polynomial degree of the USPS dataset.
The second set of experiments examines the effects of parameters choices on the overall recognition performances. Figure shows the classification accuracy of kernel KSVD as we vary the degree of polynomial kernel. The best error rate of 1.6% is achieved with the polynomial degree of 4.
30
Results and DiscussionCaltech-101 and Caltech-256 Object Recognition
Dataset • Caltech-101 database. • 101 object classes, and 1 background class – collected randomly
from Internet• Each category contains 31-800 images• Average size of the image = 300x300 pixels• Diverse and challenging dataset as it includes objects like
building, musical instruments, animals and natural scenes
Parameter Selection• The selection of parameters is done through a 5-fold cross-validation• Parameters
o K (size of dictionary) = 300 atoms.o T0 (sparsity constraint)= 5.o Maximum number of training iterations =80.o Kernel Type : polynomial kernel of degree 4.
31
Results and DiscussionCaltech-101 and Caltech-256 Object Recognition
Confusion matrix of Kernel KSVD recognition performances on Caltech 101 dataset The rows and columns correspond to true labels and predicted labels, respectively. The dictionary is learned from 3030 images where each class contributes 30 images. The sparsity is set at 30 for both training and testing. Although the confusion matrix contains all classes, only a subset of class labels is displayed for better legibility.
• Train on N images where N={5,10,15,20,25,30} and test on rest.
• Some categories are very small so end up with just one single image for testing.
• To compensate for the variations in class size they normalize the recognition results by the number of test images to get per-class accuracies.
• Final results is obtained by averaging per-class accuracies across 102 categories.
32
Results and DiscussionCaltech-101 and Caltech-256 Object Recognition
#train samples 5 10 15 20 25 30
Griffin[11] 44.2 54.5 59.0 63.3 65.8 67.6
Germet[12] - - - - - 64.16
Yang[13] - - 67.0 - - 73.2
KSVD[8] 49.8 59.8 65.2 68.7 71.0 73.2
LC-KSVD[14] 54.0 63.1 67.7 70.5 72.3 73.6
D-Kernel (MOD) 53.8 64.0 70.2 73.8 76.4 78.1
C-Kernel (MOD) 56.2 67.7 72.4 75.6 77.5 80.0
D-Kernel (KSVD) 54.2 64.5 70.2 74.0 76.5 78.5
C-Kernel (KSVD) 56.5 67.2 72.5 75.8 77.6 80.1
Performance Comparison on Caltech – 101 Dataset
#train samples 15 30
Griffin[11] 28.3 34.1
Germet[12] - 27.2
Yang[13] 34.4 41.2
D-Kernel (MOD) 34.2 41.2
C-Kernel (MOD) 34.6 42.7
D-Kernel (KSVD) 34.5 41.4
C-Kernel (KSVD) 34.8 42.5
Performance Comparison on Caltech – 256 Dataset
33
References1. R. Tibshirani, “Regression Shrinkage and Selection via the Lasso,” J. R. Stat. Soc., vol. Vol. 58, no. No. 1, pp. 267–288,
1996.2. S. S. Chen, D. L. Donoho, and M. A. Saunders, “Atomic Decomposition by Basis Pursuit,” SIAM J. Sci. Comput., vol. 20,
no. 1, pp. 33–61, Jan. 1998.3. S. G. Mallat, “Matching pursuits with time-frequency dictionaries,” IEEE Trans. Signal Process., vol. 41, no. 12, pp. 3397–
3415, 1993.4. Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonal matching pursuit: recursive function approximation with
applications to wavelet decomposition,” in Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, 1993, pp. 40–44.
5. A. Haar, “Zur Theorie der orthogonalen Funktionensysteme,” Math. Ann., vol. 71, no. 1, pp. 38–53, Mar. 1911.6. E. J. Candes and D. L. Donoho, “Curvelets: A surprisingly effective nonadaptive representation for objects with edges,”
2000.7. B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete basis set: a strategy employed by V1?,” Vision Res.,
vol. 37, no. 23, pp. 3311–25, Dec. 1997.8. M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse
Representation,” IEEE Trans. Signal Process., vol. 54, no. 11, pp. 4311–4322, Nov. 2006.9. H. Van Nguyen, V. M. Patel, N. M. Nasrabadi, and R. Chellappa, “Design of non-linear kernel dictionaries for object
recognition.,” IEEE Trans. Image Process., vol. 22, no. 12, pp. 5123–35, Dec. 2013.10. J. T. Kwok and I. W. Tsang, “The pre-image problem in kernel methods.,” IEEE Trans. Neural Netw., vol. 15, no. 6, pp.
1517–25, Nov. 2004.11. G. Griffin, A. Holub, and P. Perona, “Caltech-256 Object Category Dataset.” California Institute of Technology, 10-Mar-
2007.12. J. C. van Gemert, J.-M. Geusebroek, C. J. Veenman, and A. W. M. Smeulders, “Kernel codebooks for scene
categorization,” in Computer Vision--ECCV 2008, Springer, 2008, pp. 696–709.13. T. Huang, “Linear spatial pyramid matching using sparse coding for image classification,” in 2009 IEEE Conference on
Computer Vision and Pattern Recognition, 2009, pp. 1794–1801. 14. Z. Jiang, Z. Lin, and L. S. Davis, “Learning a discriminative dictionary for sparse coding via label consistent K-SVD,” in
CVPR 2011, 2011, pp. 1697–1704.
34
Thank YouQ&A
(let Q be sparse )