Component Analysis Methods for Signal Processing
Transcript of Component Analysis Methods for Signal Processing
![Page 1: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/1.jpg)
Component Analysis Methods
for Signal Processing
Component Analysis for Signal Processing 1
Fernando De la TorreFernando De la Torre
[email protected]@cs.cmu.edu
![Page 2: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/2.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Component Analysis for Signal Processing 2
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
![Page 3: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/3.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Structure from motion
Component Analysis for Signal Processing 3
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
![Page 4: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/4.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Spectral graph methods for segmentation.
Component Analysis for Signal Processing 4
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
![Page 5: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/5.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Appearance and shape models
Component Analysis for Signal Processing 5
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
![Page 6: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/6.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Component Analysis for Signal Processing 6
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
cocktail problem
![Page 7: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/7.jpg)
Component Analysis (ICA)
Sound
Source 1
Sound
Source 2
Mixture 1
Mixture 2
Output 1
I
Component Analysis for Signal Processing 7
Mixture 2
Output 2
I
C
A
Sound
Source 3
Mixture 3
Output 3
![Page 8: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/8.jpg)
Component Analysis for PR • Computer Vision & Image Processing
– Structure from motion.
– Spectral graph methods for segmentation.
– Appearance and shape models.
– Fundamental matrix estimation and calibration.
– Compression.
– Classification.
– Dimensionality reduction and visualization.
Component Analysis for Signal Processing 8
– Dimensionality reduction and visualization.
• Signal Processing– Spectral estimation, system identification (e.g. Kalman filter), sensor array processing (e.g. cocktail problem, eco cancellation), blind source separation, …
• Computer Graphics– Compression (BRDF), synthesis,…
• Speech, bioinformatics, combinatorial problems.
![Page 9: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/9.jpg)
Why Component Analysis for PR?
• Learn from high dimensional data and few samples.– Useful for dimensionality reduction.
• Easy to incorporate – Robustness to noise, missing data, outliers (de la Torre & Black, 2003a)
– Invariance to geometric transformations (de la Torre & Black, 2003b; de la Torre & Nguyen,2007)
Component Analysis for Signal Processing 9
features samples
• Efficient methods O( d n< <n2 )
(Everitt,1984)
Torre & Nguyen,2007)
– Non-linearities (Kernel methods) (Scholkopf & Smola,2002; Shawe-Taylor & Cristianini,2004)
– Probabilistic (latent variable models)
– Multi-factorial (tensors) (Paatero & Tapper, 1994 ;O’Leary & Peleg,1983; Vasilescu & Terzopoulos,2002; Vasilescu & Terzopoulos,2003)
– Exponential family PCA (Gordon,2002; Collins et al. 01)
![Page 10: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/10.jpg)
Are CA Methods Popular/Useful/Used?
• About 20% of CVPR-06 papers use CA.
• Google:– Results 1 - 10 of about 1,870,000 for "principal component
analysis".
– Results 1 - 10 of about 506,000 for "independent componentanalysis".
– Results 1 - 10 of about 273,000 for "linear discriminantanalysis".
Component Analysis for Signal Processing 10
• Still work to do
– Results 1 - 10 of about 65,300,000 for "Britney Spears".
– Results 1 - 10 of about 273,000 for "linear discriminantanalysis".
– Results 1 - 10 of about 46,100 for "negative matrix
factorization".
– Results 1 - 10 of about 491,000 for "kernel methods".
![Page 11: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/11.jpg)
Outline• Introduction
• Generative models– Principal Component Analysis (PCA)
– Non-negative Matrix Factorization (NMF)
– Independent Component Analysis (ICA)
– Multidimensional Scaling (MDS)
• Discriminative models– Linear Discriminant Analysis (LDA).
Component Analysis for Signal Processing 11
– Linear Discriminant Analysis (LDA).
– Oriented Component Analysis (OCA).
– Canonical Correlation Analysis (CCA).
• Standard extensions of linear models– Kernel methods.
– Latent variable models.
– Tensor factorization
![Page 12: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/12.jpg)
Principal Component Analysis (PCA)(Pearson, 1901; Hotelling, 1933;Mardia et al., 1979; Jolliffe, 1986; Diamantaras, 1996)
Component Analysis for Signal Processing 12
• PCA finds the directions of maximum variation of thedata based on linear correlation.
• PCA decorrelates the original variables.
![Page 13: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/13.jpg)
PCA
ccc ++++≈ ......µ
kbbb 21
[ ] T
nn µ1BCdddD +≈= ...21
d=d=pixels
pixels
nn= images= images
1××
××
ℜ∈ℜ∈
ℜ∈ℜ∈dnk
kdnd
µC
BD
Component Analysis for Signal Processing 13
kccc ++++≈ ......21µ
•Assuming 0 mean data, the basis B that preserve the maximumvariation of the signal is given by the eigenvectors of DDT.
B ΛBDD =Td
d
![Page 14: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/14.jpg)
Snap-shot Method & SVD• If d>>n (e.g. images 100*100 vs. 300 samples) no DDT.
• DDT and DTD have the same eigenvalues (energy) and
related eigenvectors (by D).
• B is a linear combination of the data!
• [α,L]=eig(DTD) B=D α(diag(diag(L))) -0.5
ΛDαDDαDDDDαBBΛBDDTTTT ===
(Sirovich, 1987)
Component Analysis for Signal Processing 14
SVDPCA
diagonal
nnnkkd
T
ΣIVVIUU
VΣU
VUΣD
TT ==
ℜ∈ℜ∈ℜ∈
=×××
• SVD factorizes the data matrix D as:
Λ==
ℜ∈ℜ∈
=××
TT CCIBB
CB
BCD
nkkd
TTUUΛDD =
TTVVΛDD =
(Beltrami, 1873; Schmidt, 1907; Golub & Loan, 1989)
![Page 15: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/15.jpg)
Error Function for PCA
(Eckardt & Young, 1936; Gabriel & Zamir, 1979; Baldi & Hornik, 1989; Shum et al.,
1995; de la Torre & Black, 2003a)
• Not unique solution:
• To obtain same PCA solution R has to satisfy:
kk×− ℜ∈= RBCCBRR 1
F
n
i
iiE BCDBcdCB, −=−=∑=1
2
21 )(
• PCA minimizes the following CONVEX function.
Component Analysis for Signal Processing 15
• To obtain same PCA solution R has to satisfy:
• R is computed as a generalized k×k eigenvalue problem.
Λ==
== −
TTCCIBB
CRCBRB
ˆˆˆˆ
ˆˆ 1
( ) 11 −−Λ= BRBRCC
TT(de la Torre, 2006)
![Page 16: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/16.jpg)
PCA/SVD in Computer Vision• PCA/SVD has been applied to:
– Recognition (eigenfaces:Turk & Pentland, 1991; Sirovich & Kirby, 1987; Leonardis & Bischof, 2000; Gong et al., 2000; McKenna et al., 1997a)
– Parameterized motion models (Yacoob & Black, 1999; Black et al., 2000; Black, 1999; Black & Jepson, 1998)
– Appearance/shape models (Cootes & Taylor, 2001; Cootes et al., 1998; Pentland et al., 1994; Jones & Poggio, 1998; Casia & Sclaroff, 1999; Black & Jepson, 1998; Blanz & Vetter, 1999; Cootes et al., 1995; McKenna et al., 1997; de la Torre et al., 1998b; de la Torre et al., 1998b)
– Dynamic appearance models (Soatto et al., 2001; Rao, 1997; Orriols & Binefa, 2001; Gong et al., 2000)
– Structure from Motion (Tomasi & Kanade, 1992; Bregler et al., 2000; Sturm &
Component Analysis for Signal Processing 16
– Structure from Motion (Tomasi & Kanade, 1992; Bregler et al., 2000; Sturm & Triggs, 1996; Brand, 2001)
– Illumination based reconstruction (Hayakawa, 1994)
– Visual servoing (Murase & Nayar, 1995; Murase & Nayar, 1994)
– Visual correspondence (Zhang et al., 1995; Jones & Malik, 1992)
– Camera motion estimation (Hartley, 1992; Hartley & Zisserman, 2000)
– Image watermarking (Liu & Tan, 2000)
– Signal processing (Moonen & de Moor, 1995)
– Neural approaches (Oja, 1982; Sanger, 1989; Xu, 1993)
– Bilinear models (Tenenbaum & Freeman, 2000; Marimont & Wandell, 1992)
– Direct extensions (Welling et al., 2003; Penev & Atick, 1996)
![Page 17: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/17.jpg)
“Intercorrelations among
variables are the bane of the
multivariate researcher’s struggle
for meaning”Cooley and Lohnes, 1971
Component Analysis for Signal Processing 17
![Page 18: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/18.jpg)
Part-based Representation
�The firing rates of neurons are never negative.
Component Analysis for Signal Processing 18
�The firing rates of neurons are never negative.
� Independent representations.
NMF & ICA
![Page 19: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/19.jpg)
Non-negative Matrix Factorization
• Positive factorization.
• Leads to part-based representation.
0||||)( ≥−= CB,BCDCB, FE
Component Analysis for Signal Processing 19
![Page 20: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/20.jpg)
Nonnegative Factorization
∑ −=≥≥
ij
ijijdF2
0,0)(min BC
CB
ij
ij
ijij)(
)(
BVB
DBCC
T
T
←
Inference:
Derivatives:
F)()( CBBCB
TT −=∂
(Lee & Seung, 1999;Lee & Seung, 2000)
Component Analysis for Signal Processing
• Multiplicative algorithm can be interpreted as
diagonally rescaled gradient descent.
Learning:
ij
T
ij
T
ijij)(
)(
BCC
DCBB ←
ijij
ij
F)()( CBBCB
C
TT −=∂∂
ij
T
ij
T
ij
F)()( DCBCC
B−=
∂∂
![Page 21: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/21.jpg)
Independent Component Analysis
• We need more than second order statistics to represent
the signal.
Component Analysis for Signal Processing 21
![Page 22: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/22.jpg)
ICA
• Look for si that are independent.
• PCA finds uncorrelated variables, the independent
components have non Gaussian distributions.
• Uncorrelated E(sisj)= E(si)E(sj)
• Independent E(g(si)f(sj))= E(g(si))E(f(sj)) for any non-
linear f,g
1−≈=≈= BWWDSCBCD
(Hyvrinen et al., 2001)
Component Analysis for Signal Processing 22
i j i j
linear f,g
PCA ICA
![Page 23: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/23.jpg)
ICA vs PCA
Component Analysis for Signal Processing 23
![Page 24: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/24.jpg)
Many optimization criteria
• Minimize high order moments: e.g. kurtosis
kurt(W) = E{s4} -3(E{s2}) 2
• Many other information criteria.
(Olhausen & Field, 1996)• Also an error function:
Component Analysis for Signal Processing 24
∑∑==
+−n
i
i
n
i
ii S11
)(cBcdSparseness (e.g. S=| |)
(Chennubhotla & Jepson, 2001b; Zou et al., 2005; dAspremont et al., 2004;)
• Other sparse PCA.
![Page 25: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/25.jpg)
Basis of natural images
Component Analysis for Signal Processing 25
![Page 26: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/26.jpg)
Denoising
Original
image Noisy Image
(30% noise)
Component Analysis for Signal Processing 26
Denoise
(Wiener filter) ICA
![Page 27: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/27.jpg)
Multidimensional Scaling (MDS)
• MDS takes a matrix of pair-wise distances
and finds an embedding that preserves the
interpoint distances.
Component Analysis for Signal Processing 27
![Page 28: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/28.jpg)
Component Analysis for Signal Processing 28
![Page 29: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/29.jpg)
Component Analysis for Signal Processing 29
![Page 30: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/30.jpg)
Outline• Introduction
• Generative models– Principal Component Analysis (PCA)
– Non-negative Matrix Factorization (NMF)
– Independent Component Analysis (ICA)
– Multidimensional Scaling (MDS)
• Discriminative models– Linear Discriminant Analysis (LDA).
Component Analysis for Signal Processing 30
– Linear Discriminant Analysis (LDA).
– Oriented Component Analysis (OCA).
– Canonical Correlation Analysis (CCA).
• Standard extensions of linear models– Kernel methods.
– Latent variable models.
– Tensor factorization
![Page 31: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/31.jpg)
Linear Discriminant Analysis (LDA)
∑∑= =
−−=C
i
C
j
T
jijib
1 1
))(( µµµµS
(Fisher, 1938;Mardia et al., 1979; Bishop, 1995)
∑==n
TT ddDDS
Component Analysis for Signal Processing 31
• Optimal linear dimensionality reduction if classes are
Gaussian with equal covariance matrix.
BΛSBSBSB
BSBB b
bt
t
T
T
J ==||
||)(
∑=
==i
T
ii
T
t
1
ddDDS
T
ji
c
j
C
i
jiw
i
)()(1 1
µdµdS −−=∑∑= =
![Page 32: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/32.jpg)
Oriented Component Analysis (OCA)
OCA
T
OCA
T
OCA
noiseOCA
signal
bΣb
bΣbsignal
noise
OCAb
Component Analysis for Signal Processing 32
• Generalized eigenvalue problem:
• boca is steered by the distribution of noise.
λkeki bΣbΣ =
![Page 33: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/33.jpg)
OCA for face recognition
k
T
k
T
k
ek
i
bΣb
bΣb1
1
j
T
j
j
T
j
e
i
bΣb
bΣb2
2
Component Analysis for Signal Processing 33
kek bΣb jj e
bΣb
![Page 34: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/34.jpg)
Canonical Correlation Analysis
(CCA)• PCA independently and general mapping
Component Analysis for Signal Processing 34
• Signals dependent signals with small energy can be lost.
PCA PCA
![Page 35: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/35.jpg)
Canonical Correlation Analysis (CCA)
• Learn relations between multiple data sets? (e.g. find
features in one set related to another data set)
• Given two sets , CCA finds the pair
of directions wx and wy that maximize the correlation
between the projections (assume zero mean data)
y
TT
x YwXw=ρ
ndndand
×× ℜ∈ℜ∈ 21 YX
(Mardia et al., 1979; Borga)
Component Analysis for Signal Processing 35
• Several ways of optimizing it:
• An stationary point of r is the solution to CCA.
T
y
TT
y
T
x
TT
x YwYwXwXw=ρ
=ℜ∈
=ℜ∈
= +×++×+
y
xdddd
T
Tdddd
T
T
w
ww
YY0
0XXΒ
0YX
YX0A
)()()()( 21212121 ,
ΒwAw λ=
![Page 36: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/36.jpg)
Dynamic Coupled Component Analysis
Component Analysis for Signal Processing 36
![Page 37: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/37.jpg)
Robot localization with Canonical
Correlation Analysis(Skocaj & Leonardis, 2000)
Component Analysis for Signal Processing 37
![Page 38: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/38.jpg)
Outline• Introduction
• Generative models– Principal Component Analysis (PCA)
– Non-negative Matrix Factorization (NMF)
– Independent Component Analysis (ICA)
– Multidimensional Scaling (MDS)
• Discriminative models– Linear Discriminant Analysis (LDA).
Component Analysis for Signal Processing 38
– Linear Discriminant Analysis (LDA).
– Oriented Component Analysis (OCA).
– Canonical Correlation Analysis (CCA).
• Standard extensions of linear models– Kernel methods.
– Latent variable models.
– Tensor factorization
![Page 39: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/39.jpg)
Kernel Methods
Component Analysis for Signal Processing 39
Kernel Methods
![Page 40: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/40.jpg)
Linear methods fail
Component Analysis for Signal Processing 40
![Page 41: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/41.jpg)
Linear methods fail • Learning a non-linear representation for classification
Component Analysis for Signal Processing 41
![Page 42: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/42.jpg)
Kernel Methods for Classification
Feature spaceInput space
),,(),,(),(321
22
212121
zzzxxxxxx =+→
Component Analysis for Signal Processing 42
• The kernel defines an implicit mapping (usually high dimensional and
non-linear) from input to feature space, so the data becomes linearly
separable.
• Computation in the feature space can be costly because it is
(usually) high dimensional
– The feature space is typically infinite-dimensional!
Feature spaceInput space
![Page 43: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/43.jpg)
Kernel Methods• Suppose φ(.) is given as follows
• An inner product in the feature space is
• So, if we define the kernel function as follows, there is no need to carry out φ(.) explicitly
Component Analysis for Signal Processing 43
need to carry out φ(.) explicitly
• This use of kernel function to avoid carrying out φ(.) explicitly is known as the kernel trick. In any linear algorithm that can be expressed by inner products can be made nonlinear by going to the feature space
![Page 44: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/44.jpg)
Kernel PCA(Scholkopf et al., 1998)
Component Analysis for Signal Processing 44
![Page 45: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/45.jpg)
Kernel PCA
• Eigenvectors of the cov. Matrix in feature space.
• Eigenvectors lie in the span of data in feature space.
∑=
ΦΦ=n
i
iin 1
T)()(1
ddC λ11 bbC =
n
(Scholkopf et al., 1998)
Component Analysis for Signal Processing 45
• Eigenvectors lie in the span of data in feature space.
∑=
Φ=n
i
ii
1
1 )(db α
λαKα =
λαα ])([),()(1
1 11
∑ ∑∑= ==
Φ=Φn
i
i
n
i
ijii
n
j
i Kn
dddd
![Page 46: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/46.jpg)
Latent Variable Models
Component Analysis for Signal Processing 46
Latent Variable Models
![Page 47: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/47.jpg)
Factor Analysis• A Gaussian distribution on the coefficients and noise is
added to PCA� Factor Analysis.
• Inference (Roweis & Ghahramani, 1999;Tipping & Bishop, 1999a)
Ψ+=−−=
=ΨΨ=
Ψ+==
++=
TT
d
k
E
diagNp
NpNp
BBµdµdd
0,cη
BcµdBcdI0,cc
ηBcµd
)))((()cov(
),...,,()|()(
),|(),|()|()(
21 ηηη
(Mardia et al., 1979)
Component Analysis for Signal Processing 47
• Inference (Roweis & Ghahramani, 1999;Tipping & Bishop, 1999a)
11
1
)(
)()(
),|()(
−−
−
Ψ+=
−Ψ+=
=
BBIV
µdBBBm
Vmcd|c
T
TT
Np
PCA reconstruction low error.
FA high reconstruction error (low likelihood).
),( dcp Jointly Gaussian
![Page 48: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/48.jpg)
Ppca• If PPCA.
• If is equivalent to PCA. TTTTBBBBBB
11 )()(0 −− =Ψ+→ε
d
TE Iηη ε==Ψ )(
0→ε
• Probabilistic visual learning (Moghaddam & Pentland, 1997;)
∑
=
Σ
=
Σ
== −
−−−+−−−Σ−−
∏∫
=
−−
2
)(
2
)(
21
2
2
1
12
)()()(2
1
12
)()(2
1
)2()2()2()2(
)()()(
2
1
2
11
kdkd
c
dd
eeeedppp
k
i i
iTT
πρλπππ
ρε
λεd
µdIBBµdµdµd T
ccc|dd
Component Analysis for Signal Processing 48
ΣΣ
=∏ 2
1
21
22222 )2()2()2()2(i
i πρλπππ
i
T
i dBc =
![Page 49: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/49.jpg)
Tensor Factorization
Component Analysis for Signal Processing 49
Tensor Factorization
![Page 50: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/50.jpg)
Tensor faces(Vasilescu & Terzopoulos, 2002; Vasilescu & Terzopoulos, 2003)
expressions
people
Component Analysis for Signal Processing 50
viewsilluminations
![Page 51: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/51.jpg)
Eigenfaces• Facial images (identity change)
• Eigenfaces bases vectors capture the variability in facial
appearance (do not decouple pose, illumination, …)
Component Analysis for Signal Processing 51
![Page 52: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/52.jpg)
Data Organization
• Linear/PCA: Data Matrix
– Rpixels x images
– a matrix of image vectors
• Multilinear: Data Tensor
Views
D
D
Pix
els
ImagesD
Component Analysis for Signal Processing 52
• Multilinear: Data Tensor
– Rpeople x views x illums x express x pixels
– N-dimensional matrix
– 28 people, 45 images/person
– 5 views, 3 illuminations,
3 expressions per personexilvpp ,,,iIl
lum
ina
tion
s
D
![Page 53: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/53.jpg)
N-Mode SVD Algorithm
N = 3
pixelsxexpressx
illums.x
views x
peoplex 51
UUUUU .... ZZZZDDDD 432=
Component Analysis for Signal Processing 53
![Page 54: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/54.jpg)
PCA:
TensorFaces:
Component Analysis for Signal Processing 54
![Page 55: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/55.jpg)
TensorFaces
Mean Sq. Err. = 409.15
3 illum + 11 people param.
PCA
Mean Sq. Err. = 85.75
33 parameters
Strategic Data Compression =
Perceptual Quality
Original
TensorFaces
6 illum + 11 people param.
• TensorFaces data reduction in illumination space primarily
degrades illumination effects (cast shadows, highlights)
• PCA has lower mean square error but higher perceptual error
Component Analysis for Signal Processing 55
3 illum + 11 people param.
33 basis vectors
33 parameters
33 basis vectors
Original
176 basis vectors
6 illum + 11 people param.
66 basis vectors
![Page 56: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/56.jpg)
Acknowledgments
• The content of some of the slides has been taken from previous presentations/papers of:– Ales Leonardis.
– Horst Bischof.
– Michael Black.
– Rene Vidal.
– Anat Levin.
– Aleix Martinez.
Component Analysis for Signal Processing 56
– Aleix Martinez.
– Juha Karhunen.
– Andrew Fitzgibbon.
– Daniel Lee.
– Chris Ding.
– M. Alex Vasilescu.
– Sam Roweis.
– Daoqiang Zhang.
– Ammon Shashua.
![Page 57: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/57.jpg)
Thanks
CA
Component Analysis for Signal Processing 57
![Page 58: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/58.jpg)
Bibliography
Component Analysis for Signal Processing 58
![Page 59: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/59.jpg)
Component Analysis for Signal Processing 59
![Page 60: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/60.jpg)
Component Analysis for Signal Processing 60
![Page 61: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/61.jpg)
Component Analysis for Signal Processing 61
![Page 62: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/62.jpg)
Component Analysis for Signal Processing 62
![Page 63: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/63.jpg)
Component Analysis for Signal Processing 63
![Page 64: Component Analysis Methods for Signal Processing](https://reader036.fdocuments.us/reader036/viewer/2022080222/62e86fcb0b65a013af340ed9/html5/thumbnails/64.jpg)
Bibliography
Zhou F., De la Torre F. and Hodgins J. (2008) "Aligned Cluster
Analysis for Temporal Segmentation of Human Motion“ IEEE
Conference on Automatic Face and Gestures Recognition, September,
Component Analysis for Signal Processing 64
Conference on Automatic Face and Gestures Recognition, September,
2008.
De la Torre, F. and Nguyen, M. (2008) “Parameterized Kernel
Principal Component Analysis: Theory and Applications to
Supervised and Unsupervised Image Alignment“ IEEE Conference
on Computer Vision and Pattern Recognition, June, 2008.