Nonlinear Dimensionality Reduction Frameworks
description
Transcript of Nonlinear Dimensionality Reduction Frameworks
![Page 1: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/1.jpg)
Nonlinear Dimensionality Reduction Frameworks
Rong Xu
Chan su Lee
![Page 2: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/2.jpg)
Outline
• Intuition of Nonlinear Dimensionality Reduction(NLDR)
• ISOMAP
• LLE
• NLDR in Gait Analysis
![Page 3: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/3.jpg)
Intuition: how does your brain store these pictures?
![Page 4: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/4.jpg)
Brain Representation
![Page 5: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/5.jpg)
Brain Representation
• Every pixel?
• Or perceptually meaningful structure?– Up-down pose
– Left-right pose
– Lighting direction
So, your brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!
![Page 6: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/6.jpg)
Data for Faces
![Page 7: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/7.jpg)
Data for Handwritten “2”
![Page 8: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/8.jpg)
Data for Hands
![Page 9: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/9.jpg)
Manifold Learning
• A manifold is a topological space which is locally Euclidean
• An example of nonlinear manifold:
![Page 10: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/10.jpg)
Manifold Learning• Discover low dimensional
representations (smooth manifold) for data in high dimension.
• Linear approaches(PCA, MDS) vs Non-linear approaches (ISOMAP, LLE)
d
i Ry Y
XN
i Rx
latent
observed
![Page 11: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/11.jpg)
Linear Approach- PCA
• PCA Finds subspace linear projections of input data.
![Page 12: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/12.jpg)
Linear Approach- MDS
• MDS takes a matrix of pairwise distances and gives a mapping to Rd. It finds an embedding that preserves the interpoint distances, equivalent to PCA when those distance are Euclidean.
• BUT! PCA and MDS both fail to do embedding with nonlinear data, like swiss roll.
![Page 13: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/13.jpg)
Nonlinear Approaches- ISOMAP
• Constructing neighbourhood graph G• For each pair of points in G, Computing shortest
path distances ---- geodesic distances.• Use Classical MDS with geodesic distances.
Euclidean distance Geodesic distance
Josh. Tenenbaum, Vin de Silva, John langford 2000
![Page 14: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/14.jpg)
Sample points with Swiss Roll
• Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.
![Page 15: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/15.jpg)
Construct neighborhood graph G
K- nearest neighborhood (K=7)
DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)
![Page 16: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/16.jpg)
Compute all-points shortest path in G
Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold(figure B)
![Page 17: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/17.jpg)
Find a d-dimensional Euclidean space Y(Figure c) to minimize the cost function:
Use MDS to embed graph in Rd
![Page 18: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/18.jpg)
22
222
22
2
2
)(min||||min
||)(5.0||min||)()(||min
1*11
2
1)(
2
1)(
YYXXtraceYYXX
HDDHDD
NIH
DD
HHDD
HHDD
TTTT
GYG
ijG
YY
GG
Y
Theorem: For any squared distance matrix ,there exists of points xi and,xj, such that
So
2GD
)()'(2
jijiij xxxxd
XXD TG 2
Linear Approach-classical MDS
![Page 19: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/19.jpg)
Solution
dxNdddxN
N
v
v
v
yyyY
22
11
21
|
|
|
|
|
|
VY dY2/1
)(
Y lies in Rd and consists of N points correspondent to the N original points in input space.
![Page 20: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/20.jpg)
PCA, MD vs ISOMAP
![Page 21: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/21.jpg)
• Nonlinear
• Globally optimal• Still produces globally optimal low-dimensional Euclidean
representation even though input space is highly folded,
twisted, or curved.
• Guarantee asymptotically to recover the true dimensionality.
Isomap: Advantages
![Page 22: Nonlinear Dimensionality Reduction Frameworks](https://reader036.fdocuments.us/reader036/viewer/2022062423/5681439e550346895db01b33/html5/thumbnails/22.jpg)
• May not be stable, dependent on topology of data
• Guaranteed asymptotically to recover geometric structure of nonlinear manifolds– As N increases, pairwise distances provide better
approximations to geodesics, but cost more computation
– If N is small, geodesic distances will be very inaccurate.
Isomap: Disadvantages