1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M....
-
date post
22-Dec-2015 -
Category
Documents
-
view
244 -
download
0
Transcript of 1 M. Bronstein Multigrid multidimensional scaling Multigrid Multidimensional Scaling Michael M....
1M. Bronstein Multigrid multidimensional scaling
Multigrid Multidimensional Scaling
Michael M. Bronstein
Department of Computer ScienceTechnion – Israel Institute of Technology
2M. Bronstein Multigrid multidimensional scaling
Agenda
Applications of MDS
Numerical optimization algorithms
Motivation for multiresolution MDS methods
Multigrid MDS
Experimental results
Conclusions
3M. Bronstein Multigrid multidimensional scaling
Dimensionality reduction
Visualization
Pattern recognition
Feature extraction
Data analysis
WRIST ROTATION
FIN
GE
R E
XT
EN
SIO
N
Low-dimensional representation of articulated hand images, showing intrinsic data dimensionality
Images: World Wide Web
4M. Bronstein Multigrid multidimensional scaling
Given a surface sampled at points , and the
geodesic distances on ;
Find a mapping (isometric embedding)
such that
Isometric embedding
5M. Bronstein Multigrid multidimensional scaling
GLOBE (HEMISPHERE) PLANAR MAP
Mapmaking
Given: geodesic distances between cities on the Earth
Find: the “best” (most distance-preserving) planar map of the cities
Optimal planar representation of the upper hemisphere of the Earth
6M. Bronstein Multigrid multidimensional scaling
Pattern recognition
A. Elad, R. Kimmel, Proc. CVPR 2001
ISOMETRIES OF A DEFORMABLE OBJECT
ISOMETRY-INVARIANT REPRESENTATIONS (“CANONICAL FORMS”)
Isometry-invariant representation of deformable objects using isometric embedding
7M. Bronstein Multigrid multidimensional scaling
Expression-invariant face recognition
ISOMETRIC EMBEDDING
FACIAL CONTOURCROPPING
FACESUBSAMPLING
CANONICAL FORM
Facial expressions ~ isometries of the facial surface
Obtain expression-invariant representation using isometric embedding
Compare the canonical forms
A. Bronstein, M. Bronstein, R. Kimmel, Proc. AVBPA 2003; IJCV 2005
Scheme of expression-invariant 3D face recognition based on isometric embedding
DISTANCESCOMPUTATION
8M. Bronstein Multigrid multidimensional scaling
Stress
Given a set of distances ;
and a configuration of points in -dimensional
Euclidean
space ;
Representation quality can be measured as the -distortion of the
distances (stress)
9M. Bronstein Multigrid multidimensional scaling
Multidimensional scaling
Stress in matrix form:
- matrix of geodesic distances (data);
- matrix of Euclidean coordinates (variable);
Multidimensional scaling (MDS) problem:
optimization variables
Optimum defined up to an isometry in
10M. Bronstein Multigrid multidimensional scaling
Minimization of the stress
Generic iterative optimization algorithm:
Start with an initial guess ;
At -st iteration, make a step of size in direction
such that
Repeat until a stopping condition is met, e.g.
11M. Bronstein Multigrid multidimensional scaling
Optimization algorithms
Gradient descent: , step size is constant orfound using line search
Newton: , step size is found using line search
Truncated Newton: direction obtained by inexact solution of
step size is chosen to guarantee descent
Quasi-Newton: direction obtained by estimating using the gradients ; step size is found using linesearch
12M. Bronstein Multigrid multidimensional scaling
Difficulties
Non-convex and nonlinear optimization problem (local convergence)
Hessian structured but dense
High computation complexity of and
Exact line search is prohibitive for large
13M. Bronstein Multigrid multidimensional scaling
SMACOF algorithm
SMACOF: steepest descent with constant step size
where
and
Can be also written as a multiplicative update
Complexity: per iteration
14M. Bronstein Multigrid multidimensional scaling
Multiresolution methods: motivation
Data smoothness and locality (a point can be interpolated from itsneighbors)
Complexity: - MDS problem is easier on coarser resolution
Local minima: multiple resolutions improve global convergence
15M. Bronstein Multigrid multidimensional scaling
Towards multigrid MDS
Convex nonlinear optimization is equivalent to a nonlinear equation
Multigrid spirit: solve problems of the form
at different resolution levels
- residual transferred from finer resolution levels
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
16M. Bronstein Multigrid multidimensional scaling
Modified stress
Problem: the function is unbounded
Modified stress:
The penalty term forces the center of mass of to zero
With modified stress, is bounded for every finite
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
17M. Bronstein Multigrid multidimensional scaling
Multigrid components
Hierarchy of grids
Restriction and prolongation operators to transfer data and
variables
from one resolution level to another
Hierarchy of optimization problems
Relaxation: steps of optimization algorithm
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
18M. Bronstein Multigrid multidimensional scaling
Coarsening schemes
In parameterization domain (suitable for parametric surfaces, e.g.acquired by 3D scanner)
Triangulation-based (suitable for general triangulated meshes)
Farthest point sampling (based on the distances matrix; suitable for arbitrary multidimensional data)
19M. Bronstein Multigrid multidimensional scaling
V-cycle
If (coarsest level), solve and return
Else
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
Relaxation
Compute
Apply MG on coarser resolution
Correction
Relaxation
20M. Bronstein Multigrid multidimensional scaling
Error smoothing
BEFORE RELAXATION AFTER RELAXATION
Error smoothing using SMACOF relaxation
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear
21M. Bronstein Multigrid multidimensional scaling
Numerical experiments
Embedding of the “Swiss roll” surface – comparison of MDS
algorithmsconvergence in a large scale problem
Computation of canonical forms for face recognition
Sensitivity to initialization and comparison on problems of different
size
Dimensionality reduction
22M. Bronstein Multigrid multidimensional scaling
Experiment I: Unrolling the Swiss roll
Embedding of the Swiss roll objects into R3 using MG-MDS. N=2145
INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3 ITERATION 4
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear
23M. Bronstein Multigrid multidimensional scaling
Experiment I: Convergence comparison
Convergence of different algorithms in the Swiss roll problem
COMPLEXITY (MFLOPs)
ST
RE
SS
EXECUTION TIME (sec.)
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear
24M. Bronstein Multigrid multidimensional scaling
Experiment II: Facial surface embedding
Computation of a facial canonical form using MG-MDS: as few as 3 iterations are sufficient to obtain a good expression-invariant representation. N=1997
INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear
25M. Bronstein Multigrid multidimensional scaling
Performance of SMACOF and MG (V-cycle, 3 resolution levels) MDS algorithms using random initialization
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
Experiment III: Sensitivity to initialization
26M. Bronstein Multigrid multidimensional scaling
Boosting obtained by multigrid MDS (V-cycle, 3 resolution levels) compared to SMACOF. Initialization by the original points
M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005
Experiment III: Performance comparison
27M. Bronstein Multigrid multidimensional scaling
Experiment IV: Dimensionality reduction
Dimensionality reduction of 500-dimensional random data: as few as 3 iterations are sufficient to obtain distinguishable clusters.
INITIALIZATION ITERATION 1 ITERATION 2 ITERATION 3
Two sets of random binary i.i.d. 500-dimensional vectors
Set A:
Set B:
28M. Bronstein Multigrid multidimensional scaling
MG-MDS significantly outperforms traditional MDS algorithms(~order of magnitude)
The improvement is more pronounced for large N
MG-MDS appears to be less sensitive to initialization and has better global convergence
Conclusions
29M. Bronstein Multigrid multidimensional scaling
ReferencesM. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "Multigrid multidimensional
scaling", NLAA, to appear in 2006
M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "A multigrid approach for multi-dimensional scaling", Proc. Copper Mountain Conf. Multigrid Methods, 2005.
A. M. Bronstein, M. M. Bronstein, and R. Kimmel. “Expression invariant face recognition:
faces as isometric surfaces”, in “Face Processing: Advanced Modeling and Methods”,Academic Press, 2005. in press.
A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Three-dimensional face recognition", Intl. Journal of Computer Vision (IJCV), Vol. 64/1, pp. 5-30, August 2005.
A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Expression-invariant 3D face recognition", Proc. AVBPA, Lecture Notes in Comp. Science No. 2688, Springer, pp. 62-69, 2003.