Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang,...
-
Upload
reynard-reeves -
Category
Documents
-
view
218 -
download
2
Transcript of Total Variation and Euler's Elastica for Supervised Learning Tong Lin, Hanlin Xue, Ling Wang,...
Total Variation and Euler's Elastica for Supervised Learning
Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha
Contact: [email protected]
Peking University, China
2012-6-29
1Key Lab. Of Machine Perception, School of EECS,
Peking University, China
Background• Supervised Learning:
• Definition: Predict u: x -> y, with training data (x1, y1), …, (xN, yN)
• Two tasks: Classification and Regression
• Prior Work:• SVM:
• RLS: Regularized Least Squares, Rifkin, 2002
2
Hinge loss:
Squared loss:
3
Background• Prior Work (Cont.):
• Laplacian Energy: “Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples,” Belkin et al., JMLR 7:2399-2434, 2006
• Hessian Energy: “Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction,” K.I. Kim, F. Steinke, M. Hein, NIPS 2009
• GLS: “Classification using geometric level sets,” Varshney & Willsky, JMLR 11:491-516, 2010
4
Motivation
SVM Our Proposed Method
5
Large margin should not be the sole criterion; we argue sharper edges and smoother boundaries can play significant roles.
3D display of the output classification function u(x) by the proposed EE model
6
• General:
• Laplacian Regularization (LR):
• Total Variation (TV):
• Euler’s Elastica (EE):
1min ( ( ), ) ( )
n
i iiuL u x y S u
2 2min ( ) | |u
u y dx u dx
2 2min ( ) ( ) | |u
u y dx a b u dx
2min ( ) | |u
u y dx u dx
| |
u
u
Models
7
TV&EE in Image Processing• TV: a measure of total quantity of the value change• Image denoising (Rudin, Osher, Fatemi, 1992)
• Elastica was introduced by Euler in 1744 on modeling torsion-free elastic rods
• Image inpainting (Chan et al., 2002)
8
• TV can preserve sharp edges, while EE can produce smooth boundaries
• For details, see T. Chan & J. Shen’s textbook: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, 2005
9
Decision boundary
The mean curvature k in high dimensional space can have same expression except the constant 1/(d-1).
Framework
10
11
• The calculus of variations → Euler-Lagrange PDE
3
1 1( ) ( '( ) | |) ( ( '( ) | |))
| | | |V n u u u u
u u
2min [ ] ( ) ( )J u u y dx S u
2( ) | |LRS u u dx
( ) | |TVS u u dx
2( ) ( ) | |EES u a b u dx
2( ) 0 (#)u u y 2( ) 0| |
uu y
u
2( ) 0V u y
2( ) a b
Energy Functional Minimization
12
Solutions
a. Laplacian Regularization (LR)
Radial Basis Function Approximation
b. TV & EE: We develop two solutions• Gradient descent time marching (GD)• Lagged linear equation iteration (LagLE)
13
Experiments: Two-Moon Data
SVM
EE
Both methods can achieve 100% accuracies with different parameter combinations
14
Experiments: Binary Classification
15
Experiments: Multi-class Classification
16
Experiments: Multi-class Classification
Note: Results of TV and EE are computed by the LagLE method.
17
Experiments: Regression
18
Conclusions• Contributions:
• Introduce TV&EE to the ML community
• Demonstrate the significance of curvature and gradient empirically
• Achieve superior performance for classification and regression
• Future Work:• Hinge loss
• Other basis functions
• Extension to semi-supervised setting
• Existence and uniqueness of the PDE solutions
• Fast algorithm to reduce the running time
End, thank you!