SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open...
Transcript of SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open...
![Page 1: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/1.jpg)
SUPPORT VECTOR MACHINE IN MATLAB
Weicheng Guo
![Page 2: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/2.jpg)
PROBLEM DEFINITION
2
We classify red points (in a circle with radius of 1) as 1 and green points (in a ring with radius from 1 to 2) as -1 in this figure. We want to find the hyperplane separating these two classes. Our decision boundary will have equation:
π€π€ππx+b=0
![Page 3: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/3.jpg)
FINDING THE DECISION BOUNDARY
3
Let {x1,β¦, xn} be the training data set and let yiβ{1,-1} be the class label of xi
The decision boundary should classify all points correctly with
The decision boundary can be found by solving the following constrained optimization problem
π¦π¦ππ π€π€πππ₯π₯ππ + ππ β₯ 1 β ππππ
minββ12βπ€π€β2 + πΆπΆοΏ½ππππ
ππ
ππ=1π π π π πππ π π π π π π π ββββπ π π‘π‘ββββπ¦π¦ππ π€π€πππ₯π₯ππ + ππ β₯ 1 β ππππ
![Page 4: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/4.jpg)
REFORMULATING AS A LAGRANGIAN
4
We can introduce Lagrange multipliers to represent the condition and thus have the following formulation:
j1 1 1
1
1max x x2
subject to 0 , 0
n n nT
i i j i j ii i j
n
i i ii
y y
C y
Ξ± Ξ±Ξ±
Ξ± Ξ±
= = =
=
β
β€ β€ =
β ββ
β
![Page 5: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/5.jpg)
KERNEL TRICK
5
Since it is a nonlinear classification in this case, we can use kernel functions that operates on the lower dimension vectors xi and xj to produce a value equivalent to the dot product of the higher-dimensional vectors. Therefore, the data can be linearly separated in that space and the formulation is transformed to:
1 1 1
1
1max ( , )2
subject to 0 , 0
n n n
i i j i j i ji i j
n
i i ii
y y K x x
C y
Ξ± Ξ±Ξ±
Ξ± Ξ±
= = =
=
β
β€ β€ =
β ββ
β
![Page 6: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/6.jpg)
KERNEL FUNCTIONS
6
Linear kernel
Polynomial kernel with Ξ³, d and r
Radial basis function kernel with Ξ³
Sigmoid kernel with Ξ³ and r
πΎπΎ(π₯π₯ππ , π₯π₯ππ) = π₯π₯πππππ₯π₯ππ
πΎπΎ(π₯π₯ππ , π₯π₯ππ) = πΎπΎπ₯π₯πππππ₯π₯ππ + r ππ
πΎπΎ π₯π₯ππ , π₯π₯ππ = exp(βπΎπΎ||π₯π₯ππ β π₯π₯ππ||2)
πΎπΎ π₯π₯ππ , π₯π₯ππ = tanh(πΎπΎπ₯π₯πππππ₯π₯ππ + r)
![Page 7: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/7.jpg)
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
7
LIBSVM[1] is an open source machine learning library developed at the National Taiwan University and written in C++ though with a C API. LIBSVM implements the SMO algorithm for kernelized support vector machines (SVMs), supporting classification and regression. [1] https://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html
Differences between LIBSVM and SVM function in Matlab
LIBSVM SVM function in Matlab
Classification or regression Both Classification
Classification type Two-class and multi-class Two-class
SVM formulations C-SVC, nu-SVC, one-class SVM, epsilon-SVR, nu-SVR C-SVC
![Page 8: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/8.jpg)
LIBSVM -- A LIBRARY FOR SUPPORT VECTOR
8
PROCEDURE 1. Transform data to the format of an SVM package
2. Conduct simple scaling on the data
3. Consider the RBF kernel K(π₯π₯ππ,π₯π₯ππ) = exp(βΞ³| π₯π₯ππ β π₯π₯ππ |2)
4. Find the best parameter C and Ξ³
5. Use the best parameter C and Ξ³ to train the whole training data
6. Test
![Page 9: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/9.jpg)
TRAINING RESULT
9
GAMMA=0.1
![Page 10: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/10.jpg)
TRAINING RESULT
10
GAMMA=3
![Page 11: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/11.jpg)
TRAINING RESULT
11
GAMMA=50
![Page 12: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/12.jpg)
TESTING RESULT
12
GAMMA=3
![Page 13: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/13.jpg)
TESTING RESULT
13
GAMMA=3
![Page 14: SUPPORT VECTOR MACHINE IN MATLAB...LIBSVM -- A LIBRARY FOR SUPPORT VECTOR. 7 . LIBSVM [1] is an open source machine learning library developed at the National Taiwan University and](https://reader035.fdocuments.us/reader035/viewer/2022081617/6053f2ee3cd3b54b2f54a2d1/html5/thumbnails/14.jpg)
CONCLUSION
14
SVM is a new method of machine learning based on statistics theory. In contrast to βblack boxβ learning approaches (artificial neural network), SVM is supported by certain mathematical models.
The training of SVM is relatively easy. Unlike in neural network, SVM can get global optimum and the training time does not depend on dimensionality of feature space any more by using the kernel trick.
The key to training SVM is to select a kernel function and its parameters, which cannot be conducted by a principled manner.