© Mori/Möller

(Kernels +)Support Vector Machines

Machine LearningTorsten Möller

© Mori/Möller

Reading

• Chapter 5 of “Machine Learning—An Algorithmic Perspective” by MarslandChapter 6+7 of “Pattern Recognition

and Machine Learning” by Bishop• Chapter 12 of “The Elements of

Statistical Learning” by Hastie, Tibshirani, Friedman

2

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

3

© Mori/Möller

Generalized linear model

• This is called a generalized linear model• f (·) is a fixed non-linear function, e.g.

• Decision boundary between classes will be linear function of x

• Can also apply non-linearity to x4

y(x) = f(wTx+ w0)

f(u) =

⇢1 if u � 0

0 otherwise

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

−1 −0.5 0 0.5 1−1

−0.5

0

0.5

1

© Mori/Möller

Perceptron learning illustration

5

© Mori/Möller

Limitations of Perceptrons

• Perceptrons can only solve linearly separable problems in feature space (same as the other models in this chapter)

• Canonical example of non-separable problem is X-OR (real datasets can look like this too)

6

Expressiveness of perceptrons

Consider a perceptron with g = step function (Rosenblatt, 1957, 1960)

Can represent AND, OR, NOT, majority, etc.

Represents a linear separator in input space:

!jWjxj > 0 or W · x > 0

I 1

I 2

I 1

I 2

I 1

I 2

?

(a) (b) (c)

0 1

0

1

0

1 1

0

0 1 0 1

xor I 2I 1orI 1 I 2and I 1 I 2

Chapter 20 11

© Mori/Möller

Non-linear decision boundaries

• it’s not linear in x anymore• separation may be easier in higher dim

7

y(x) = f(wT�(x) + b)

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

8

© Mori/Möller

Non-linear mappings• Last week, for logistic regression (classification),

we looked at models with wTφ(x)• The feature space φ(x) could be high-dimensional• This is good because if data aren’t separable in

original input space (x), they may be in feature space φ(x)

9

© Mori/Möller

Non-linear mappings

• We’d like to avoid computing high-dimensional φ(x)

• We’d like to work with x which doesn’t have a natural vector-space representation– e.g. graphs, sets, strings

10

© Mori/Möller

Kernel trick• Before, we would explicitly compute φ(xi) for each

datapoint– Run algorithm in feature space

• For some feature spaces, can compute dot product φ(xi)Tφ(xj) efficiently

• Efficient method is computation of a kernel function k(xi, xj) = φ(xi)Tφ(xj)

• The kernel trick is to rewrite an algorithm to only have x enter in the form of dot products

• The menu:– Kernel trick examples– Kernel functions

11

||�(xi)� �(xj)||2 = �(xi)T�(xi)� 2�(xi)

T�(xj) + �(xj)

T�(xj)

= k(xi,xi)� 2k(xi,xj) + k(xj ,xj)

© Mori/Möller

A kernel trick• Let’s look at the nearest-neighbour classification

algorithm• For input point xi, find point xj with smallest

distance:

• If we used a non-linear feature space φ(·)

• So nearest-neighbour can be done in a high-dimensional feature space without actually moving to it

12

||xi � xj ||2 = (xi � xj)T (xi � xj)

= xiTxi � 2xi

Txj + xj

Txj

k(x, z) = (1 + x

Tz)2

© Mori/Möller

A Kernel Function• Consider the kernel function• we find (x & z be 2D vectors)

• So this particular kernel function does correspond to a dot product in a feature space (is valid)

• Computing k(x, z) is faster than explicitly computing φ(x)Tφ(z)– In higher dimensions, larger exponent, much faster

13

k(x, z) = (1 + x1z1 + x2z2)2

= 1 + 2x1z1 + 2x2z2 + x

21z

21 + 2x1z1x2z2 + x

22z

22

= (1,p2x1,

p2x2, x

21,p2x1x2, x

22)(1,

p2z1,

p2z2, z

21 ,p2z1z2, z

22)

T

= �(x)T�(z)

© Mori/Möller

Why kernels?• Why bother with kernels?

– Often easier to specify how similar two things are (dot product) than to construct explicit feature space φ.

– There are high-dimensional (even infinite) spaces that have efficient-to-compute kernels

• Separability

• So you want to use kernels– Need to know when kernel function is valid,

so we can apply the kernel trick14

© Mori/Möller

Valid kernels• Given some arbitrary function k(xi, xj), how do we

know if it corresponds to a dot product in some space?

• Valid kernels: if k(·, ·) satisfies:– Symmetric;k(xi,xj)=k(xj,xi)– Positive definite; for any x1,...,xN, the Gram matrix K

must be positive semi-definite:• Positive semi-definite means xTKx≥0 for all x

then k(·, ·) corresponds to a dot product in some space φ

• a.k.a. Mercer kernel, admissible kernel, reproducing kernel

15

© Mori/Möller

Examples of kernels

• Linear kernel k(x1,x2)=x1Tx2

– φ(x)=x• Polynomial kernel k(x1,x2) = (1+x1Tx2)d

– Contains all polynomial terms up to degree d• Gaussian (radial) kernel

k(x1,x2) =exp(−||x1−x2||2/2σ2)– Infinite dimension feature space

16

© Mori/Möller

Constructing kernels

• Can build new valid kernels from existing valid ones:– k(x1,x2) = c k1(x1,x2), c>0– k(x1,x2) = k1(x1,x2) + k2(x1,x2)– k(x1,x2) = k1(x1,x2) k2(x1,x2)– k(x1,x2) = exp(k1(x1,x2))

• Table on p. 296 gives many such rules

17

© Mori/Möller

More kernels• Stationary kernels are only a function of the difference

between arguments: k(x1, x2) = k(x1 − x2)– Translation invariant in input space: k(x1, x2) =k(x1 +c,x2 +c)

• Homogeneous kernels, a.k.a. radial basis functions only a function of magnitude of difference: k(x1, x2) = k(||x1 − x2||)

• Set subsets k(A1, A2) = 2|A1∩A2|, where |A| denotes number of elements in A

• Domain-specific: think hard about your problem, figure out what it means to be similar, define as k(·, ·), prove positive definite (Feynman algorithm)

18

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

19

© Mori/Möller

Non-linear decision boundaries

• consider two-class classification• let’s assume

– moved training data into high-dimensional space feature space

– data are (indeed) linearly separable in that space

• could now apply (simple, linear) classifier, BUT ...

20

y(x) = f(wT�(x) + b)

© Mori/Möller

... there are many decision boundaries!

• which one to pick?

21

© Mori/Möller

Maximum margin

• We can define the margin of a classifier as the minimum distance to any example

• In support vector machines the decision boundary which maximizes the margin is chosen

22

y = 1y = 0

y = �1

margin

© Mori/Möller

Marginal geometry

• Recall from lecture 3, for• projection of x in w dir. is • y(x) = 0 when wTx = -b, or • So is

signed distance todecision boundary

23

y(x) = w

Tx+ b

w

Tx

||w||

w

Tx

||w|| =�b

||w||w

Tx

||w|| ��b

||w|| =y(x)

||w||x2

x1

w

x

y(x)kwk

x?

�w0kwk

y = 0y < 0

y > 0

R2

R1

tny(xn)

||w||

© Mori/Möller

Support Vectors• Assuming data are

separated by the hyperplane, distance to decision boundary is

• The maximum margin criterion chooses w, b by:

• Points with this min value are known as support vectors

24

argmax

w,b

⇢1

||w|| min

n[tn(w

T�(xn) + b)]

�

y = 1

y = 0

y = �1

© Mori/Möller

Canonical representation• This optimization problem is complex:

• Note that rescaling w → κw and b → κb does not change distance tny(xn)/||w|| (many equiv. answers)

• So for x∗ closest to surface, can set:

• All other points are at least this far away:

25

argmax

w,b

⇢1

||w|| min

n[tn(w

T�(xn) + b)]

�

t⇤(wT�(x⇤) + b) = 1

8n , tn(wT�(xn) + b) � 1

© Mori/Möller

Canonical representation• This optimization problem is complex:

• Under these constraints, the optimization becomes:

• Can be formulated as a constrained optimization problem:

26

argmax

w,b

⇢1

||w|| min

n[tn(w

T�(xn) + b)]

�

argmax

w,b

1

||w|| = argmin

w,b

1

2

||w||2

© Mori/Möller

Canonical representation

• So the optimization problem is now a constrained optimization problem:

• To solve this, we need to take a detour into Lagrange multipliers

27

argminw,b

1

2||w||2

s.t. , 8n , tn(wT�(xn) + b) � 1

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

28

© Mori/Möller

Lagrange Multipliers• Consider the problem:

• Points on g(x) = 0 must have ∇g(x) normal to surface

• A stationary point must have no change in f in the direction of the surface, so ∇f(x) must also be in this same direction

• So there must be some λ s. t. ∇f(x) + λ∇g(x) = 029

max

x

f(x)

s.t. , g(x) = 0

rf(x)

rg(x)

xA

g(x) = 0

© Mori/Möller

Lagrange Multipliers• Consider the problem:

• So there must be some λ s. t. ∇f(x) + λ∇g(x) = 0• Define Lagrangian:

• Stationary points of L(x,λ) have• ∇xL(x, λ) = ∇f(x) + λ∇g(x) = 0 and ∇λL(x, λ) =g(x) =0• So are stationary points of constrained problem!

30

max

x

f(x)

s.t. , g(x) = 0

rf(x)

rg(x)

xA

g(x) = 0

L(x,�) = f(x) + �g(x)

• Consider the problem

• Lagrangian:

• Stationary points require:

• So stationary point is© Mori/Möller

Lagrange Multipliers Example

31

max

x

f(x1, x2) = 1� x

21 � x

22

s.t. g(x1, x2) = x1 + x2 � 1 = 0

L(x,�) = 1� x

21 � x

22 + �(x1 + x2 � 1)

@L/@x1 = �2x1 + � = 0

@L/@x2 = �2x2 + � = 0

@L/@� = x1 + x2 � 1 = 0

(x⇤1, x

⇤2) = (

1

2,

1

2), � = 1

g(x1, x2) = 0

x1

x2

(x?1, x

?2)

© Mori/Möller

Lagrange Multipliers - Inequality Constraints• Consider the problem:

• Optimization over a region – solutions either at stationary points (gradients 0) in region or on boundary

L(x, λ) = f(x) + λg(x)• Solutions have either:

– ∇f(x)=0 and λ=0 (in region),or– ∇f(x)=−λ∇g(x) and λ>0 (on boundary, > for maximizing f).– For both, λg(x) = 0

• Solutions have g(x) ≥ 0,λ ≥ 0,λg(x) = 032

max

x

f(x)

s.t., g(x) � 0

rf(x)

rg(x)

xA

xB

g(x) = 0g(x) > 0

© Mori/Möller

Lagrange Multipliers - Inequality Constraints• Consider the problem:

• Exactly how does the Lagrangian relate to the optimization problem in this case?

L(x, λ) = f(x) + λg(x)• It turns out that the solution to optimization problem is:

33

max

x

f(x)

s.t., g(x) � 0

rf(x)

rg(x)

xA

xB

g(x) = 0g(x) > 0

max

x

min

��0L(x,�)

© Mori/Möller

Max-min• Lagrangian

L(x, λ) = f(x) + λg(x)• Consider the following:• If constraint g(x) ≥ 0 is not satisfied, g(x) < 0

– λ can be made ∞, and minλ≥0 L(x,λ) = −∞• Otherwise, minλ≥0 L(x,λ) = f(x), (with λ=0)

34

min��0

L(x,�)

min

��0L(x,�) =

⇢�1 constraint not satisfied

f(x) otherwise

© Mori/Möller

Min-max (Dual form)

• So the solution to optimization problem is (called the primal problem):

• The dual problem is when one switches the order of the max and min:

35

LP (x) = max

x

min

��0L(x,�)

LD(�) = min

��0max

x

L(x,�)

© Mori/Möller

Min-max (Dual form)

• These are not the same, but it is always the case the dual is a bound for the primal (in the SVM case with minimization, LD(λ) ≤ LP(x))

• Slater’s theorem gives conditions for these two problems to be equivalent, with LD(λ) = LP(x).

• Slater’s theorem applies for the SVM optimization problem, and solving the dual leads to kernelization and can be easier than solving the primal

36

LP (x) = max

x

min

��0L(x,�) LD(�) = min

��0max

x

L(x,�)

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

37

© Mori/Möller

Now where were we?

• So the optimization problem is now a constrained optimization problem:

• For this problem, the Lagrangian (with N multipliers an) is:

38

argminw,b

1

2||w||2

s.t. , 8n , tn(wT�(xn) + b) � 1

L(w, b,a) =||w||2

2�

NX

n=1

an�tn(w

T�(xn) + b)� 1

© Mori/Möller

Now where were we?

• The Lagrangian (with N multipliers an) is:

• We can find the derivatives of L wrt w, b and set to 0:

39

L(w, b,a) =||w||2

2�

NX

n=1

an�tn(w

T�(xn) + b)� 1

w =NX

n=1

antn�(xn)

0 =NX

n=1

antn

© Mori/Möller

Dual form• Plugging those equations into L removes w and b

and results in a version of L where ∇w,bL = 0:

• this new L is the dual representation of the problem (maximize with constraints)– Note that it is kernelized– It is quadratic, convex in a– Bounded above since K positive semi-definite– Optimal a can be found– With large datasets, descent strategies employed

40

L(a) =NX

n=1

an � 1

2

NX

n=1

NX

m=1

anamtntm�(xn)T�(xm)

© Mori/Möller

Examples

• SVM trained using Gaussian kernel• Support vectors circled• Note non-linear decision boundary in x

space

41

21

4.3. Some Examples of Nonlinear SVMs

The first kernels investigated for the pattern recognition problem were the following:

K(x,y) = (x · y + 1)p (74)

K(x,y) = e!"x!y"2/2!2(75)

K(x,y) = tanh(!x · y ! ") (76)

Eq. (74) results in a classifier that is a polynomial of degree p in the data; Eq. (75) givesa Gaussian radial basis function classifier, and Eq. (76) gives a particular kind of two-layersigmoidal neural network. For the RBF case, the number of centers (NS in Eq. (61)),the centers themselves (the si), the weights (#i), and the threshold (b) are all producedautomatically by the SVM training and give excellent results compared to classical RBFs,for the case of Gaussian RBFs (Scholkopf et al, 1997). For the neural network case, thefirst layer consists of NS sets of weights, each set consisting of dL (the dimension of thedata) weights, and the second layer consists of NS weights (the #i), so that an evaluationsimply requires taking a weighted sum of sigmoids, themselves evaluated on dot products ofthe test data with the support vectors. Thus for the neural network case, the architecture(number of weights) is determined by SVM training.

Note, however, that the hyperbolic tangent kernel only satisfies Mercer’s condition forcertain values of the parameters ! and " (and of the data "x"2). This was first noticedexperimentally (Vapnik, 1995); however some necessary conditions on these parameters forpositivity are now known14.

Figure 9 shows results for the same pattern recognition problem as that shown in Figure7, but where the kernel was chosen to be a cubic polynomial. Notice that, even thoughthe number of degrees of freedom is higher, for the linearly separable case (left panel), thesolution is roughly linear, indicating that the capacity is being controlled; and that thelinearly non-separable case (right panel) has become separable.

Figure 9. Degree 3 polynomial kernel. The background colour shows the shape of the decision surface.

Finally, note that although the SVM classifiers described above are binary classifiers, theyare easily combined to handle the multiclass case. A simple, e!ective combination trains

© Mori/Möller

Examples• From Burges, A Tutorial on Support Vector Machines for

Pattern Recognition (1998)• SVM trained using cubic polynomial kernel

• Left is linearly separable– Note decision boundary is almost linear, even using cubic

polynomial kernel• Right is not linearly separable

– But is separable using polynomial kernel

42

k(x1,x2) = (xT1 x2 + 1)3

21

4.3. Some Examples of Nonlinear SVMs

The first kernels investigated for the pattern recognition problem were the following:

K(x,y) = (x · y + 1)p (74)

K(x,y) = e!"x!y"2/2!2(75)

K(x,y) = tanh(!x · y ! ") (76)

Eq. (74) results in a classifier that is a polynomial of degree p in the data; Eq. (75) givesa Gaussian radial basis function classifier, and Eq. (76) gives a particular kind of two-layersigmoidal neural network. For the RBF case, the number of centers (NS in Eq. (61)),the centers themselves (the si), the weights (#i), and the threshold (b) are all producedautomatically by the SVM training and give excellent results compared to classical RBFs,for the case of Gaussian RBFs (Scholkopf et al, 1997). For the neural network case, thefirst layer consists of NS sets of weights, each set consisting of dL (the dimension of thedata) weights, and the second layer consists of NS weights (the #i), so that an evaluationsimply requires taking a weighted sum of sigmoids, themselves evaluated on dot products ofthe test data with the support vectors. Thus for the neural network case, the architecture(number of weights) is determined by SVM training.

Note, however, that the hyperbolic tangent kernel only satisfies Mercer’s condition forcertain values of the parameters ! and " (and of the data "x"2). This was first noticedexperimentally (Vapnik, 1995); however some necessary conditions on these parameters forpositivity are now known14.

Figure 9 shows results for the same pattern recognition problem as that shown in Figure7, but where the kernel was chosen to be a cubic polynomial. Notice that, even thoughthe number of degrees of freedom is higher, for the linearly separable case (left panel), thesolution is roughly linear, indicating that the capacity is being controlled; and that thelinearly non-separable case (right panel) has become separable.

Figure 9. Degree 3 polynomial kernel. The background colour shows the shape of the decision surface.

Finally, note that although the SVM classifiers described above are binary classifiers, theyare easily combined to handle the multiclass case. A simple, e!ective combination trains

© Mori/Möller

Today

• Motivation• Kernels• Support Vectors

– the Idea!– Lagrange Multipliers– Min-max vs. Max-min– solving constraint problem– non-separable data

43

© Mori/Möller

Non-separable data (soft-margin classifier)

• For most problems, data will not be linearly separable (even in feature space φ)

• Can relax the constraints fromtny(xn) ≥ 1 to tny(xn) ≥ 1−ξn

• The ξn ≥ 0 are called slack variables– ξn = 0, satisfy original problem,

so xn is on margin or correct sideof margin

– 0 < ξn < 1, inside margin, but stillcorrectly classified

– ξn > 1, mis-classified44

y = 1

y = 0

y = �1

⇠ > 1

⇠ < 1

⇠ = 0

⇠ = 0

© Mori/Möller

Loss function for non-sep data• Non-zero slack variables are bad, penalize

while maximizing the margin:

• Constant C > 0 controls importance of large margin versus incorrect (non-zero slack)– Set using cross-validation

• Optimization is same quadratic, different constraints, convex

45

minCNX

n=1

⇠n +1

2||w||2

y = 1

y = 0

y = �1

⇠ > 1

⇠ < 1

⇠ = 0

⇠ = 0

© Mori/Möller

SVM Loss function

• The SVM for the separable case solved the problem:

• Can write this as:

• where E∞(z) = 0 if z ≥ 0, ∞ otherwise46

argminw

1

2||w||2

s.t. 8n , tnyn � 1

argminw

NX

n=1

E1(tnyn � 1) + �||w||2

© Mori/Möller

SVM Loss function

• The SVM for the separable case solved the problem:

• Non-separable case relaxes this to be:

• where ESV(tnyn − 1) = [1 − tnyn]+ hinge loss– [u]+ =u if u≥0, 0 otherwise

47

argminw

1

2||w||2

s.t. 8n , tnyn � 1

argminw

NX

n=1

ESV (tnyn � 1) + �||w||2

�2 �1 0 1 2z

E(z)

© Mori/Möller

Loss functions

• Linear classifiers, compare loss function used for learning– Black is misclassification

error– Simple linear classifier,

squared error: (yn − tn)2– Logistic regression,

cross-entropy error: tn ln yn

– SVM, hinge loss: ξn = [1 − tnyn]+

48

© Mori/Möller

Summary• Kernels

– high-dim spaces good for separation, bad for computation!

– many algorithms can be re-written with only dot products of features

– NN, perceptron, regression, PCA, SVMs• SVMs

– Maximum margin criterion for deciding on decision boundary

– Linearly separable data– Relax with slack variables for non-separable case– Global optimization is possible: Convex problem (no

local optima) 49