Sample Classification Knearest

7/29/2019 Sample Classification Knearest

1/21

Introduction The Basic K-NN Extensions Prototype Selection Summary

5. K-NEAREST NEIGHBOR

Pedro Larranaga

Intelligent Systems GroupDepartment of Computer Science and Artificial Intelligence

University of the Basque Country

Madrid, 25th of July, 2006
http://find/http://goback/


2/21


Outline

1 Introduction

2 The Basic K-NN

3 Extensions of the Basic K-NN

4 Prototype Selection

5 Summary
http://find/


3/21


Outline

1 Introduction

2 The Basic K-NN



5 Summary


4/21


Basic Ideas

K-NN IBL, CBR, lazy learning

A new instance is classified as the most frequent class of

its K nearest neighbors

Very simple and intuitive idea

Easy to implement

There is not an explicit model (transduction)

K-NN instance based learning (IBL), case based

reasoning (CBR), lazy learning
http://find/


5/21


Outline

1 Introduction

2 The Basic K-NN



5 Summary

S S


6/21


Algorithm for the basic K-NN

BEGIN

Input: D= {(x1 , c1), . . . , (xN, cN)}

x = (x1 , . . . , xn) new instance to be classified

FOR each labelled instance(xi

,

ci)calculate

d(xi,

x)

Order d(xi, x) from lowest to highest, (i= 1, . . . , N)

Select the K nearest instances to x: DKx

Assign to x the most frequent class in DKx

END

Figure: Pseudo-code for the basic K-NN classifier

I t d ti Th B i K NN E t i P t t S l ti S
http://find/


7/21



Example

Figure: Example for the basic 3-NN. mdenotes the number of

classes, n the number of predictor variables, and N the number

of labelled cases

Introduction The Basic K NN Extensions Prototype Selection Summary
http://find/


8/21



The accuracy is not monotonic with respect to K

Figure: Accuracy versus number of neighbors

http://find/


9/21


Outline

1 Introduction

2 The Basic K-NN



5 Summary

Introduction The Basic K -NN Extensions Prototype Selection Summary
http://find/


10/21


K-NN with rejection

Requiring for some guarantees

Demanding for some guarantees before an instance isclassified

In case that the guarantees are not verified the instance

remains unclassified

Usual guaranty: threshold for the most frequent class in the

neighbor

http://find/


11/21


K-NN with average distance

Figure: K-NN with average distance



12/21


K-NN with weighted neighbors

Figure: K-NN with weighted neighbors

http://find/


13/21

yp y

K-NN with weighted neighbors

d(xi, x) wix1 2 0.5

x2 2 0.

5x3 2 0.5

x4 2 0.5

x5 0.7 1/0.7x6 0.8 1/0.8

Figure: Weight to be assigned to each of the 6 selected

instances

http://find/


14/21

yp y

K-NN with weighted variables

X1 X2 C

0 0 1

0 0 1

0 0 1

1 0 11 0 1

1 1 1

0 1 0

0 1 0

0 1 01 1 0

1 1 0

1 0 0

Figure: Variable X1 is not relevant for C

http://find/


15/21

K-NN with weighted variables

d(xl, x) =n

X

i=1

widi(xl,i, xi) with wi = MI(Xi,C)

MI(X1,C) = p(X1,C)(0, 0) log

p(X1,C)(0, 0)

pX1(0) pC(0)

+ p(X1,C)(0, 1) log

p(X1,C)(0, 1)

pX1(0) pC(1)

+

p(X1,C)(1, 0) log

p(X1,C)(1, 0)

pX1(1) pC(0)

+ p(X1,C)(1, 1) log

p(X1,C)(1, 1)

pX1(1) pC(1)

=

3

12log

312

612

612

+3

12log

312

612

612

+3

12log

312

612

612

+3

12log

312

612

612

= 0

MI(X2,

C) = p(X2,C)(0

,

0) log

p(X2,C)(0, 0)

pX2(0)

pC(0) + p(X2

,C)(0,

1) log

p(X2,C)(0, 1)

pX2(0)

pC(1) +

p(X2,C)(1, 0) log

p(X2,C)(1, 0)

pX2(1) pC(0)

+ p(X2,C)(1, 1) log

p(X2,C)(1, 1)

pX2(1) pC(1)

=

1

12log

112

612

612

+5

12log

512

612

612

+5

12log

512

612

612

+1

12log

112

612

612

http://find/


16/21

Outline

1 Introduction

2 The Basic K-NN



5 Summary

http://find/


17/21

Wilson edition

Eliminating rare instances

The class of each labelled instance, (xl, c(l)), is comparedwith the label assigned by a K-NN obtained with all

instances except itself

If both labels coincide the instance in maintained in the file.

Otherwise it is eliminated

http://find/


18/21

Hart condensation

Maintaining rare instances

For each labelled instance, and following the storage

ordering, consider a K-NN with only the previous instancesto the one to be considered

If the true class and the class predicted by the K-NN are

the same the instance is not selected

Otherwise (the true class and the predicted one are

different) the instance is selected

The method depends on the storage ordering

http://find/


19/21

Outline

1 Introduction

2 The Basic K-NN



5 Summary

http://find/


20/21

K-nearest neighbor

Intuitive and easy to understand

There is not an explicit model: transduction instead of

induction

Variants of the basic algorithm

Storage problems: prototype selection

http://goforward/http://find/http://goback/


21/21

5. K-NEAREST NEIGHBOR

Pedro Larranaga

Intelligent Systems GroupDepartment of Computer Science and Artificial Intelligence

University of the Basque Country

Madrid, 25th of July, 2006
http://find/

Sample Classification Knearest

Documents

Transcript of Sample Classification Knearest