Quadratic Classifiers(QC)

Quadratic Classifiers(QC)

J.-S. Roger Jang J.-S. Roger Jang ((張智星張智星 ))

CS Dept., National Taiwan Univ.CS Dept., National Taiwan Univ.http://mirlab.org/janghttp://mirlab.org/jang

jang@mirlab.orgjang@mirlab.org

2010 Scientific Computing

2010 Scientific Computing

2

Bayes ClassifierBayes Classifier

Bayes classifier A probabilistic framework for classification problem

Conditional probability

Bayes theorem

112/04/18 2

)(/)()|(

)(/)()|(

iii

ii

CPCAPCAP

APACPACP

)(

)()|()|(

AP

CPCAPACP ii

i

2010 Scientific Computing

3 112/04/18 3

PDF ModelingPDF Modeling

• Goal:• Find a PDF (probability density function) that can

best describe a given dataset

• Steps:• Select a class of parameterized PDF• Identify the parameters via MLE (maximum

likelihood estimate) based on a given set of sample data

• Commonly used PDFs:• Multi-dimensional Gaussian PDF• Gaussian mixture models (GMM)

2010 Scientific Computing

4 112/04/18 4

PDF Modeling for ClassificationPDF Modeling for Classification

• Procedure for classification based on PDF• Training stage: PDF modeling of each class based

on the training dataset• Test stage: For each entry in the test dataset, pick

the class with the max. PDF

• Commonly used classifiers:• Quadratic classifier (n-dim. Gaussian PDF)• Gaussian-mixture-model classifier (GMM PDF)

2010 Scientific Computing

5 112/04/18 5

1D Gaussian PDF Modeling1D Gaussian PDF Modeling

• 1D Gaussian PDF:

• MLE of and •Detailed derivation

2

2

12

2

1),;(

x

exf

n

ii

n

ii

xn

xn

1

22

1

1

1

2010 Scientific Computing

6 112/04/18 6

1D Gaussian PDF Modeling via MLE1D Gaussian PDF Modeling via MLE

• MLE: Maximum Likelihood Estimate•Given a set of observations, find the parameters of the PDF such that the overall likelihood is maximized.

• Detailed derivation

2 2

2 21 2

1

1

22

1

; , ; ,

, ; , ; ,

1

1

i i i

n

n ii

n

ii

n

ii

x x p x g x

X x x x p X g x

xn

xn

2010 Scientific Computing

7 112/04/18 7

1D Gaussian PDF Modeling via MLE1D Gaussian PDF Modeling via MLE

Normal dist.estimated bynormal dist.

Uniform dist.estimated bynormal dist.

2010 Scientific Computing

8 112/04/18 8

d-dim. Gaussian PDF Modelingd-dim. Gaussian PDF Modeling

• d-dim. Gaussian PDF g(x, , )

• MLE of and :•Detailed derivation

)()(2

1

2/12/

1

)2(

1),;(

xx

d

T

exg

n

i

Tii

n

ii

xxn

xn

1

1

1

1

2010 Scientific Computing

9 112/04/18 9

d-dim. Gaussian PDF Modelingd-dim. Gaussian PDF Modeling

• d-dim. Gaussian PDF g(x, , )

• Likelihood of x in class j (governed by g(x, j, j))

)()(2

1

2/12/

1

)2(

1),;(

xx

d

T

exg

)()(2

1

2/12/

1

)2(

1),;(

jjT

j xx

jd

jj exg

2010 Scientific Computing

10 112/04/18 10

2D Gaussian PDF2D Gaussian PDF

Bivariate normal density:

Density function Contours

25.0

5.03,

0

0

2010 Scientific Computing

11

2D Gaussian PDF Modeling2D Gaussian PDF Modeling

gaussianMle.m

112/04/18 11

-2 -1 0 1 2

-1

0

1

-20

2

-1.5

-1

-0.5

0

0.5

1

1.5

0.1

0.2

0.3

0.4

2010 Scientific Computing

12

Steps of QCSteps of QC

Training stage• Select a type of Gaussian PDF• Identify the PDF of each class

Test stage• Assign each sample to the class with the highest

PDF value

112/04/18 12

2010 Scientific Computing

13

Characteristics of QCCharacteristics of QC

If each class is modeled by an Gaussian PDF, the decision boundary between any two classes is a quadratic function.• That is why it is called quadratic classifier.• How to prove it?

Different selections of the covariance matrix:• Constant times an identity matrix• Diagonal matrix• Full matrix (hard to use if the input dimension is

large)

112/04/18 13

2010 Scientific Computing

14

QC Results on Iris Dataset (I)QC Results on Iris Dataset (I)

Dataset: IRIS dataset with the last two inputs

112/04/18 14

2010 Scientific Computing

15

QC Results on Iris Dataset(II)QC Results on Iris Dataset(II)

PDF for each class:

112/04/18 15

2040

605

10152025

2

4

PDF of class 1

2040

605

10152025

0.20.40.60.8

11.2

PDF of class 2

2040

605

10152025

0.10.20.30.40.5

PDF of class 3

20 40 60

5

10

15

20

25

20 40 60

5

10

15

20

25

20 40 60

5

10

15

20

25

2010 Scientific Computing

16

QC Results on Iris Dataset (III)QC Results on Iris Dataset (III)

Decision boundaries among classes:

112/04/18 16

10 20 30 40 50 60

5

10

15

20

253 error points denoted by "x".

sepal length

sepa

l wid

th

44