Seungchan Lee Intelligent Electronic Systems Human and Systems Engineering Department of Electrical...
-
Upload
rosalyn-lawrence -
Category
Documents
-
view
219 -
download
4
Transcript of Seungchan Lee Intelligent Electronic Systems Human and Systems Engineering Department of Electrical...
Seungchan LeeIntelligent Electronic Systems
Human and Systems EngineeringDepartment of Electrical and Computer Engineering
Software Release and Support Vector Machine
Research Presentation:
ISIP: Research Presentation Page 2 of 24
Overview
• Software Release Isip_lm_tester Isip_network_builder Debugging utility : Purify
• Verification System Isip_verify Support Vector Machine
• Audio File Generation
• Next Plans
ISIP: Research Presentation Page 3 of 24
Isip_lm_tester, Isip_network_builder
• Dummy Symbol generation problem Sentence generation terminated when met with dummy symbol
at the highest level. Dummy Symbol should not show at the output sentence.
Include Dummy Symbol check routine
• Exclude Symbol generation problem When turn on exclude symbol flag, lm_tester should not generate exclude
symbol. It need to modify HierarchicalSearch class.
Isip_network_buider Add save option for ABNF, BNF Need to correct dummy symbol generation problem
When generating dummy symbol without any subgraph, it generates error message.
ISIP: Research Presentation Page 4 of 24
Debugger Utility
• Purify What is problem?
Compilation error : When instrumenting purify, it generates error message. It is not easy to figure out the reason because we have so many linking
process when compiling.
How to resolve? Simple program without IFC class works fine Narrow down which classes are problem.
Exclude all linking process, and then add one class repeatedly.
Solution : After track down the linking process, I can figure out the problem is
originated from sphere utility. How can correct it?
I’m currently doing this.
ISIP: Research Presentation Page 5 of 24
Isip_verify
• When doing HMM training, it generates segmentation fault. This happens at the end of the program related to HierarchicalDigraph
object. Recently, we have many changes in IFC classes, but this problem might exists
sometimes ago.
• When doing SVM training, it generates checksum error. This error did not happen before I was recompiling whole repository.
isip_verify utilty also need to be throughly investigated using
purify utility
ISIP: Research Presentation Page 6 of 24
Audio File Generation
• Load one or two SWB CDs
• Select 100 conversations
• For each conversation, strip the NIST header
• Grab every other byte starting with the first byte (first channel) and put that into a raw audio file; PRESERVE the 8-bit ulaw data (do not use or convert to 16-bit)
• Convert this file to Sun ".au" using Sox
ISIP: Research Presentation Page 7 of 24
What to Learn?
• Audio File format .au file format
Widely used in UNIX machine and originated by SUN. Header + Variable length information + audio data Support various encoding types
NIST SPHERE file Raw format
PERL Programming Language It is simple programming language which performs extracting and printing out
information from a text file. Interpreted Language ( not compiled)
Conversion Utilities w_decode Sox “od” command
ISIP: Research Presentation Page 8 of 24
Why Support Vector Machine ?
• This is new learning technology to be noticed recently.
• Even though it has been situated as a subfield of machine learning, it still have many issues about theory and algorithm.
• To be more familiar with verification system, it is required to review one field for the next step.
ISIP: Research Presentation Page 9 of 24
How it works?
• Suppose we have low dimensional feature space. It is consist of positive examples and negative examples
0x
0x
• How about the following case?
• How can we classify this?
0x
ISIP: Research Presentation Page 10 of 24
How it works?
Simple idea : Low dimensional feature space map into high dimensional feature space using kernel function.
0x
2 ( , )k k k kx z x x
ISIP: Research Presentation Page 11 of 24
How can we determine maximum margin?
• To explain this, we need to know the following concepts.
Margin concepts
Lagrange multiplier
Primal and dual representation
Karush-Kuhn-Tucker Conditions (KKT)
Risk Bounds and Minimization
Maximal Margin classifier
ISIP: Research Presentation Page 12 of 24
Hyperplane
• Linear classification
function. thecontrolthat
parameters theare b)(w, , offunction linear a is )( where
)(
1
xx
xwx
f
bxw
bf
i
n
ii
0 bxw
• Input space X is split into two
parts by the hyperplane defined
by the equation
b
w
x xx
x
x
xx
Objective Function
ISIP: Research Presentation Page 13 of 24
Margin
x xx
x
x
xx
xxx
x
xx
x1
functional margin of an example (x,y) with respect to a hyperplane
.i i iy b
w
w x
Geometric margin of two points The margin of training set
ISIP: Research Presentation Page 14 of 24
Maximal Margin Classifier
• The Simplest model, but works only for data which are linearly separable in the feature space.
easy to understand and main building block for more complex SVMs
Margin
w
H1H2
• Plus-plane =
• Minus-plane =
• Separating hyperplane =
Classify as.. +1 if
-1 if
}1:{ bxwx}1:{ bxwx
1 bxw1 bxw
1 bxw0 bxw
1 bxw
{ : 0}b x w x
ISIP: Research Presentation Page 15 of 24
Maximal Margin Classifier
x
Margin
w
x
• Computing the margin width
2
1
2
1
w wx x
w w
w x w x
w
1 bxw0 bxw
1 bxw
ISIP: Research Presentation Page 16 of 24
Maximal Margin Classifier
2
2 M
w
x
Margin
w
x
• Computing the margin width
,...,l, i
,by
,
ii
1
1 subject to
minimize bw,
xw
ww
problemon optimizati thesolve that b)(w, hyperplane the
)),(),...,,((xS
sample trainingseparablelinearly aGiven
11 ll yxy
2w1/margin geometric with hyperplane maximal therealises
1 bxw0 bxw
1 bxw
ISIP: Research Presentation Page 17 of 24
Maximal Margin Classifier
x
Margin
w
x
• How to transform this optimization problem into dual problem?
l
iii
l
iiii
y
y
1
1
0
,
xw
, and respect to with atingdifferentiby found is dual ingcorrespond The
s.multiplier Lagange theare 0 where
12
1),,(
1
b
bybL
i
l
iii
w
xwwwαw
Hypothesis can be described as a linear combination of the training points.
1 bxw0 bxw
1 bxw
2
2 M
wLagrange
ISIP: Research Presentation Page 18 of 24
Maximal Margin Classifier
x
1 bxw
Margin
w
0 bxw1 bxw
x
• How to transform this optimization problem into dual problem?
.2
1
2
1
12
1),,(
1,1
11,1,
1
l
jijijiji
l
ii
l
ii
l
jijijiji
l
jijijiji
l
iii
yy
yyyy
bybL
xx
xxxx
xwwwαw
2
2 M
w
ISIP: Research Presentation Page 19 of 24
Maximal Margin Classifier
* *
1
*
2
Then the weight vector w realises the maximal margin hyperplane with
geometric margin 1/
l
i i ii
y
x
w
x
1 bxw
Margin
w
0 bxw1 bxw
x
• How to transform this optimization problem into dual problem?
*
1 , 1
Suppose the parameter solve the following quadratic optimization problem:
1maximize W( ) ,
2
l l
i i j i j i ji i j
y y
x x
1
subject to 0, 0, 1,..., .l
i i ii
y i l
ISIP: Research Presentation Page 20 of 24
Maximal Margin Classifier
x
1 bxw
Margin
w
0 bxw1 bxw
x
• How to transform this optimization problem into dual problem?
2
minmax *
1
*1
*iyiy
ii
bxwxw
* * * 1 0, 1,..., .i i iy b i l w x zero.-non
ingcorrespond
theare hyperplane
thecloset to lie
efore that therand one
ismargin functional
thefor which iput
:ctor Support ve
*
ix
Only these points are involved for the weight vector.*
SVi
*
*
1
***
),,(
by
bybf
ii
l
iii
xx
xxx
ISIP: Research Presentation Page 21 of 24
Maximal Margin Classifier
x
1 bxw
Margin
w
0 bxw1 bxw
x
• How to transform this optimization problem into dual problem?
zero.-non
ingcorrespond
theare hyperplane
thecloset to lie
efore that therand one
ismargin functional
thefor which iput
:ctor Support ve
*
ix
,1),,( ****
SViiiii byybfy xxx
SVj
***
SVj
*
SVj
*
1,
**
.)1(
SViiij
jiiijj
l
jijijiji
by
yy
yy
xx
xxww **
ISIP: Research Presentation Page 22 of 24
Maximal Margin Classifier
x
1 bxw
Margin
w
0 bxw1 bxw
x
• How to transform this optimization problem into dual problem?
zero.-non
ingcorrespond
theare hyperplane
thecloset to lie
efore that therand one
ismargin functional
thefor which iput
:ctor Support ve
*
ix
.1/
margin geometric with hyperplanemargin
maximal therealises tor weight vec
2/1*
2
1
*
SVii
l
iiiiy
w
xw
ISIP: Research Presentation Page 23 of 24
Review Maximal Margin SVMs
• Can be slow in practice
• Dose not control the number of support vector (Sparseness)
• Only one degree of freedom is the choice of kernel model selection
• Cannot be used non linear separable feature space
many real world problems deal with nonlinear, noisy data.
• However, it is a starting point for the more sophisticated SVMs.
ISIP: Research Presentation Page 24 of 24
Next Plan
• Software Release Resolve purify compilation problem Examine memory leak problem using purify utility Track down remaining bugs Test several cases
• Verification System Do NIST 2003 Experiment using new isip_verify Implemenation techniques of support vector machine Algorithm comparison between several SVM softwares Resolve memory leak problems
ISIP: Research Presentation Page 25 of 24
Reference
• An introduction to Support Vector Machines and other kernel-based learning methods by “Nello Cristianini and John Shawe-Taylor”, 2000, Cambridge Press
• Support Vector Machines Tutorial Slides by Andrew W. Moore
http://www.autonlab.org/tutorials/svm15.pdf
• Practical Perl Programming
http://www.cs.cf.ac.uk/Dave/PERL/