Anshu Paper2(1)

8/13/2019 Anshu Paper2(1)

http://slidepdf.com/reader/full/anshu-paper21 1/7

To Improve the Performance of Handwritten digit

Recognition using Support Vector Machine

Anshuman Sharma

([email protected])

Abstract: Handwritten Numeral recognition plays a vital role in postal automation services especially in

countries lie !ndia where multiple languages and scripts are used "iscrete Hidden #arov #odel (H##)

and hy$rid o% Neural Networ (NN) and H## are popular methods in handwritten word recognition

system. &he hy$rid system gives $etter recognition result due to $etter discrimination capa$ility o% the NN.

A ma'or pro$lem in handwriting recognition is the huge varia$ility and distortions o% patterns. lastic

models $ased on local o$servations and dynamic programming such H## are not e%%icient to a$sor$ this

varia$ility. ut their vision is local. ut they cannot %ace to length varia$ility and they are very sensitive todistortions. &hen the S*# is used to estimate glo$al correlations and classi%y the pattern. Support *ector

#achine (S*#) is an alternative to NN. !n Handwritten recognition+ S*# gives a $etter recognition result.&he aim o% this paper is to develop an approach which improve the e%%iciency o% handwritten recognition

using arti%icial neural networ Keyword Handwriting recognition+ Support *ector #achine+ Neural Networ

!" Introduction

Handwritten ,ecognition re%ers to the process o%

translating images o% hand-written+ typewritten+ or

printed digits into a %ormat understood $y user %or

the purpose o% editing+ indeing/searching+ and a

reduction in storage si0e. Handwritten recognitionsystem is having its own importance and it is

adopta$le in various %ields such as online

handwriting recognition on computer ta$lets+

recogni0e 0ip codes on mail %or postal mail

sorting+ processing $an chec amounts+ numeric

entries in %orms %illed up $y hand and so on. &here

are two distinct handwriting recognition domainsonline and o%%line+ which are di%%erentiated $y thenature o% their input signals. !n o%%line system+

static representation o% a digiti0ed document is

used in applications such as che2ue+ %orm+ mail or

document processing. 3n the other hand+ online

handwriting recognition (3H,) systems rely on

in%ormation ac2uired during the production o% the

handwriting. &hey re2uire speci%ic e2uipment that

allows the capture o% the tra'ectory o% the writing

tool. #o$ile communication systems such as

4ersonal "igital Assistant (4"A)+ electronic padand smart-phone have online handwriting

recognition inter%ace integrated in them.&here%ore+ it is important to %urther improve on

the recognition per%ormances %or these

applications while trying to constrain space %or

parameter storage and improving processing

speed. igure 1 shows an online handwritten

6ord recognition system. #any current systemsuse "iscrete Hidden #arov #odel $ased

recogni0er or a hy$rid o% Neural Networ (NN)

and Hidden #arov #odel (H##) %or the

recognition

3nline in%ormation captured $y the input device

%irst needs to go through some %iltration+

preprocessing and normali0ation processes. A%ter

1



normali0ation+ the writing is usually segmented

into $asic units (normally character or part o%

character) and each segment is classi%ied andla$eled. 7sing H## search algorithm in the

contet o% a language model+ the most liely word

path is then returned to the user as the intended

string 819. Segmentation process can $e per%ormed

in various ways. However+ o$servation

pro$a$ility %or each segment is normally o$tained

$y using a neural networ (NN) and a Hidden

#arov #odel (H##) estimates the pro$a$ilities

o% transitions within a resulting word path. &hisresearch aims to investigate the usage o% support

vector machines (S*#) in place o% NN in a

hy$rid S*#/H## recognition system. &he main

o$'ective is to %urther improve the recognition

rate8+;9 $y using support vector machine (S*#)

at the segment classi%ication level. &his is

motivated $y success%ul earlier wor $y

<anapathira'u in a hy$rid S*#/H## speech

recognition (S,) system and the wor $yahlmann 8=9 in 3H,. <anapathira'u o$tained

$etter recognition rate >ompared to hy$rid

NN/H## S, system. !n this wor+ S*# is %irst

developed and used to tra 0in an 3H, systemusing character data$ases. S*# with pro$a$ilistic

output are then developed %or use in the hy$rid

system. ventually+ the S*# will $e integrated

with the H## module %or word recognition.4reliminary results o% using S*# %or character

recognition are given and compared with results

using NN reported $y 4oisson 8?9. &he %ollowing

data$ases were used: !,3N3+ 7N!4N and

the miture !,3N3-7N!4N data$ases.

#" $%isting Techni&ues

#"! Modified discrimination function

'M()*+ ,-assifier<. S. ehal and Nivedan hatt 819 designed a

recognition system %or handwritten "evangari

Numeral using #odi%ied discrimination %unction

(#B") classi%ier. A recognition rate and a

con%usion rate were o$tained as =?C and D.5C

respectively.

#"# .eura- .etwor/ on )evenagari .umera-s

,. a'a'+ . "ey+ S. >haudhari 8119 used neuralnetwor $ased classi%ication scheme. Numerals

were represented $y %eature vectors o% three types.

&hree di%%erent neural classi%iers had $een used

%or classi%ication o% these numerals. inally+ theoutputs o%

the three classi%iers were com$ined using a

connectionist scheme. A E-layer #4 was used

%or implementing the classi%ier %or segment-$ased

%eatures. &heir wor produced recognition rate o%

=?.=C.

#"0 1aussian )istribution *unction

,. F. ,amtee et.al applied classi%iers on G

numerals images o$tained %rom di%%erent

individuals o% di%%erent pro%essions. &he results o%4>A+ correlation coe%%icient and pertur$ed

moments are an eperimental success as

compared to #!s. &his research produced ?G.G=C

recognition rate $y considering ;; %eature

dimensions.

#"2 *u33y c-assifier on Hindi .umera-s

#. Hanmandlu+ A.*. Nath+ A.>. #ishra and *..

#adasu used %u00y mem$ership %unction %orrecognition o% Handwritten Hindi Numerals and

produce ?C recognition rate. &o recogni0e theunnown numeral set+ an eponential variant o%

%u00y mem$ership %unction was selected and it

was constructed using the normali0ed vector

distance.

#"4 Mu-ti-ayer Perceptron

7''wal hattacharya+ . . >haudhuri 8119 used a

distinct #4 classi%ier. &hey wored on

"evanagari+ engali and nglish handwritten

numerals. A $ac propagation (4) algorithm was

used %or training the #4 classi%iers. !t provided

??.G;C and ??.DC recognition accuracies on the

original training and test sets o% "evanagarinumeral data$ase+ respectively.

#"5 (uadratic c-assifier for )evanagari

.umera-s

7. 4al+ &. 6aa$ayashi+ N. Sharma and .

imura 81D9 developed a modi%ied 2uadratic

classi%ier %or recognition o% o%%line handwritten

numerals o% si popular !ndian scripts vi0. &hey

had used D dimensional %eatures %or high-speed

recognition. A %ive-%old cross validation techni2ue

has $een used %or result computation and o$tained

??.5C accuracy %rom "evnagari scripts+respectively.

0" Proposed Approach

0"! Support Vector Machine 'SVM+

S*# in its $asic %orm implement two class

classi%ications. !t has $een used in recent years as

an alternative to popular methods such as neural

networ. &he advantage o% S*#+ is that it taes

into account $oth eperimental data and structural

$ehavior %or $etter generali0ation capa$ility $asedon the principle o% structural ris minimi0ation

(S,#). !ts %ormulation approimates S,#

principle $y maimi0ing the margin

o% class separation+ the reason %or it to $e nownalso as large margin classi%ier. &he $asic S*#

%ormulation is %or linearly separa$le datasets. !t

can $e used %or nonlinear datasets $y indirectly

mapping the nonlinear inputs into to linear %eature

space where the maimum #argin decision

%unction is approimated. &he mapping is done $yusing a ernel %unction. #ulti class classi%ication

can $e per%ormed $y modi%ying the G class

G



scheme. &he o$'ective o% recognition is to

interpret a se2uence o% numerals taen %rom the

test set. &he architecture o% proposed system isgiven in %ig. E.&he S*# ($inary classi%ier) is

applied to multi class numeral recognition

pro$lem $y using one-versus-rest type method.

&he S*# is trained with the training samples

using linear ernel. >lassi%ier per%orms its

%unction in two phases &raining and &esting. 8G?9

A%ter preprocessing and eature traction

process+ &raining is per%ormed $y considering the

%eature vectors which are stored in the %orm o%matrices. ,esult o% training is used %or testing the

numerals. Algorithm %or &raining is given in

algorithm.

0"# Statistica- 6earning Theory

Support *ector #achines have $een developed $y

*apni in the %ramewor o% Statistical earning

&heory 81E9. !n statistical learning theory (S&)+

the pro$lem o% classi%ication in supervisedlearning is %ormulated as %ollows:

6e are given a set o% l training data and its class+

{( x 1,y1)...( x l,yl)} in Rn × R sampled according to

unnown 'oint pro$a$ility distri$ution P( x ,y)characteri0ing how the classes are spread in Rn

× R. &o measure the per%ormance o% the classi%ier+

a loss %unction L(y+%(%)) is de%ined as %ollows:

L(y+%(%)) is 0ero i% f classi%ies correctly+ one

otherwise. 3n average+ how f per%orms can $e

descri$ed $y the ,is %unctional:

,# principle states that given the training set

and a set o% possi$le classi%iers in the hypothesis

space F + we Should choose f ⊂ F that minimi0es

Remp(f). However+ which generali0es well to

unseen data due to over %itting phenomena.

Remp(f) is a poor+ over-optimistic approimation

o% R(f)+ the true ris. Neural networ classi%ierrelies on ,# principle. &he normal practice to

get a more realistic estimate o% generali0ation

error+ as in neural networ is to divide the

availa$le data into training and test set. &raining

set is used to %ind a >lassi%ier with minimal

empirical error (optimi0e the weight o% an #4neural networs) while the test set is used to %ind

the generali0ation error (error rate on the &est set).

!% we have di%%erent sets o% classi%ier hypothesisspace F1, F2 … e.g. #4 neural networs with

di%%erent topologies+ we can select a classi%ier

%rom each hypothesis space (each topology) with

minimal Remp(f) and choose the %inal classi%ier

with minimal generali0ation error. However+ to do

that re2uires designing and training potentially

large num$er o% individual classi%iers. 7sing S&+

we do not need to do that. <enerali0ation errorcan $e directly minimi0ed $y minimi0ing an upper

$ound o% the ris %unctional R(f).

&he $ound given $elow holds %or any distri$ution

4(+y) with pro$a$ility o% at least 1- η :

where the parameter h denotes the so called *>(*apni->hervonenis) dimension. φ is the

con%idence term de%ined $y *apni 819 as :

,# is not su%%icient to %ind good classi%ier

$ecause even with small Remp(f), when h is large

compared to l + φ

will $e large+ so R(f) will also $e large+ ie: notoptimal. 6e actually need to minimi0e

Remp(f)and φ at the same time+ a process which is

called structural ris

#inimi0ation (S,#). y S,#+ we do not need

test set %or model selection anymore. &aing

di%%erent sets o% classi%iers F1, F2 … with nown

h1, h2 … we can select f

%rom one o% the set with minimal Remp(f)+

compute

and choose a classi%ier with minimal R(f).No

more evaluation on test set needed+ at least in

theory. However+ we still have to train potentially

very largenum$er o% individual classi%iers. &o avoid this+ we

want to mae h tuna$le (ie: to cascade a potential

classi%ier Fi with *> dimension I h and choose

an optimal f %rom an optimal Fi in a single

optimi0ation step. &his is done in large margin

classi%ication.

0"0 SVM formu-ations

S*# is reali0ed %rom the a$ove S& %ramewor.

&he simplest %ormulation o% S*# is linear+ where

the decision hyper plane lies in the space o% the

input data %.

E



!n this case the hypothesis space is a su$set o% all

hyper planes o% the %orm: f(x) = w⋅ x +b. S*#

%inds an optimal hyper plane as the solution to the

learning 4ro$lem which is geometrically the

%urthest %rom $oth classes since that will

generali0e $est %or %uture unseen data.

&here are two ways o% %inding the optimaldecision hyper plane. &he %irst is $y %inding a

plane that $isects the two closest points o% the twoconve hulls de%ined $y the set o% points o% each

class+ as shown in %igure G. &he second is $y

maimi0ing the margin $etween two supporting

planes as shown in %igure E.

oth methods will produce the same optimal

decision plane and the same set o% points that

support the solution (the closest points on the two

conve hulls in %igure G or the points on the two

parallel supporting planes in %igure E). &hese are

called the support vectors.

2" *eature $%traction

2"! Moment Invariants

&he moment invariants (#!s) 819 are used to

evaluate seven distri$uted parameters o% a

numeral image. !n any character ,ecognition

system+ the characters are processed to etract

%eatures that uni2uely represent properties o% the

character. ased on normali0ed central moments+

a set o% seven moment invariants is derived.

urther+ the resultant image was thinned andseven moments were etracted. &hus we had 1D

%eatures (; original and ; thinned)+ which are

applied as %eatures %or recognition using <aussian

"istri$ution unction. &o increase the success

rate+ the new %eatures need to $e etracted $y

applying A%%ine !nvariant #oment method.

2"# Affine Moment Invariants

&he A%%ine #oment !nvariants were derived $y

means o% the theory o% alge$raic invariants. ull

derivation and comprehensive discussion on the

properties o% invariants can $e %ound. our

%eatures can $e computed %or character

recognition. &hus overall 1= %eatures have $een

used %or Support *ector #achine.

4" $%periment

4"! )ata Set )escription

!n this paper+ the 7>! #achine learning data set

are used. &he 7>! #achine earning

,epository is a collection o% data$ases+ domain

theories+ and data generators that are used $y themachine learning community %or the empiricalanalysis o% machine learning algorithms. 3ne o%

the availa$le datasets is the 3ptical ,ecognition

o% the Handwritten "igits "ata Set. &he dataset o%

handwritten assamese characters $y collecting

samples %rom D5 writers is created. ach writer

contri$uted 5G $asic characters+ 1 numerals and

1G1 assamese con'unct consonants. &he total

num$er o% entries corresponding to each writer is

1=E (I 5G characters J 1 numerals J 1G1

con'unct consonants). &he total num$er o%samples in the dataset is =GE5 ( I D5 K 1=E ).

&he handwriting samples were collected on ani$all =7 eternal digiti0ing ta$let connected to

a laptop using its cordless digital stylus pen. &he

distri$ution o% the dataset consists o% D5 %olders.

&his %ile contains in%ormation a$out the character

id (!")+ character name (a$el) and actual shape

o% the character (>har).

!n the raw 3ptdigits data+ digits are represented as

EGEG matrices. &hey are also availa$le in a pre-

processed %orm in which digits have $een divided

into non-overlapping $locs o% DD and thenum$er o% on piels have $een counted in each

$loc. &his generated == input matrices whereeach element is an integer in the range .1.

D

http://archive.ics.uci.edu/ml/




http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits









*ig2 7 Sample digits etracted %rom the raw

3ptdigits dataset.

4"# )ata Preprocessing

or the eperiments using S*#+ eample isolated

characters are preprocessed and ; local %eatures

%or each point o% the spatially resample online

signal were etracted. or each eample character

there are E5 %eature values as input to the S*#.

6e use S*# with , ernel+ since , ernel

has $een shown to generally give $etterrecognition result 8;9. <rid search was done to

%ind the $est values %or the > and gamma

(representing in the original , ernel

%ormulation) %or the %inal S*# models and $y

that+ > I = and gamma I were chosen.

4"2 $%perimenta- Resu-ts

4"2"! Test application Analysis

&he test application accompanying the source

code can per%orm the recognition o% handwritten

digits. &o do so+ open the application (pre%era$ly

outside *isual Studio+ %or $etter per%ormance).

>lic on the menu ile and select 3pen. &his willload some entries %rom the 3ptdigits dataset into

the application.

Fig.5 !pt"igits "ata loa"e" into the

application

&o per%orm the analysis+ clic the ,un Analysis

$utton. 4lease $e aware that it may tae some

time. A%ter the analysis is complete+ the other ta$s

in the sample application will $e populated with

the analysisL in%ormation. &he level o% importanceo% each %actor %ound during the discriminant

analysis is plotted in a pie graph %or easy visual

inspection.

3nce the analysis is complete+ we can test its

classi%ication a$ility in the testing data set. &he

green rows have $een correctly identi%ied $y thediscriminant space uclidean distance classi%ier.

6e can see that it correctly identi%ies ?=C o% the

testing data. &he testing and training data set are

dis'oint and independent.

Fig.# $sing the "efa%lt &al%es in the

application

4"4 Resu-ts

A%ter the analysis has $een completed and

validated+ we can use it to classi%y the new digits

drawn directly in the application. &he $ars on the

right show the relative response %or each o% thediscriminant %unctions. ach class has a

discriminant %unction that outputs a closeness

measure %or the input point. &he classi%ication is

$ased on which %unction produces the maimumoutput.

Fig # 'e can see the analysis also pefos

athe well on copletely new an" pe&io%sly

%nseen "ata.

periments were per%ormed on di%%erent samples

having mied scripting languages on numerals

using single hidden layer.

5

http://www.codeproject.com/KB/recipes/handwriting-kda/kda3-7.png

http://www.codeproject.com/KB/recipes/handwriting-kda/kda3-5.png



Tab-e !: "etail ,ecognition per%ormance o% S*#

on 7>! datasets

Tab-e # "etail ,ecognition per%ormance o% S*#

and H## on 7>! datasets

Tab-e 0 ,ecognition ,ate o% ach Numeral in

"A&AS&.

!t is o$served that recognition rate using S*# is

higher than Hidden #arov #odel. However+ %ree

parameter storage %or S*# model is signi%icantly

higher. &he memory space re2uired %or S*# will

$e the num$er o% support vectors multiply $y the

num$er o% %eature values (in this case E5). &his

is signi%icantly large compared to H## whichonly need to store the weight. H## needs less

space due to the weight-sharing scheme.

However+ in S*#+ space saving can $e achieved

$y storing only the original online signals and the

penup/ pen-down status in a compact manner.

"uring recognition+ the model will $e epanded

dynamically as re2uired. &a$le E shows the

comparison o% recognition rates $etween H##and S*# using all three data$ases. S*# clearly

outper%orms in all three isolated character cases.&he result %or the isolated character cases a$ove

indicates that the recognition rate %or the hy$rid

word recogni0er could $e improved $y usingS*# instead o% H##. &hus+ we are currently

implementing word recogni0er using $oth H##

and S*# and comparing their per%ormance.

*ig 8 <raph ,epresentation $etween H## and

S*#

5" ,onc-usion

Handwriting recognition is a challenging %ield in

the machine learning and this wor identi%ies

Support *ector #achines as a potential solution.

&he num$er o% support vectors can $e reduced $y

selecting $etter > and gamma parameter values

through a %iner grid search and $y reduced set

selection 6or on integrating the S*# character

recognition %ramewor into the H## $ased wordrecognition %ramewor is on the way. !n the

hy$rid system+ word preprocessing and

normali0ation needs to $e done $e%ore S*# is

then used %or character hypothesis recognition andword lielihood computation using H##. !t is

envisaged that+ due to S*#Ms $etter

discrimination capa$ility+ word recognition rate

will $e $etter than in a H## hy$rid system.

R$*R$.,$S 7

819 Sandip aur+ ,ecognition o% Handwritten

"evanagri Script using eature ased on Oernie

#oments and Ooning and Neural Networ

>lassi%ierP+ A #. &ech. &hesis ,eport+ 4an'a$i7niversity+ 4atiala+ GD+ pp.

8G9 <aurav Fain+ Fason o+ Handwritten "igits

,ecognitionP+ #ultimedia Systems+ 4ro'ect,eport+ 7niversity o% &oronto+ Novem$er G1+

G=+ pp. 1-E.

8E9 Scott ". >onnell+ ,.#.. Sinha+ Ani1 . Fain

,ecognition o% 7nconstrained 3n-ine

"evanagari >haractersP+ G+ !.

8D9 A.. Fain+ ,o$ert 4.6."uin+ Fianchang #ao+

Statistical 4attern ,ecognition: A ,eviewP+ !&rans. 4A#!+ *ol.GG+ No. 1+ G.

859 Anu' Sharma+ 3nline Handwritten <urmuhi

>haracter ,ecognitionP+ A 4h. ". &hesis report+

School o% 89 Shu$hangi ".>.+ 4.S.Hiremath+

#ulti->lass S*# >lassi%ier %or nglish

Handwritten "igit ,ecognition using #anual>lass SegmentationP+ 4roc. !ntMl >on%. on

Advances in >omputing. >ommunication and>ontrol (!>A>EM?) G?+ pp. E5E-E5.

89 Sa$ri A. #ahmoud and Sameh #. Awaida+

,ecognition 3% 3%%-ine Handwritten Ara$ic

(!ndian) Numerals 7sing #ulti-Scale eatures

And Support *ector #achines *s. Hidden

#arov #odelsP &he Ara$ian Fournal or



Science And ngineering+ *olume ED+ Num$er

G$+ 3cto$er + G?+4p. DE-DDD.

8;9 A.or'i+ and #. Hamidi+ Support *ector#achine %or 4ersian ont ,ecognitionP+

!nternational Fournal o% !ntelligent Systems and

&echnologies+ Summer G;+ pp. 1=D-1=;

8=9 >. *asantha ashmi+ ,itu Fain+ >.

4atvardhan+ Handwritten "evanagari Numerals

,ecognition 6ith Higher AccuracyP+ 4roc. o%

! !nt. >on%. on

>omputational !ntelligence and #ultimidia

Application+ G;+ pp G55-G5?.8?9 7.hattacharya+ ..>haudhari+ Handwritten

Numeral "ata$ases o% !ndian Scripts and

#ultistage ,ecognition o% #ied NumeralsP+

! &rans. on 4A#!+ *ol.E1+ No.E+ G?+

pp.DDD-D5;.

819 7. 4al+ &. 6aa$ayashi+ N. Sharma and .

imura+ Handwritten Numeral ,ecognition o%

Si 4opular !ndian ScriptsP+ 4roc. ?th !>"A,+

>uriti$a+ ra0il+ *ol.G (G;)+;D?-;5E.8119 >hristopher #. ishop+ 4attern ,ecognition

and #achine earningP+ Springer 4u$lication+

Singapore+ G+ 4p. 1-E+ E=-EG.

;

Anshu Paper2(1)

Documents

Transcript of Anshu Paper2(1)