Segmentation and Recognition using Artificial Neural Networks

DR. RENU VIG Department of Computer Science

Technical Teachers' Training Institute Sector 26, Chandigarh - 160019

INDIA

TANGKESHWAR THOKCHOM Department of Computer Science

Manipur University Canchipur, Imphal -795 003

INDIA

Abstract:- The following is a proposal for separating cursive characters into separate and distinctive characters for recognition. The proposed method begins with thresholding of gray level image into binarised image using iterative thresholding selection method. The binary image is then slant corrected. Using Neuro-heuristic technique, the slant corrected word is over segmented and artificial neural network trained with segmentation points is used to verify the segmentation points found and each segmented character is further extracted using character matrix extraction module. At last, using neocognitron simulator, the ANN is trained with the extracted character patterns and then, input pattern file containing a set of test patterns are tested for recognition on the basis of similarity in shape between patterns, but with a little effect from deformation, changes in size or shifts in position. Key-Words:- Thresholding, Binarisation, Slant Correction, Neuro-heuristic Segmentation, Character matrix extraction, Artificial Neural Networks, Pattern Recognition.

1 Introduction

Many scientists are trying to construct electronic texts so as to share materials on networks.Given the importance of written language in human transactions its automatic recognition has practical significance[1].However,evidence suggests that computer handwriting recognition will never be perfect[2].The primary difficulty of handwritten character recognition is said to come from shape variations due to writers’ habits, styles, times and so on. Thus, finding appropriate heuristic features and fast algorithms based on these features for segmenting the characters and neural based segmentation techniques for validating segmentation points are being investigated [3-7].In the proposed system, for the task of segmentation, a simple heuristic segmentation algorithm is used which finds segmentation points in cursive handwritten words. A neural network trained with

valid segmentation words is used to assess the correctness of the segmentation points. Following segmentation, character matrices are extracted from the words and recognition of characters is tested using a trained neocognitron (ANN) simulator.Neocognitron simulator is also tested for Panjabi and Manipuri scripts.Meetei Mayek(Manipuri script) experts say the script dates back as early as 3900 years ago[8].The organisation of the remainder of the paper is as follows: section 2 discusses the proposed techniques, section 3 details the experimental results, section 4 discusses the performance analysis of the results and finally a conclusion is drawn in section 5.

2 Proposed Techniques The technique used is as follows: 1. Scanning, 2. Binarisation, 3. Slant correction of word image, 4.

Segmentation using a Neuro-heuristic Technique and 5. Character matrix extraction and Normalisation of segmented characters 6.Recognition of characters using a trained neocognitron ANN. The entire system is shown in Fig.1. Fig.1. Segmentation and Recognition System 2.1 Scanning An IBM compatible Personal Computer running Windows 95 and a scanner were utilized to capture the sample-handwritten words. HP-Deskscan software package is used for the scanning of the necessary data. The images captured are monochrome (black and white) images. This is required for the images to be binarised in further steps. 2.2 Binarisation Windows bitmap was converted into a binary representation of the handwriting. Binarisation is

Fig.3 Thresholded Image performed using the Iterative thresholding selection method[9],which is implemented in Matlab. The algorithm for optimum thresholding (binarisation) using iterative thresholding selection method is given below.Fig.2 and Fig.3 show the original and thresholded images. Iterative thresholding selection Algorithm is shown below: Step 1. Select an initial estimate of the threshold, T. A good initial value is the mean intensity. The corners of the image are taken as initial values for the background Step 2. Partition the image into two groups, R 1 and R 2, using the threshold, T. Step 3. Calculate the mean intensity values µ1 and µ2 of the partitions R 1 and R2. Step 4. Select a new threshold: T = (µ1 + µ2)/ 2 Step 5. Repeat steps 2-4 until the mean values µ1 and µ2 do not change in successive iterations. 2.3 Slant Estimation and Correction For each word, all horizontal runs of foreground pixels that are greater than some predetermined threshold are detected. The entire horizontal line or strip that contains any such runs is removed and ignored in further processing. Lengthy horizontal runs of foreground pixels may occur as a result of ligatures in cursive script or as a result of horizontal bars in such letters as "t". Once these strips are removed, any small strokes that remain (which are sometimes located between the horizontal strips)

Fig.2 Original Image

are also disregarded and removed. What should remain following this initial processing is a set of near vertical strokes occupying the areas between removed strips. It is at this point that the extremities of these strokes are detected and bounding boxes or windows are used to enclose each near vertical stroke. Each stroke is then divided in half and the centre of gravity (based on the distribution of foreground pixels) is estimated for each half. The

Original Word Image

Thresholding of Grey-Level Image

Slant Correction of Word Image

Segmentation using a Neuro-heuristic Technique

Character matrix extraction and Normalisation of Segmented Characters

Recognition of Characters using a Trained Neocognitron ANN

two points located are then joined to produce a near vertical line that may be used to produce an angle estimate for that particular stroke. This operation is repeated for each stroke and the mean stroke angle is calculated, to produce an estimate of the word’s slant (b). Once this value is obtained, a transformation is applied to all foreground pixels with coordinates x and y in the following manner: x’ = x - y x tan(b - def), y’ = y

In some cases it was found that due to an over-estimate of the word’s slant, the word was over corrected. In an attempt to eliminate over correction, the "def" parameter was included in the calculation (Usually set to 5). Fig.4 below shows example of a slant corrected word.

Fig.4. Slant corrected word 2.4 Neuro-Heuristic Segmentation Technique An overview of the technique is shown in Fig.5. Fig.6 gives procedure for testing the validity of Segmentation areas

Fig.5.Taining process for segmentation point Validation 2.4.1Overview of the Feature-based Heuristic Segmentation Many excellent segmentation algorithms have been proposed and yet handwriting segmentation is still

Fig.6 Procedure for testing the validity of Segmentation areas challenging and remains an active research area. In this work,straight vertical cuts are employed to segment the word. It is assumed that each word had been successfully slant corrected. For each segmentation point, only the x-coordinate is stored. Some of the important features inherent in handwritten script are identified and are used in the heuristic segmenter. Average letter width is an important measurement and has been employed in machine-printed and handwritten word segmentation. If correct width is calculated, it may assist in confirming whether a segmentation point exists in a particular area with the assistance of other evidence obtained from local features. In the heuristic algorithm, average character width is obtained by vertically scanning the word and identifying characters or character segments that are separated by white space. The horizontal measurements of each character component are recorded and added to a running total. There is one restriction (heuristic) that is employed to ensure that the width is not over-estimated. Upon obtaining measurements for the width of each character component, a rule is used to discount measurements that exceeded some threshold. Therefore, cases could arise whereby a number of measurements would be excluded from the running total. After all legal measurements are totalled, the average width is calculated by dividing the total width by the number of components. This is a very crude method, however in many cases is adequate for obtaining a rough estimate of the character width.

The algorithm then proceeded to search the word vertically once again, to identify areas of low pixel density. As mentioned previously, it is assumed that the words are slant corrected, and readings of low pixel density in the vertical direction would indicate possible segmentation points. Upon examination of a vertical column in the image, if the pixel density is zero, this indicated a white space and automatically warranted a segmentation point with no further investigation. However, if a foreground pixel is encountered upon scanning the vertical column, the pixel density is subsequently calculated. To decide whether the word is to be segmented at that particular point, the vertical pixel density for that column had to be below a certain threshold. The threshold is set to be equal to the following expression: (top-most y-coordinate of the word - bottom-most y-coordinate of the word) X DENSITY_THRESH DENSITY_THRESH is a fixed value, experimentally set to 0.3. Therefore to summarise the above, the vertical pixel density has to be below a value of approximately one-third of the word height to be considered as a possible segmentation point. The final heuristic to confirm whether segmentation should occur at the aforementioned point checked to see if other segmentations has proceeded it at a distance smaller than that of the average character width. This is called the Recent Segmentation Search (RSS). If another segmentation had occurred anywhere in that area, a segmentation point was not set at the point being analysed. Following pixel density analysis, other heuristics are employed to confirm whether the zone under investigation could possibly be deemed a segmentation point. If a vertical column is rejected as being a segmentation point on the basis of pixel density analysis, two further heuristics are activated. The first checks to see whether the point under examination is located on a "valley" or "saddle" of the lower word contour. To achieve this, the pixels surrounding the point in question (three pixels on each side) are examined for possible gradient changes (moving away from the horizontal in the positive y-direction). The purpose of this local feature analysis is to determine possible ligatures between characters in cursive script. These ligatures may sometimes be characterised as being part of a minimum in the word contour. The technique

described above is simple, however coupled with the RSS technique is able to locate zones in the word image that are ligatures. Upon early testing it is found that although minima could be accurately located in the word, some of those found belonged to characters rather than ligatures. Examples of such characters include: ’u’, ’a’, ’b’, ’c’, etc. It is for this reason that prior to allocating a segmentation point at the minimum point of the word contour, the surrounding features in the area are examined. The main focus of this examination targeted the location of "holes" or concavities in the local vicinity of the minimum. Holes are features located in such letters as ’a’, ’o’, ’b’ etc. However, the hole detection method that shall be described here, also allows for the detection of properties that are inherent in certain characters i.e. "partially-closed" holes or concavities. Example characters that contain "partially-closed" holes include: ’c’, ’e’, ’u’ etc. The hole detection algorithm is described below. Prior to detecting a prospective hole, an appropriate Search Zone (SZ) is located. The SZ is rectangular and is centred horizontally about the current segmentation point being examined. The width measured half the average character width to the left of the current point and half the average character width to the right. As the baseline is not used in this heuristic segmenter, the height of the SZ is taken as the length of the top-most y-coordinate of the word subtracted from the bottom-most y-coordinate. The next step is to detect any holes or concavities. This is achieved by identifying whether vertical or slightly curved line segments existed on either side or at the top of the SZ. Put another way, an enclosed region is detected if a convex contour could be detected at the four extremities of the SZ. Therefore, if all three regions are found to contain convex segments (in addition to the bottom ligature), the area in question will be marked as containing a hole. To assist in detecting these "convex segments", a technique is employed that is especially designed to locate objects with hole-like properties in a binary image. The objects composed of background pixels surrounded by foreground pixels are known as "lakes". Conversely, the objects composed of foreground pixels, surrounded by background pixels are known as "islands". To facilitate the location of "lakes" and "islands", they proposed a transformation technique, whereby the binary image is initially divided into 2x2 windows. To transform the image,

each pixel is given a value that indicates the value of its neighbouring three pixels. Some examples are shown below:

0 0 0 1 1 0 1 1 0 1 1 0 0 1 1 0 Value: 1 6 9 14 The numbers above represent the transformed values of the bottom-right pixel in each case. They are obtained by performing the following calculation: value of the bottom right hand pixel = value of top left-hand pixel x 2 3 + value of top right hand pixel x 2 2 +value of bottom left hand pixel x 2 1 + value of the bottom right hand pixel x2 0 The calculation above is repeated for each 2x2 window in the image (scanning in raster order) until all values are transformed. The transformed pixels are analysed in raster order, the beginning of a lake can be detected by locating the values 6 and 14. This search is for the hole detection technique. Although the objective of the hole detection procedure is to locate a lake in the midst of the SZ, the strategy locates the outer boundary (contour) that enclose the lake (or hole). This is achieved by attempting to locate the edges of the contour, denoted by appropriate transformed values, by scanning the SZ in various directions. Below are the search directions accompanied by the set of transformed values that would indicate the commencement of an appropriate outer contour region: Scanning from left to right (left-most contour segment): {5} Scanning from right to left (right-most contour segment): {10} Scanning from top to bottom (top-most contour segment): {3, 2, 1} Scanning from bottom to top (bottom-most contour segment): {14, 13, 12} While scanning in the various directions, a tally of the appropriate values in each direction is recorded. Based on the tally of transformed values in each region, an estimate of the general topology of the SZ can be obtained. As an example, a high concentration of the following values: {14, 13, 12} at the top-most part of the SZ, will indicate the presence of a convex upper contour. If equivalent concentrations of corresponding values are obtained at the left and right-most regions, then the area would be deemed a fully enclosed hole. Other

combinations may include the left and right-most areas alone being populated with relevant values. This situation would also be considered and recorded as a concavity or partially enclosed hole. To establish the various regions in the SZ i.e. left-most, right-most, top-most or bottom-most, heuristics are used to locate/calculate the various boundaries. These heuristics are based on fractions of the SZ height and average character width. Therefore, if a search are being conducted for appropriate transformed values in the left-most region, the search would be confined between the left-most x-coordinate of the SZ and the average character width multiplied by a threshold (value between 0 and 1, experimentally set to 0.5). Similar heuristics are used for the remaining regions. In conclusion, if the SZ was found to contain a complete or partial, convex outer contour, the entire region was marked as "not to be segmented". This is due to the fact that the SZ contains sufficient evidence of the existence of a fully or partially enclosed hole. This in turn suggests that a character, rather than a ligature occupies the SZ. The algorithm of the entire segmentation process is shown in Fig. 6 The above algorithm leads to under segmentation of the word, which is quite damaging to further success in recognizing the word correctly. The algorithm segments each word sequentially from left to right. Each point that is set do not properly take into account the existence of other segmentation points in the local vicinity. The only heuristic that attempts something along those lines is the “recent segmentation point” heuristic. Prior to setting a segmentation point (based on the location of a suspected local minimum), the heuristic algorithm checks the area directly to its left, for a distance of the average character width, to see whether it is too close to a previous point. This heuristic does not take into account the existence of segmentation points in the opposite direction. Therefore, in the algorithm described in this section, the segmentation technique initially segments the word as many times as possible based on a number of features described previously. A novelty in this algorithm is that depending on the feature that is located, a confidence value is assigned to the segmentation point. Subsequently all segmentation points are then examined whereby heavy concentrations of segmentation points are removed to leave the most representative points in a particular area.

Finally, all segmentation points are examined, this time, in the context of points to the left and right and if necessary, points are removed based on their proximity to other points.

2.4.2 Segmentation area analysis: feature selection It is necessary to determine the appropriate boundaries of each area to be validated. Upon observation of a handwritten word following the heuristic segmentation phase, it may be possible to examine the area centred about each vertical segmentation to determine whether it is a valid segmentation point or an invalid segmentation point. The next step is to distinguish differences between the appearance of areas that are valid segmentation areas as opposed to invalid areas. Upon a visual inspection of the areas described, it is concluded that a majority of "valid" segmentation zones are sparsely populated with foreground (black) pixels, as opposed to "invalid" segmentation zones that are more densely populated with same. Secondly, due to the nature of most "valid" segmentation zones, it may be found that either they contain a nominal number of foreground pixels (usually found in words where the characters are separated by whitespace), or the foreground pixels are distributed horizontally at the bottom or centre of the segmentation area. The latter occurs mainly in the case of cursive writing where ligatures occur close to the baseline in many cases, while in the remainder of cases occurring horizontally, slightly elevated above the baseline (when an ’o’ joins a ’t’). Although the general vicinity of the segmentation areas (SAs) are located, to process each word it is necessary to formalise how each area is to be extracted if they are to be classified in further stages. Therefore, it is deemed that the SA to be extracted would be centred about each vertical segmentation point, with the following dimensions: Height of Word x Width Constant. The word height is calculated by locating the top-most foreground pixel and subtracting the y-coordinate of the bottom-most foreground pixel in a word image. Although the width constant would not change in dimension from one SA to another, the word height would not remain constant from one word to another. Therefore to address this problem a re-

scaling procedure based on nearest neighbour interpolation is employed to reduce each SA to a constant size. Various SA sizes are tested and raw re-sized SA matrices are used in conjunction with a classifier. However, it is observed that even with re-scaling, the raw matrices are still too large to be used for classification. Therefore, a feature extraction technique is a possible solution in reducing the size of the matrix as well as retaining vital global features about each SA. The heuristic feature detector is used to segment all words that shall be required for the training process. The segmentation points output by the heuristic feature detector are manually analysed so that the x- coordinates can be categorised into “correct” and “incorrect” segmentation point classes. For each segmentation point, a matrix of pixels representing the segmentation area is extracted and stored in an ANN training file. Each matrix is first normalised in size, and then significantly reduced in size by a simple feature extractor. The feature extractor breaks the segmentation point matrix down into small windows of equal size and analyses the density of black and white pixels. Therefore, instead of presenting the raw pixel values of the segmentation points to the ANN, only the densities of each window are presented. 2.4.3 Neural network configuration The neural architecture is a feed-forward, fully interconnected, multi-layer network with a single hidden layer. Weights are initially set to small random values, and each neuron used a simple, sigmoidal activation function to process the weighted sum (net). The number of inputs corresponded to the size of the raw, re-scaled segmentation area or to the size of the re-sampled density feature vector. For preliminary experiments, re-scaling is the sole pre-processing stage prior to classifier training and recognition. Experiments are conducted with the use of a backpropagation network. It accepts SA patterns with the following dimensions: 20x7 (140 inputs), 10x9 (90 inputs), 9x9 (81 inputs). In other experiments, normalisation is accompanied by density feature extraction, which created inputs of varying sizes: 7x3 (21 inputs), 10x3 (30 inputs), 14x3 (42 inputs). The number of outputs remains constant throughout the experiments. This output can be in the range of 0 to

1, whereby an actual output of <0.5 would indicate an "invalid" segmentation point. Conversely an actual output of >= 0.5 would indicate a "valid" segmentation point. The backpropagation algorithm is the standard one with a momentum variable (α). Therefore many experiments may be conducted, varying α, the learning rate (η ) and the number of hidden units. 2.5 Character Matrix Extraction Character Matrix Extraction refers to the process of locating and storing an area in a given word image delineated by a specified boundary. The extracted area, composed of a matrix of background and foreground pixels, represents the proposed character image. Following heuristic segmentation and neural validation of segmentation points, the boundaries for character extraction may be defined in terms of x-coordinates. The heuristic segmenter is instructed to output a number of proposed segmentation points. Each point is represented by an x-coordinate. Similarly, following neural validation, two segmentation point classes are output. The first is of the "valid" type and could be used to delineate segmentation boundaries. Conversely, the segmentation points deemed as "invalid" are not used in the character extraction procedure. Any "separate" (non-cursive/printed) character components are extracted separately and further dissected if necessary. Therefore, the entire character extraction process first proceeds to sequentially extract all non-cursive/printed character components through the use of character component analysis. Finally, x-coordinates for each connected character component defined either solely by the heuristic segmenter or in conjunction with neural validation are used to define the vertical boundaries of each character matrix. To locate the horizontal boundaries , the area bounded vertically via x-coordinates or the boundaries found as a result of connected component analysis, is examined from the top and bottom. The first instances of foreground pixels located by searching from the top or bottom are deemed as the top-most and bottom-most y-coordinates for the character matrix respectively. The character sets used for training and testing the chosen classifiers are extracted from words. As the words have already been globally preprocessed for slant, noise etc., it is not deemed

necessary to further preprocess the individual extracted characters. However, due to the nature of the classifiers used (neural networks) one type of preprocessing is essential: re-scaling. As neural networks only accept input vectors of uniform size, it is necessary to employ a re-scaling technique as for segmentation area normalisation. 2.6 Recognition of Segmented Characters using neocognitron Using the segmentation points generated for training, segmented characters are extracted to train a neocognitron neural network simulator which is an implementation of Fukushima’s Neocognitron neural network. Characters used for testing are to be extracted using the character matrix extraction algorithm. Following neural network training, segmented test characters are passed through the neocognitron ANN and are classified. 3 Experimental Results

Fig.7. Program output of the image segmented with heuristic algorithm and Image after segmentation points have been verified by neural networks.

Fig.8(a) Original Image

(b)Thresholded at T=120.0363 using iterative threshold method.

Fig. 9 (a) Original Image(Manipuri script)

(c)Thresholded at T= 126.2814

The train pattern file “chd10_2.ptt” for the word

“Chandigarh” is shown below

The input pattern file “chd10_2i.ptt” for testing is shown below:

The output of the test mode and some of the recognised patterns are shown below:

4 Performance Analysis of the net results: The performance of the net is observed by varying parameters such as r,q, S-column, alpha parameter and the decision parameter. It is observed that the best recognition is 90.00% which is detected when the sensitivity parameter (r) is 2; learning parameter q1=104.0, q2=10006.0 and q3=10005.0; S-columns are 1X1; Alpha parameter is 0.0001 and Decision parameter is 14.

Layer Plane number Plane size Mask size S-column r par. q par. U0 1 12 x 12 --- Us1 20 10 x 10 3 x 3 1 x 1 2.00 104.0 Uc1 20 8 x 8 3 x 3 Us2 20 6 x 6 3 x 3 1 x 1 2.00 10006.0 Uc2 20 4 x 4 3 x 3 Us3 10 2 x 2 3 x 3 1 x 1 2.00 10005.0 Uc3 10 1 x 1 2 x 2 Alpha parameter : 0.0001 Decision parameter : 14 Best recognition : 90.00 % Total running time : 0 : 00 : 08 The following pattern table shows the train pattern file “ch04.ptt”.

And the Input pattern file ch12.ptt is shown below:

The recognised screen outputs are shown below:

The train pattern file “m4.ptt” for the word “Manipur “

( ) is shown below:

Input pattern file “m12.ptt” for testing is:

5 Conclusions and future directions The main focus of this paper is the proposal, development and investigation of the different stages that may be involved in a segmentation and recognition system of cursive text. The thresholded cursive English text is employed with neuro-heuristic segmentation and then follows character matrix extraction and recognized using a trained neocognitron. The recognition results found are very good. The iterative thresholding method is employed for different patterns such as Punjabi and

manipuri scripts using different writing stensils and different background and the thresholded images are found suitable for further processing.Recognition Neocognitron is also tested for Punjabi and Manipuri scripts. For the Punjabi script and Manipuri script ,a detailed algorithm based on the features of the corresponding script can be developed for segmenting the characters.

The basic features for segmenting the scripts may be character width estimation, search for the columns of zeros, find a point where the density of pixels is sparse and when a point like this is found the program monitors either side of the point to make sure that it is in between an area of high pixel density.

6 References [1] Sargur N. Srihari & Rohini K. Srihari , 1996, Survey of the State of the Art in Human Language Technology, State University of New York at Buffalo, New York, USA. http://cslu.cse.ogi.edu/HLTsurvey/ch2node3.html#98 [2] Gregory D. Abowd and Elizabeth D. Mynatt, Ubiquitous Computing: Past, Present and Future,GVU Center and College of Computing,Georgia Tech,Atlanta, GA 30332-0280, [abowd,mynatt]@cc.gatech.edu [3] M. Blumenstein and B. Verma, 1999, Neural Solutions for the Segmentation and Recognition of Difficult Words from a Benchmark Database, Proceedings of the Fifth International Conference on Document Analysis and Recognition, (ICDAR ’99), Bangalore, India, 281-284. [4] M. Blumenstein and B. Verma, 1998, A Neural Network for Real-World Postal Address Recognition, Soft Computing in Engineering Design and Manufacturing, P.K. Chawdhry, R. Roy and R.K. Pant (eds.), Springer-Verlag, London. [5] B. Verma, M. Blumenstein and S. Kulkarni, 1998, Recent Achievements in Off-line Handwriting Recognition Systems, Proceedings of the Second International Conference on Computational Intelligence and Multimedia Applications,(ICCIMA ’98), Gippsland, Australia. 27-33. [6] M. Blumenstein and B. Verma, A Neural Based Segmentation and Recognition Technique for Handwritten Words, Proceedings of the World Congress on Computational Intelligence (WCCI '98), Anchorage, Alaska, 1738-1742. [7] M. Blumenstein and B. Verma, 1998, Conventional Vs. Neuro-Conventional Segmentation Techniques for Handwriting Recognition: A Comparison, Proceedings of the Second IEEE International Conference on Intelligent Processing Systems, Gold Coast, Australia, 473-477. [8] D.N.S BHAT & M.S. NINGOMBA , Manipuri Grammar, Central Institute of Indian Languages,(ISBN 3 89586 191 X.LINCOM Studies in Asian Linguistics 1997/II.) http://www.e-pao.net/epPageExtractor.asp?src=education.Learn_Manipuri.html.. http://www.arbornet.org/~prava/eeyek/history.html [9] Ridler, T. and Calvard, S., 1978, Picture Thresholding Using an Interactive Selection Method , IEEE Trans. on Systems, Man, and Cybernetics, Vol. SMC-8, No. 8, pp. 630-632 [10] Philip D. Wassserman, 1989,Neural Computing Theory and Practice, Van No strand Reinhold, New York., NY

[11] M. Blumenstein,2000,Intelligent Techniques for Handwriting Recognition School of Information Technology, Faculty of Engineering and Information Technology, Griffith University-Gold Coast Campus [12] Jack M.Zurada,1997, Introduction to Artificial Neural Systems ,Jaico Publishing house,New Delhi.