A Brief Description of Recognition of Fractions and other vertical groupingsObjects

from Handwriting Strokes.By Kevin Lin

BackgroundOur goal for SKEMEthe research project [] is to develop a multimodal application that incorporates both speech and handwriting for the input of mathematical expressionssymbols. To simplify the implementation, we are using existing software from Microsoft Speech SDK and Microsoft Tablet PC SDK. Microsoft Tablet PC SDK is reasonably accurate for input of words written in the normalregular horizontal charactersorientation; however, it does not recognize vertically displaced characters that are input in a vertical fashion. For example, has some chance of being recognized, will probably be recognizes properly, (though it might not be the first

alternative), but , canwill not be recognized properly: the recognizer program

essentially emulates keyboard input and therefore does not even provide for a possibility of non-linear “typing.”.

A possible approach to solve this problem is to collect the ink strokes, and then, based on the separate bounding boxes for each stroke, determine if the ink strokes are likely to be part of a fraction. After separating the ink strokes into horizontal components, the result can be passed into a handwriting recognizer again, and the end result would be a higher probability of the proper recognition of mathematical symbols that contain elements oriented horizontally.

What we have so far…So far, there is a demonstration program as to how this approach can be implemented specifically for fractions. In order for an object to be a fraction or a partinclude of a fraction, there needs to be a nearly-horizontal bar, indicating a fra.ction. Once the horizontal bar is detected, the algorithm can search segment the remaining elements strokes for objects whose bounding boxes fall above and below the line and also roughly within the left and right boundaries of the horizontal object line. If the bounding boxes do not fall within the left and right boundaries, then the object cannot be part of the fractionSegmentation must also consider. Furthermore, the objects the need need for numerator and denominator objects to be in close proximity with the fraction bar (See diagram below)

ToleranceHowever, it It is very difficult for the user to input a perfectly straight horizontal line. Therefore, the, we idea of tolerance is introduce a tolerance leveld. In short, the tolerance level to gives leeway for the user’s imperfections in handwriting. recognition.. The default tolerance level is 30 pixels (based on the display pixels; the drawing units for the tablet are much finer grain).

Numerator.Bottom

Numerator.Top

Numerator.Left

Numerator.Right

HLine.Top

HLine.Bottom

Denominator.Top

Denominator.Left

Denominator.Bottom

Denominator.Right

HLine.Height

A tolerance level of 30 pixels means that the width of the bounding boxHline-Height shown in the diagram that governs the horizontal line ca can be no greater than 30 pixels. The Another (or the same…) tolerance also defines how close the objects need to be to the bottom or the top of the horizontal line. A tolerance of 30 indicates that the numerator can extend no more than 30 pixels lower than the top of the horizontal bar, and the numerator can extend no more than 60 pixels higher than the top of the horizontal bar. Similarly, a tolerance of 30 pixels means that the denominator of the fraction can rise no further than 30 pixels above the bottom of the horizontal bar, and a denominator of the fraction needs to be within 60 pixels of the bottom of the horizontal bar. A mathematical explanation of tolerance is listed below.

Table 1.1t = tolerance (example t = 30)Horizontal Line Maxheight = t The horizontal line can have

a maximum height of 30.Numerator HLine.Top >

(Numerator.Bottom - t) AND HLine.Top < Numerator.Bottom + 2*t

If HLine.Top = 300, the numerator bottom needs be between 240 and 330

Denominator HLine.Bottom < (Denominator.Top + t) AND HLine.Top >

If the HLine.Bottom = 300, the denominator needs to be between 270 and 360.

Denominator.Top - 2*t

….I think this is heading in the wrong direction. You are duplicating the facilities of the handwriting recognizer if you work on a stroke-by-stroke basis.

1. Handwriting recognizer should be used FIRST. If it does not produce a plausible result, then we can try segmenting.

2. We should segment as little as possible. Find a fraction bar, above, below, left, right. Try again.

3. The technique must be generalized for super/subscripts4. There are other reasons for horizontal strokes. As you point out, “+” has a

horizontal stroke. So also do E, T, =, ~, ≤,≠,,±, etc.Where we go from here…These are the preliminary steps in developing an all-encompassing algorithm that accurately identifies fractions and other objects. .

You have to generate multiple possibilities, assign probabilities. The word “handle” doesn’t describe what you are proposing.

The program needs to handle cases of multiple strokes. Also, some small characters such as the multiplication “dot” can be mistakenly identified as a horizontal bar. Therefore, the algorithm above needs to be extended to compare the width to length ratio. Also, characters like “+” can be mistakenly identified as fractions, so exceptions need to be handled on a case by case basis. Furthermore, we can extend the tolerance principle to assign each set of handwriting input strokes a probability. If the probability exceeds a threshold, then the objects can be grouped together as a fraction.

A similar algorithm can be applied to superscript and subscript. If a bounding box has a smaller bounding box to the upper right or lower right, these can be identified as superscripts.

CODE (This is for reference purposes only)

(defun getboxes () (let ((strokes (list-of-strokes)) (listofboxes nil)) (applytoall strokes listofboxes) ))

;;THERE IS PROBABLY A SIMPLER FUNCTION (i.e. MAP) TO THIS;;THIS APPLYS THE FIND-BOUNDING-BOX function to make a list of boxes

(defvar tolerance 30)

(defun applytoall (lista listb) (cond ((not (eq lista nil)) (cons (find-bounding-box (car lista)) (applytoall (cdr lista) listb)))))

(defun checkforfraction (checklist totallist) (cond ((not (eq checklist nil))

(cond ((> (box-height (car checklist)) tolerance)

(checkforfraction (cdr checklist) totallist)) (t (print "fractionbar") (fractionhelper (car checklist) totallist) (checkforfraction (cdr checklist) totallist))))))

(defun fractionhelper (item totallist) (let ((otheritem nil)) (cond ((not (eq totallist nil)) (setf otheritem (car totallist)) (cond ((and (< (box-left item) (box-left otheritem))

(> (box-right item) (box-right otheritem)) (or (and (< (box-bottom item) (+ (box-top otheritem) tolerance))

(> (box-bottom item) (- (box-top otheritem) (* tolerance 2))))

(and (> (box-top item) (- (box-bottom otheritem) tolerance)) (< (box-top item) (+ (box-top otheritem) (* tolerance

2)))))) (addtext (box-left otheritem) (box-top otheritem) purple 1 "Fraction"))) (fractionhelper item (cdr totallist))))))

(defun drawall (drawlist) (cond ((not (eq drawlist nil)) (draw-box SKEMEwindow (car drawlist)) (drawall (cdr drawlist)))))