BlackBerry Exhibit 1017, pg. 1...1.1 BlackBerry Exhibit 1017, pg. 12 ( Input to a computer can be...

BlackBerry Exhibit 1017, pg. 1

HAND BOUND srz CAMPUS BIMOERY 108-ll!J Harbol'd $t. Torortto, Ont. MSI 1Ge l e:I. 881-2100


NOTE:

(a)

or

THE UNIVERSITY·oF TORONTO LIBRARY

MANUSCRIPT THESIS AUTHORITY TO.DISTRIBUTE

The AUTHOR vill sign (a) or (b)~ It is the intention of the University that there be NO RESTRICTION on the distribution of theses save in exceptional cases.

I agree that this thesis ma.y be lent, and :that single copies r:;;ay be made for individual readers.

Author's signature •••• J\NYl S/JL ..... . vithout restriction.

Date ... C •. ~~oo~\9o~2- .- .

(b) except that-until •••••••••••••••• 19 .• (normal ma.ximu'!:l delay is t-wo years) it may be consulted in the University Lib~ary only vith vritten ·permission on eac? occasion from me •

. .Author.is signature • • • • • • • • • • • Cl •••••••• Date "' •• .it •••••••••••• 4 ••

This restriction is ~11+.~rirized for reasons vhich see."'.!l. to me, as Head of the Graduate Depar~men~ oI ••••••••••••••.•••••••••••• ,to be sufficient.

Sig~ature of Graduate Depart~ent Read •o•••••••••t:••~••••••••••••• ••

. Date . c ... o •• e ••• ~ •• o ...... o • c •••••••••

BOP.ROWERS undertake to give proper credit for ar:.y use made-cf the thesis, and to obtain the consent of the author if it is propo.sed to make extensive quotations, or to reproduce the tC.esis in ;;hole or in pa::-t..

Signature of borrover Address Date


A FLEXIBLE HUMAN MACHINE INTERFACE

by

Nimish Mehta

A thesis submitted in conformity with the requirements for the

Degree of Master of Applied Science in the University of Toronto

October, 1982


Acknowledgments

I wish to thank the many people without whose willing help, this

effort would have been impossible.

In particular, I wish to record with gratitude, the extent of support I

received from Professors F.E. Holmes and K.C. Smith. Their enthusiasm and

generosity has not gone inappreciated.

I owe a special thanks to Anita, for simply being there when. I

needed a friend. I also wish to thank my friends, Surd and Vinod for

understanding in spite of everything, and Laura for being very helpful.

Finally, and I hesitate to say this because it seems like a formality

when I read it, which it is not - I sincerely thank my family for their patience,

and support throughout the past 2 years. Without them, this would have never

been possible.

10th. September, 1982 N.Mehta

i


Abstract

The human-machine interface is an increasingly critical elemeot in

modern computer systems. While input devices such as keyboards, digitizers,

mice and others exist, they all suffer from applications limitations. A new

solution which achieves a better balance between human factors and flexibility

is offered.

The interface described in this thesis features freedom of key shape

and size. It also incorporates attributes of existing input transducers. This

versatility is achieved with a departure from conventional electrical

techniques. The system uses a TV camera focussed on a panel area to generate

electrical signals in response to optical events. The system simply 'looks' at the

panel area and activates processes in response to perceived action. In use,

keyboards and complex control panels are reduced to drawings on a transparent

sheet allowing both, great flexibility of change as well as the ability to create

·human-engineered input devices.

ii


Table of Contents

Aclatowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

I - Introduction

1.0 Introduction. . . . . . . 1.1 Organization of Thesis • ..

II - System Overview

2.0 Introduction •••••••••••••••.•• 2.1 Interface Response Time limits. 2.2 Interface operation overview •• 2.3 Interface components overview. 2.4 Conclusions .•••••••••••••••.••

III - Front End Implementation Details

3.0 Introduction •••••••••••.••• 3.1 The Enclosure •••••••••••••• 3.2 The Analogue Buff er and Converter. 3.3 Conclusions .................. .

IV - The Image Processor

4.0 Introduction ••••• 4.1 Pr9cessor Capabilities 4.2 Processor use and Software Support

. .

...

. .

.... 4.3 Conclusions •••••••••••.•••.•• . . . . . . .

V - Algorithms and Software Strategies

. . . .

.. . .... . 1.1 . ... . 1.5

• .2.1 • .2.1 • .2.4

• •• .2.4 • .2.7

. . . • .3.1 ... . ..... . 3.1 • •• 3.2 . ... ...... . 3.5

..... . . . . . ....

• •• 4.1 ••• 4.1

•• 4.5 • .4.6

5.0 Introduction •••••••••••••••• . . . . . . . . ... .• 5.1 • .5.2 5.1 Image Thresholding and Storage •••••

5.2 Noise Reduction and Thinning. 5.3 Tracking •••••••••••• 5.4 Code search and Output • . . 5.5 Conclusions .•••••.••• . . . . ...

VI - System Integration

6.0 Introduction •••••••••••• 6.1 Use of the prototype system •••••••• 6.2 Spatial Resolution Testing • 6.3 Interface Response Time Delay ••

iii

. .

. .

. .

. . . . . . .... .....

. ... . .

•• • 5.3 • .5.4

•• • 5.6 .5.7

. . . ••• .6.1 .6.1 . ...

• • .6.2 • • .6.8


6.4 Typical Applications of the Interface. 6.5 Conclusions .................. .

VII - Condusions

7 .0 Introduction ••••••••• 7. l Possible enhancements 7 .2 Conclusions •••••••••

•••• 6..10 . ...... . 6..12

. ... . 7.1 •• 7 .. 1

.7 .. 4

List of References ......................................... R .. l

Append.ix A

Calculation of The Minimum Microcycle Time of the Image Processor •• A .. I

Append.ix B

Details of Software Implementation and Tools ••••••••••••••••••• A..4

iv


List of Figures

Chapter I

1.0 Tablet ............................................ . 1.3 1.1 Mouse .•.....••..•..•...••....••.••..•..••.••....• . 1.3 1.2 Basic components of the Human-Machine Interface .••••••••••.• 1..4

Chapter II

2.0 Model of a human operator •••••••••••••••••.•••••••••••• .2.2 2.1 Optical System configuration ••••••••••••••••••••.••••••• .2.4 2.2 Block diagram of System components ••••••••••••••••••••.• .2.4

Chapter III

3.0 Construction details of the FHMI ••••.•••••••••.•••••.••••• 3.2 3.1 Block diagram of Analog Buffer and A/D Converter ..••••••••.•• 3.3 3.2 EIA RS-330 Video Signal Specification ••••..•••.••.•.•••••.•• l.4

Chapter IV

4.0 Block diagram of System components •••.•••••••.•••.••••.•• 4.3 4.1 Block Diagram of the Image Processor ••••.•.•.••••••••.•••• 4.3

Chapter V'

Chapter VI

6.0 Overlay image under bright ambient light ••••.•••••••••••••• .6.4 6.1 Overlay image under normal light •••.••••••••••••••.•••••• .6.4 6.2 Overlay image under very low ambient light ••••••••••••••••• .6.4 6.3 Overlay image taken with a digitization threshold of 6 •••••••••• .6.6 6.4 Overlay image taken with a digitization threshold of 15 •••••••• .6.6 6.5 Overlay image taken with a digitization threshold of 32 ••••••••• .6.6 6.6 Overlay image taken with a digitization threshold of 39 ••••••••• .6.6 6.7 Finger proximity test at separation of 0 mm ••••••••••••••••• .6.7 6.8 Partial centroid computation of blobs in Figure 6.7 •••••••••••• .6.7 6.9 Finger proximity test at separation of 1 mm ••••••••••••.•••• .6.7 6.10 Partial centroid computation of blobs in Figure 6.9 •••••••••••.• 6.7 6.11 Finger proximity test at separation of 2 mm •••••••••••••••••• 6.7 6.12 Partial centroid computation of blobs in Figure 6.11 ••••••••••• .6.7 6.13 Finger proximity test at separation of 3 mm ••.•••••••••••••• .6.7 6.14 Partial centroid computation of blobs in Figure 6.13 ••••..••••• .6.7 6.15 Overlay Image taken with two separated metal pieces placed on it •• 6.8 6.16 Attempt to compute centroids of the blobs in Figure 6.16 ••••••• .6.8 6.17 Overlay Image taken with two touching metal pieces placed on it . .6.8 6.18 Attempt to compute centroids of the blobs in Figure 6.17 ••••••• .6.8

v


6.19 Overlay Image taken with two overlapping metal pieces ••••••••• 6.8 6.20 Attempt to compute centroids of the blobs in Figure 6.19 ••••••• .6.8

Chapter VII

Appendix A

A.O Block Diagram of the Image Processor •••••.••.•••.•.••••. A.l

vi


Chapter I : Introduction

1.0 Introduction

In computer systems today, there is a continuing problem of finding

a good human-machine interface that will allow suitable physical gestures to

represent logical ideas. This problem will grow considerably in the future with

the increased use of computers. It is estimated by Montgomery (1) that the

time is rapidly approaching when 200 million or more alphanumeric keyboards

will be needed in North America alone. Moreover, keyboards are not the only

mode of input: transducers such as tablets, joysticks, light pens and 'mice'

(described below) are also b~ing used to advantage in many complex computer

systems. However, users of such systems are often inconvenienced by the need

to operate many input devices simultaneously in order to fully utilise the

capabilities of the system. The problem is related to the fact that most input

transducers cannot be configured to perform all the necessary input functions

efficiently.

The human machine interface using a TV Camera and additional

hardware described in this thesis solves many of the existing problems by being

capable of emulating tablets, joysticks, keyboards and other devices

simultaneously. Also, the interface is reconfigurable by software, thereby

lending itself to extreme flexibility and human engineering.

1.1


(

Input to a computer can be classified as either discrete or

continuous (2). Discrete input essentially consists of easily distinguishable

quanta of information. Generally, each quantum is sep~rately identifiable, and

need not be viewed in context to be understood. However, Continuous input

does not comprise distinguishable quanta. The information communicated may

not be discernible without knowing its neighbourhood or context. For example,

contrast the function of a radio on/off switch with that of a volume control

knob. However, each type of input can be made to simulate the other:

reducing the quantum size sufficiently, makes a series of discrete actions

appear continuous, while establishing a level-threshold on a continuous input

device makes it appear discrete. It should be emphasized that each type of

input device has it's own area of application. A typical computer system will

most likely need both types of input. A particular system implementation may

choose to simulate one type with another for a variety of reaso.ns such as,

reduction of hardware/software and therefore cost. However, the end result is

almost invariably a relatively inefficient human-machine interface. Three

major types of input transducers (keypads, tablets and 'mice') will be discussed

in this chapter.

The keyboard remains the most popular entry device today. A

particular case of this type is the standard typewriter keyboard (also called the

'QWERTY' keyboard). Another example is the cash register keypad found in

many supermarkets. Irrespective of the actual physical arrangement, the basic

component is a 'key'. Many types of keys exist: those which possess a physical

travel of a few mm., those with a travel of a few tenths of a mm. and finally

those which need only be touched. The actual sensing mechanisms vary widely

1.2


too, from Hall effect sensing to capacitance change detection. But regardless

of these details, keys allow only discrete input: they are incapable of accepting

continuous signals. Furthermore, the physical key layout is fixed. It is 'possible

however, that keys can assume user-defined meanings.

A 'tablet' is used to describe a flat surface on which the user draws

with a stylus (Figure 1.0). The likeness of the tablet and stylus to paper and

pencil makes them a particularly natural combination for graphical input. A

special case of the tablet is a sensor/ detector arrangement that fits on the

screen of a computer terminal, allowing the user to point at menus drawn on

the screen by the computer. In this case, the stylus is a finger. All types of

tablets sense input by resolving the position of the stylus along two orthogonal

axes. This information about its physical position is used by the controlling

computer to activate appropriate processes. The tablet, therefore, is

essentially a discrete input device which can be made to appear continuous if

its resolution is increased sufficiently. An overlay can be placed on a tablet to

make it resemble a keypad. Keys on this simulated keypad can assume user

defined meanings. However, this mode of coordinate-based detection

automatically precludes multiple inputs since the position of more than one

stylus cannot be resolved unambiguously. This further implies that a tablet

cannot emulate a keypad fully and vice versa.

A mouse consists of a small enclosure resting on two small wheels

whose orthogonal axes are horizontal. Each wheel is connected to a

potentiometer (Figure 1.1). As the mouse is pushed around on any flat surface,

its movement in two orthogonal directions is translated into rotation of the

1.3


CUl'l)or

+ screen

Figure 1.0 Tablet

Figure 1. J Mouse


potentiometers. The controlling computer now has access to position

coordinates relative to the previous position, and uses this information to

activate appropriate processes. The joystick has been included in this type

because functionally, it bears some similarity to the mouse. It comprises a

small metal rod which is capable of moving in two directions. The rod is

coupled to two orthogonal potentiometers and its movement is translated into

their rotation. These two devices, unlike those mentioned previously, are of the

continuous input type whose outputs are discretized by the digital computer.

The mouse has two real disadvantages: it cannot be used for tracing data from

paper, since a small rotation of the mouse or a slight loss of contact will cause

a cumulative error in all the readings; furthermore it is very difficult to hand

print characters for recognition by the computer. The joystick is suited only

for cursor movement and positioning tasks since it simply cannot be used for

any data entry at all.

The human-machine interface (3) described in this thesis was

developed to produce a general input device with the desirable attributes of the

conventional input devices. It consists of a 'keyboard' area, a TV Camera

looking at the area and a high speed image processor to process the video

output from the camera (Figure 1.2). Keys on the keyboard are virtual rather

than physical, being created by opaque drawing of their outline on a plastic

sheet. The camera 'sees' all action on the keyboard, passes the video

information to the image processor which activates processes in response to

relevant features of the video frame. The result is an input device that can be

used, under constraints, as a discrete data entry device and a continuous input

device simultaneously. Also, it has the potential of being even more general

1.4


COMPUTER

.-- OVERLAY

'KEYBOARD' --AREA

--TV CA.."'1ERA

Ol:TPLT ---~DEVICE (CRT)

Figure 1.2 - Basic components of the Human-Machine Interface


than that since all optical events on the keyboard can be mapped onto an

electrical output, with the mapping being totally software alterable. Flexibility

of use arises mainly as a consequence of three powerful attributes of the

interface. The first is that any set of closed shapes can be drawn on a plastic

sheet (called the overlay) and used as keys whose function is defined by the

user. This implies for instance that the shapes themselves can be used to

describe the functions they represent. Thus for example, an overlay intended

for use in a computer graphics system could depict pots of paint which when

touched serve to change the colour being output to the associated colour

display. While one part of the overlay could contain these 'pots of paint' as

discrete functions, some other part of the overlay could be defined as a

'sketching pad'. Any movement on this area would be replicated on the monitor.

This is an example of continous input to the interface. The ability to accept

both continuous and discrete inputs is the second powerful feature of the

interface. The third powerful aspect of the interface is the ability to change

overlays easily, and at will. In terms of the painting example, this implies that

artists of different capabilities could be accommodated: an overlay for the

beginner would consist of fewer colours, and without any 'palette' for mixing

them. In general, these three capabilities allow simulation of most input

devices, as well as the ability to incorporate completely new features. Also,

one interface can be rapidly adapted to a variety of modes and applications by

simply changing overlays.

1.1 Organization of the thesis

Chapter 2 examines the consequences of using a configuration such

1.5


as the one described above. It establishes the essentiql physical and electrical

components of such a system and estimates limits on their performance.

Chapter 3 describes the physical implementation details of the resulting

apparatus referred to hereafter as the Flexible Human-Machine Interface

(FHMI). Chapter 4 describes the hardware implementation details of the FHMI.

Chapter five describes the software implementation details of the FHMI.

Chapter 6 outlines some system integration aspects of the FHMI and results of

performance evaluation tests on the prototype. Finally, Chapter 7 suggests

avenues for enhancement and concludes the thesis.

1.6


Chapter II : System Overview

2.0 Introduction

This chapter discusses the fundamental response constraints of an

ideal human-machine interface. The keystroke response time limits of the

FHMI are established. An implementation of the interface using a standard TV

Camera is presented. Constraints imposed on such a system are also discussed.

2.1 Interface response time limitations

Interface response time is the time interval between the appearance

of a stimulus at the interface and the actual response to that stimulus. A range

of acceptable response times will be estimated in this section. Knowledge of

such a range is a prerequisite to the design of an effective human-machine

interface.

An ideal human-machine interface would respond to stimulii without

any delay. However an equally good, and more achievable, response time is one

which makes the response appear instantaneous to the operator. This is the

optimum response time that the design process was aimed toward. The

response time becomes unacceptable when the interface is unable to keep up

with the response time of the operator (i.e. the interface becomes slower than

the operator) since this precludes the use of the interface in interactive

2.1


systems. Both these times wil be estimated in this section by analyzing a model

of the human operator (4). Results from other studies will be used to confirm

the validity of the estimates.

The model of a human operator as described by K. Ramakrishna (4)

is presented in Figure 2.0. It comprises three 'processors': the perceptual, the

cognitive and the motor. These processors have cycle times tp, tc, and tm

respectively, that vary quite widely (in the figure they are shown first as an

average and then as a range). Humans perform tasks in three steps. First, the

perceptual processor converts data from· external senses to symbols in the

working memory of the cognitive processor. Second, the cognitive processor

interprets this symbolic data and generates, an instruction stream. This stream

is interpreted and executed by the motor processor. The total time to perform

a task depends on the number of cycles taken by each processor. Experienced

operators exhibit a high degree of pipelining (performing multiple tasks

simultaneously) between all three processors, as well as a low number of total

cycles. The response time range can be calculated assuming first an

inexperienced operator and then an experienced operator.

The slowest response time which would still be acceptable for an

interactive system can be found by determining the operator response time. It

is assumed that the operator is pressing keys as fast as possible, but is

inexperienced at the task and therefore cannot 'pipeline' his mental processes.

In other words, his cognitive processor instructs his motor processor to press a

key, and then waits for his perceptual processor to perceive the key depression

before commencing another cycle. Thus, this time is the sum of the times

2.2


EXTERN SENSES

VOLUNT MOTOR

.• ACTION

AL

ARY

s

--

-

PERCEPTUAL PROCESSOR -t = 100 ms.

p (50-200)

-~OTOR -PROCESSOR t = 100 ms. m

(70-360)

Figure 2.0 - \.iodel of the human operator

IMAGE STORE

LONG-TERM

WORKING MEMORY

MEMORY

j. %

I f , COGNITIVE PROCESSOR t = 100 ms.

c (25-170)


taken by each processor multiplied by the number of processor iterations

necessary to press a key. (In more complicated tasks, the inexperienced

operator moves his hand towards the target in small steps, and examines the

result after each step). Assuming one cycle (the minimum), this value works

out to (50 + 70 +25) = 145 ms. Other studies can be used to confirm the validity

of this number. Card et al (5) use a version of Pitt's Law (6) to characterize the

time required to make straight line hand movements. (Fitt's Law predicts the

time to position a hand on a target of a known size, given an initial position of

the hand at a known distance from the centre of the target). They derive the

information processing capacities of the eye-hand coordinate system to be

about 100 ms./bit. They use this information to show that the tapping rate

(repeated depressions of the same key) for step keys in a cursor-positioning

application would be 150 ms./keystroke. Their experiments confirm this time

estimate. The typing rate for random words has been reported by Devoe (7) to

be 209 ms./keystroke. Thus, it may be concluded that the slowest acceptable

interface response time is 150 ms. per keystroke.

The optimum interface response time can be determined by

estimating the time taken by an operator to pipeline his perceptual and

cognitive processors only. The reason is that an experienced operator normally

presses a key on an overlay knowing the result (an example of this is an

experienced typist typing). His perceptual and cognitive processors simply

confirm the expectation. Assuming the minimum processor times of the model,

the FHMI must respond within 50 ms. of a stimulus for the response to appear

'instantaneous'. This corresponds to a typing rate of just over 200 words per

minute.

2.3


2.2 Interface operation overview

The optical system configuration is repeated here in Figure 2.1. It

is assumed that the keyboard plate fills the viewable field. It is apparent

immediately that the camera will view some unnecessary and distracting details

beyond the overlay through the keyboard plate. Chapter 3 describes the

techniques used to perform spatial filtering to correct this problem. The field

viewed by the camera is such as to contain only dark outlines of the keys drawn

on the overlay and of any other objects (such as fingers) on or close to the

overlay. The analog video output of the camera is converted into a digital

image and stored in the memory of an attached computer. Then, an image of

only the overlay (taken on 'power-up' and there!ore already existing in memory)

is subtracted from each new image. The resulting difference image will only

contain dark spots corresponding to fingers pressed on the overlay, smce all

other details were common and were subtracted. The 'overlay' image and the

'finger' image can be used to determine any keys being pressed on the overlay.

The technique utilized is to detect a shadow of a finger, and determine whether

any key outline encircles it. If such an outline is successfully determmed, the

code corresponding to the key is provided at the output. A block diagram of the

components needed to perform these operations is provided in Figure 2.2.

2.3 Interface components overview

This section describes the components needed to configure an FHMI

as described in the previous section. It also describes the tasks that have to be

performed, their subdivision amongst the components and the corresponding

2.4


' \

' \

' \ ' ' \

' \

' \ ' \ '

I I

I I I I I

I I I

I I

I

TRANSLUCEN':J PLASTIC FILTER'

GLASS

I SHEET

I I

I ENCLOSURE

TV CAMERA ~----_,FOCAL POINT

Figure 2.1 - Optical System Conf i~uration

~·


TV CAMERA

OUTPUT PORT

A/D CONVERTER

HOST cm!PrTER

HIGH SPEED IMAGE PROCESSOR

Figure 2.2 - Block Diagram of System Components

SYSTEM BUS

MEMORY


hardware-software tradeoffs which result.

The TV Camera video output is fed into an analog-to-digital

converter in the attached computer which creates a digital image ot the scene

as viewed by the camera. Assuming standard video rates, the active view time

per line is about J3 microseconds. It is also assumed that the outlme resolution

of the interface is about 2 mm., and the overlay is 50 cm. long x 20 cm. wide.

Under these conditions, the number of pomts sampled per lme (called pixels)

must be 500 mm./2 mm. = 250. The time per pixel equals is /us/250 = 212 ns.

This implies that the conversion rate of the A/D converter must be 1/212 ns. or

about 5 megasamples per second. Commercial 'Flash' converters capable of

digitizing analog signals at speeds of 5 megasamples per second are

prohibitively expensive. The reason is that they are designed to resolve the

analog signal into 4-8 bits of digital data per pixel, which requires extensive

hardware. However, in the present application, image processing occurs only on

'binary images' - that is, outlines on an overlay are allowed to be only black or

white. Even if available, greyscale images would have to be converted into

binary images by setting all pixels darker than a predetermined threshold to

black and the rest to white. This suggests the use of a 1 bit resolution A/D

converter. Such a converter would be considerably lower in cost, but would

seem inflexible if the black-white threshold is to be changed. Chapter 3

discusses details of the solution adopted. The pixel rate of 1 every LOO ns.

automatically implies that more than a standard microprocessor is required,

since having cycle times of 1-2 1us, they are incapable of operating on single

pixels as they are generated. Accordingly, a high speed image processor

capable of processing data at these rates was designed and implemented. '

2.5


Chapter 4 gives details of the image processor.

Arter digitization, the image has to be subtracted trom one already

existing in memory. It can be shown why this step needs to be executed

considerably faster than is possible using a general purpose microcomputer: m

the earlier analysis, it was concluded that each raster line would be digitized

into 250 pixels. A similar analysis mdicates that about l 00 scan lmes must to

be digitized. This implies a total of 25,000 pixels. Even at best, using one bit

to represent a pixel being black or white, this converts to a storage requirement

of 3K bytes. Subtraction of two bytes would require the execution of no less

than 3 instructions per byte, taking 10 /US (assuming a 1.5 1us processor cycle

time). Thus the total time to subtract one entire frame from the other would be

30 ms. The minimum response time goal derived in the earller section was .50

ms. Assuming standard video rates, a single frame scan time is 16 ms •. This

implies that only .54 ms. would be available for processing the frame. In view

of this, 30 ms. was deemed an unacceptably large percentage of it and design of

* faster hardware was undertaken as described in .Chapter 4.

The result of the subtraction process is an image containing dark

spots corresponding to fmgers pressed on the overlay. The objective is to

determine whether or not any key outlines encircle each dark spot. This can be

done by one of several 'blob' analysis algorithms described in Chapter 5. They

all essentially reduce images containing dark spots t'blobs') to a set of

descriptive parameters including centroids and hierarchical blob links ('parent',

'child' and 'sibling') and others. These parameters are used to determine

encircling key . outlmes (called parent-outlines). Chapter 5 shows why the

* 2.6 In the implemented version of the interface, the image processor was not used to perform this task since the overall response time of the interface was found to lie within acceptable limits. However, its existence gave us the flexibility of changing to a higher speed scheme if needed.


algorithm used utilizes a superior technique compared to the others available.

Smee the code etficiency of an implementation ot these algorithms depends on

the software capabillties of the computer being used, a commercially available

host processor having a very powerful instruction set and sottware development

tools was felt to be necessary. To develop similar support for custom designed

hardware would mean an extensive hardware-sortware ef!ort. A 6~00

microprocessor based system available at the University of Toronto was used as

the host computer. Chapter 4 describes the hardware system configuration, and

details the interconnection between the high speed image processor and the

host microcomputer.

Once the encircling keys are known, the codes corresponding to each

can be output. This is done entirely by the host computer. Chapter 5 describes

the data structures implemented to achieve this efficiently.

2.4 Conclusions

An overview of the system has been presented. With it, the

sottware tasks as well as the hardware components necessary to achieve the

system goals have been identified and the hardware-software tradeoffs

outlined. The next 5 chapters contain detailed descriptions of various aspects

of the system. Chapter 3 describes the physical implementation details of the

FHMI as well as the analog-to-digital converter mentioned earller. Chapter 4

describes the high speed image processor implemented to provide the

computational speed that was shown to be necessary. Chapter 5 contams

descriptions of the 'blob' analysis routines as well as the code output process

2.7


outlined earlier. Chapter 6 contains details of tests done on the prototype. It

also includes mterpretation of the results of these tests as well as rules ror

creation of effective overlays. The last section in Chapter 6 lists a few

situations in which the FHMI may be used to advantage. Fmally, Chapter 7

suggests avenues for improvement of the prototype and draws final conclusions

regarding this entire experiment.

2.8


Chapter III : Front End Implementation Details

3.0 Introduction

This chapter describes the physical implementation details of the

FHMI. The mterface basically comprises an enclosure with opaque sides wnose

top surf ace which is the 'keyboard area'. The enclosure contains a TV camera

aimed at the underside of a rigid transparent top surface on which is placed a

translucent film. 'fhe film and it's purpose are also described. The output of

the camera is fed to an analog buffer and then to an analog-to-digital

converter. The circuits implemented to accomplish this are described in

Section J.2.

3.1 The enclosure

The design and construction of the enclosure is relatively

straighttorward. An old teletype metal box was modit1ed to serve as the basic

structure. A custom-cut glass sheet forms the top. The camera 'looks' looks

through the glass sneet from underneath. To compensate tor the long focal

length of the camera lens, the the camera was made to point downwards, with a

mirror placed at the bottom of the enclosure. This effectively doubles the

optical path length. The mirror is rendered unnecessary if a lens with a shorter

focal length is used.

3.1


To use the interface, an 'overlay' must first be created. An overlay

is a arawing of closed shapes Ontenaed to be used as keys) on a transparent

plastic sheet. Rules for creation of overlays are described m detail in Chapter

6. The overlay is placed on the glass top. Smee the top tills the field of view

of the camera, activity on the overlay can be 'seen' and interpreted by the

attached computers. In response, appropriate processes can be activated.

However, smce both the giass top and overlay are transparent, the camera will

view some distracting details beyond the overlay. This could impair accurate

interpretation of the releV'ant features of the video frame. Hence, a

translucent !llm was placed on the glass top with the intention or reaucmg lignt

transm1ssivity. Several films from various sources were tried on an on a trial

and success basis. A projection screen !llm manufactured by Edmund Sc1entl:l:ic

was found to be appropriate because of a kmd of spatial selectivity it provides.

In this particular application, everything a few mm. above the top surtace

appears as a grey blur. However, definition of closer ob1ects is considerably

better. Further experimentation showed that untortunately, these optkal

characteristics {and those of other available films too) were useable only under

normal overhead lighting. The image definition detenoratea unaer very low and

very high ambient light. To compensate for this,. a small fluorescent lamp was

placed on the same plane as the glass top and at one end o:t it. This also serves

to nullify the effect of single light sources which create shadows. (Figure 3.0)

3.2 The analog buffer and converter

The output of the TV Camera is fed into an analog buff er and then

converted into digital data using an A/D converter. A block diagram of the

3.2


FLl'ORESCE:JT LA .. '-fP

GLASS TOP WITH

PLASTIC FILTER

ENCLOSURE~~~~

KEYBOARD --- AREA

--TV CAMERA

Figure 3.0 - Construction Details of the FH~I


circuits implemented is provided in Figure 3.1. The functional blocks consist of

an analog buffer, an analog-digital converter and a syncnronizat1on pulse

stripper.

An LM318 based circuit forms a low noise ampli!ier/butfer with a

high input impedance. lt was selected oecause of the hign oandwidth neeaea to

ampli!y video signals: a resolution of 256 pixels per scan line implles

approximately LOO ns. per pixel. within this period, the v1aeo signal can make a

transition from 'black' to 'white'. This implies an amplifier a bandwidth of at

least U/LUOns.JxlUOO = 5 MHz. Also, the signal to noise ratio of such an

ampli!ler has to be fairly high. This is because the video signal amphtude

(approximately l V peak) is thresholded mto 'black' or 'white' with a resolunon

of l part in L5b. This converts to approximately 4 m V. Hence, the overall

noise component cannot exceed a few m V. rms tor a single step to oe

meaning!ul. *

The A/D converter was selected as a single bit converter with a

variable comparison level. Normally, a 4-8 bit flash A/D would be used. The

advantage of flash conversion is that a complete picture o! the viewed field can

exist in the frame buffer at all times, and black-white threshold dec1s1ons ror

bmary image operations may be made in software as and when needed. This

precludes trie need tor repeated digitization at d1!ferent tnresnold levels.

Unfortunately, it also has numerous disadvantages. Firstly, flash converters

require 2n-1 analog comparators, wnere n is the number of bits the signal is

being quantized into. For 8 bits, this means 25.'.> comparators ! Also, it it is

assumed that the picture is being digitized to L.'.>6 pixels x b4 pixels, men this

* 3.3 Both the LM318 buffer and the high speed comparator (LM360) have input offset voltages of 3-5 mV. that must be compensated for accurate digitization.


VIDEO IN

VERT SYNC GEN

1

BUFFER

D/A CONVERTER

HORIZ SYNC GEN

SYSTEM BUS

TO IMAGE 2 3 PROCESSOR

· Fip,urc 3.1 - Block Diagram of the Analog Ruffer and J\/D Convc:rtpr


means a storage requirement of 16Kbytes per frame. Lastly, most flash

converters available today are extremely expensive. lt was tor tnese reasons

that a single bit converter circuit with controlled threshold was selected. This

method has the advantage ot reduced storage requirements and cost. A bit map

of the viewed field can be created and maintained in memory. For a 256 x 64-

pixel image, this means a storage requirement of only 2 Kbytes/trame. The

disadvantage is that of the inflexibility of the black-white threshold, once

taken. To circumvent that, and attord as much flexibility as possible to the

controlling software, a circuit in which the comparator reference level of the

A/D converter could be varied under computer control was implemented. This

comprises a latch and a D/ A converter which converts the outputs of the latch

mto an analog signal tor the comparator reference. The result is a circmt tnat

generates a bit map of the viewed field, but with the decision of the black

white boundary under program control with an 8 bit resolution.

The buffer - converter combination worked well, digitizing reliably

and consistently. The noise superimposed on the buffered video signal was

measured to be 0.7 mV. average. The linearity of the threshold generator was

found to be .±,2 mv. These parameters cause a worst case digitizanon error of.:!:.

1 lsb. The software processing is done in recognition of this fact.

The synchronization pulse stripper extracts the nonzontal and

vertical synchronization pulses from the composite video. These pulses occur

at the beginning of a line and a trame respectively. They are used by the image

processor to synchronize its video data acquisition to the raster scan. The

camera output contormed to the ElA H.S-jJQ monocnromatic video

3.4-


r 'Xf.- (0.71 V)

~.3f..

l--t 1.0

I I I

. j4. 1 s--f 4. 2 s I

I I I --1 I I

v h" w i.te

I I

I I I I I I I I I I

Notes: Diagram not drawn to scale

v blank

Vblack (r/J.r/J7 V)

( r/J V REFERENCE)

-V (r/J.28 V) - ~n~---

Times (in1usecs.) are shown for Horizontal synchronization

Vertical Synchronization lasts for 1200 usecs. (including blanking) Voltage levels shown are valid for both ~orizontal and Vetical Synchronization

Figure 3.2 - EIA RS-330 Specification for Monochromatic Closed Circuit Television ·


specification. (Figure 3.'l). Both the synchronization pulses are extracted by a

voltage comparison ot the video signal to -u.3 v. A norizontal syncnronization

pulse is differentiated from a vertical synchronization pulse by testing the

duration of the pulse itselt. Upon the occurence of a synchronization pulse, a

monostable multivibrator is fired. The output of the one-shot is adjusted to

last tor approximately 1 O 1us. It the synch. pulse persists beyond tnat mterval,

it is steered to the vertical synchronization pulse output line, else it is

mterpreted as a horizontal synchronization pulse.

3.3 Conclusions

The enclosure designed served its purpose reasonably well. It's

ma1or deficiency is aesthetic. The circuits described worked reliably too.

Chapter 7 discusses the changes recommended if they were to be re-designed to

accomodate a faster camera. As noted earlier, the output ot the A/D converter

as well as the synchronization pulse stripper block is fed into a high speed

image processor which processes the digitized image. The next chapter

describes this processor.

3.5


Chapter IV : The Image Processor

4.0 Introduction

This chapter describes the high speed microprogrammable image

processor which was designed and implemented (9) for the FHMI. lt is used to

store digitized overlay-image picture elements derived from the A/D converter

in its image memory as well as to apply simple transtorms to large blocks of

data. lt's capabillties are outlined in the next section. The Image Processor co

exists with a 'host processor' on the host processor bus. The next section

explains the reasons for retaining the host instead of creating a more powerful

standalone image processor. The communications linK oetween the host and the

image processor is described later in this chapter. The last section describes

the microcode written tor this nardware and briefly describes the sottware

tools developed to test it.

4.1 Processor capabilities

The image processor is capable of transferring digitized pixels from

the A/D converter to a predesignated area in host memory. lt can do this m

'real time', meaning as fast as the video scanning occurs in the TV Camera. It

can also perform Boolean operations on blocks of data residing in nost memory

at rates considerably higher than those of which the host is capable. As well,

subtraction ot two frames is best done using the image processor. The dec1s1on

4.1


to incorporate only these capabillties from the many options possible was based

on the timing-requirement analysis outlined in Chapter 2 and repeated here: in

this appllcation, the video signal from the camera is digitized in 'real time',

with a L.J6 picture element tpixel) resolution per scan line. Assuming standard

video rates, the active view time per line is 52 1us. This implies pixels are

issued by the A/D converter at a rate approximately ot one pixel every LUO ns.

A general purpose computer would simply be unable to operate at these rates,

hence a faster 'logic block' capable ot transterring data into memory was

needed. It is implicit here that the logic block must have a time resolution

lclock) of atleast LUO ns. Also, the ability to specify the area to be d1g1tized m

the viewed field was deemed important. It was felt that this abihty would

facihtate analyzing only parts 01 the overlay tsuch as activity-intensive areas).

lt also reduces the size of the memory butter needed to store the frame.

Hence, some 'intelligence' in the 'frame graboer' was viewed as necessary.

Once a frame tor part thereof) has been stored in image memory, simple

transtorms need to be applled to it. An example of such a transtormat1on is the

subtraction of two images, which is an exclusive-or operation between them.

Assuming frame dimensions of :L.J6 pixelsxb4 pixels, this converts to an

execution time of 10 ;us./8 pixels, or 20 ms. on the host computer. The

optimum response time goal 01 Chapter 2 of )0 ms. allows only j4 ms. tor all

computation 06 ms. is needed for frame digitization at standard video rates).

An identical Boolean operation on a faster lLOO ns. cycle time) processor could

be accomplished in 3;us./8 pixels or, 6 ms. per frame. In general, such block

operation capabillties are the most time consuming. * Accordingly, the 1inal

implementation of the image processor included add, subtract, shift left or

right, logical 'and', 'or', exclusive 'or' and complement of input data. Further

4.2 * In the implemented version of the interface, the image processor was not

used to perform this task since the overall response time of the interface was found to be within acceptable limits. However, its existence gave us the option of a higher speed scheme if needed.


programmabihty in the image processor was felt to be counterproductive. This

is because extensive software capability is useable only with good sortware

tools such as an assembler, compller, debugger etc. which would then have to

be created. Consequently, the decision to retain a nost processor to snare the

computational load was reinforced. The image processor was designed with a

1.50 ns. cycle time objective (arbitrarily cnosen to be 50 ns. lesser tnan the

projected need). Figure 4-.0 illustrates the interconnection between all the

electronic components of the FHMI.

Several machine architectures l8) were considered with the DU ns.

microcycle time objective. lt was apparent tnat pipelining was was necessary

to achieve that speed. Timing analyses showed two pipeline stages to be

adequate. An instruction-data based pipeline scheme was cnosen.

The image processor was designed using Schottky TTL mainly since

STTL is su:tfic1ently fast and easily interfaced. The circuits used are members

of the 2900 senes. This choice was based on ease of use, famiharity and

recommendat10n by other memoers o:t the Computer Group at the Department

of Electrical engineering. A block diagram of the processor is provided in

Figure L/-.1. Appendix A calculates the minimum microcycle time achievable.

Over ·a U-5ooc temperature range, the value is 125 ns. Hence a clock of 150 ns.

is expected to result in error-:tree operation.

It was decided that no encoding was necessary in the

microinstruction format since the number of fields needed was llmited. A

single microinstruction is 4-0 bits wide. The data word size is 8 bits to match

4.3


I

\

HOST CPU

INTERFACE

IMAGE PROCESSOR

MEMORY 1/0

. ,..

\

,

.... ~ ---l.__C_A_M_E_R_A_-..1>

~igure 4.U - Block Diagram of System Components


,. ~

MUX BUFF-~ . ~ ERS ... ~ ~

+ SEQUENCER DEC H. ~ .. c A+ 1

1 r

' -CONTROL STORE

l(A + 1) 140 PIPELINE REG ... :..-

I I(A) I SYSTEM 1ir BUS

.... ...

TX REQ ~ ALU

SYNC+ I S(A) • -. TX ACK

STATUS REG

I SA-( 1)

Figure 4 .l - BJock Diagram of tlH~ image Processor


that of the host. A writeable control store was implemented for ease q_f

debugging the microcode. Also, the image processor execution-start adaress is

specifiable using the 'Address Register' in the address sequencer. This means

several tasks can reside simultaneously in the microstore.

The communications link with the host computer utilizes

programmed 1/0 as well as Direct Memory Access. The reason tor

implementing a DMA interface was to allow the fastest possible host memory

access. The A/D converter issues pixels at ZOO ns. per pixel. 8 pixels are

concatenated to form a byte, and therefore bytes are created at the rate of 1.6

1us. The host memory access time via OMA was measured to be 1.2 1us per

byte. Hence, the Image Processor can write digitized data bytes into the host

memory contmuously, i.e. in video 'real time'. This precludes the need tor

special high speed image buffers. The Motorola 6844 DMA controller chip has

been employed to perform the required bus functions correctly.

The Image Processor has to synchronize to the output of the A/D

converter which digitizes the video signal in real time, as well as the nost

computer clock. The latter is particularly critical during OMA since the data

transfer rate could be nearly at the maximum allowable bus rate. Smee pixels

are created as fast as l per 200 ns., it was decided to drive the image processor

and the A/0 converter with the same clock. This greatly reduces the

synchronization effort. The image processor now only has to wait for the

specified start point in the video :trame by monnoring the vertical and

horizontal sync pulses from the sync stripper. Then, it can correctly read a

pixel on every clock pulse. A:tter concatenating the pixels to :torm bytes, it

4.4


writes them into host memory via IJMA. Synchronization with the host

processor clock is achieved indirectly by a 'mailbox' data register which is

written into by the image processor and subsequently enabled by the OMA

controller onto the host processor bus. Handshake between the image processor

and the OMA controller occurs using a two flip-flop edge triggered circuit to

minimize chances of metastability.

The horizontal and vertical sync pulses from the sync stripper block

described in Chapter 3, are two of the four external 'cond1t1on codes' latcned

into the status register on every leading clock edge. The programmer can use

these lines to,1mplement a W AlT !or VSYNC = l (or s1m1lar test). The machme

will only advance the Program Counter when the test is true. Many variations

on this test are possible.

4.2 Processor Use and Software Support

In the version of microcode implemented, the image processor is

used to write digitized data bytes mto the host computer memory. lt would

have been possible to perform the next two most time intensive tasks, namely

reduction of noise in the picture and subtraction of two digitized frames could

have been done using the image processor too. However, this was not done

smce the time response 01 the overall mterface was found to be acceptable.

To reduce the difficulty of hand-assembling the microcode, an

assembler called SPASM, was designed and implemented. It takes an editor

created text file from a mass storage device and translates it into predefined

4'.5


microinstruction fields. It also formats the file into two versions: one is human

readable, and is dumped onto console, while the second is a machine readable

binary image. This image is compatible with images created by other system

utilities and tnerefore the system loader is used to load it before execution.

Another routine called PITS, (Programmable Interface Test System) - - - -

was created to help access the hardware in a convenient way. Testing of key

functions, such as OMA, Interrupt handling, and frame grabbmg were simpli:tied

by using dedicated functions in Pl TS.

4.4 Condusions

The image processor was built and tested. Some problems were

encountered and surmounted. Even though the image processor shared it's clock

with the A/D converter, the synchronization pulse signals themselves were

asynchronous. This is because the pixel clock is a tree running 5 MHz. clock

having no time relation to the horizontal or vertical sync pulses. It was decided

not to try to synchronize the pixel clock to these pulses for two reasons: the

first being that most inexpensive video cameras tsuch as the one used) generate

sync pulses usmg free running RC oscillators. As a result, these pulses undergo

considerable' frequency variation. The second reason is that in low cost

systems, mere is no synchronizat10n oetween the norizontal and vertical sync

pulses themselves from the camera, so that even if we were to lock onto one

type of pulse, the exact time of occurence of the second would not be

predictable. The result of this scheme unfortunately, is a pixel position

ambiguity of +Y2 pixel. The software noise-cleaning and thmnmg algorithm

f./..6


described in the next Chapter performs corrections on the image to compensate

for this effect.

Another problem encountered was that of glitches in the

synchronization pulses. These are up to 50 ns. wide and occur aue to v1aeo level

shifts on the analog output of the camera. The microcode running on the image

processor per!orms de-glitching by testing the state of pulses !or a minimum

duration of 5 /US before executing the appropriate task.


Chapter V : Algorithms and Software Strategies

5.0 Introduction

This chapter outlines the software tasks to be performed and details

the algorithms and strategies adopted.

The first task at hand is that of picture digitization. A binary

digitizer has been implemented; however the decision threshold is set by

software. Three binary thresholding techniques were considered. They are

detailed in 5.1. Arter setting the threshold, the image processor is directed to

digitize a frame. On completion, the picture butfer contains a digitized image

of the scene. This would contain outlines of the keys, and possibly, blobs

corresponding to fingers pressed on the overlay. The image is subtracted from

one containing only outlines of keys that was taKen immediately atter 'power

up'. The result is a picture containing only blobs corresponding to fingers.

However due to digitization errors and subtle ambient light variations, the

subtraction is rarely exact. Many small spots teach 1-2 pixels wide/high) are

created on the edges of key outlmes that were subtracted out. Hence, a noise

reduction routine is imperative. Section 5.2 lists popular techniques as well as

the reasons tor choosing the one that was selected. Arter noise reduction, the

fmger blob has to be reduced to a single representative point. Section 5.2

shows how this has been combined with the noise reauction to reauce

computation time. Section 5.3 details how it is determined which key encloses

5.1


this point and the last section explains the ASCH code search and output

process.

5.l Image thresholding and storage

Three major image thresholding techniques were considered (10):

lilobal thresholding, Bounded thresholding and Local thresholding. For lilobal

thresholding, the black-white decision point is held constant over the entire

frame. This leads to computationally erficient post-processing but lacks

sensitivity in low contrast regions of the frame. Bounded thresholding is a

natural extension of the global technique. All pixels between preselected

brightness levels are assigned to be the complement of the background colour.

Thus some degree or riltering is built mto the algorithm. The result is some

improvement in sensitivity but requires two digitizations and a Boolean frame

operation. For Local thresholding, the frame is divided mto regions with the

threshold varying across regions if needed. The threshold of a region is often

determined by its average brightness. Within regions, the digitization can be

global or even bounded. This leads to excellent sensitivity but unfortunately,

very complicated post processing and data structures.

The global technique was selected for its simplicity. It was found

experimentally that the viewed field brightness of the keyboard did not change

appreciably from point to point.

The digitized pixels were concatenated to form bytes and stored in a

frame bu!fer by the image processor. An alternative here was to 'run-length'

5.2


encode the image prior to storage. Run-length encoding (11) consists of storing

the position of colour transitions on every scan line instead of the digitized

.image itself. This technique proves efficient from a storage point of view for

relatively sparse images only: to store a binary image :L56 pixels x 64 Imes

requires 2K bytes. In the same storage space, 2K transition addresses could be

stored. This implies space for 32 transition events per scan Hne, or

equivalently, only 8 key outlines. Hence it was deemed unsuitable for the

application and the decision was made to store unmodified bmary images are

storea in the frame buff er.

5.2 Noise reduction and thinning

General filtering algorithms usually require that a n x m matrix be

scanned over the raster once or more. Typically, n and m are between 2 and 5

(12). The problem with these algorithms is mainly speed. Performance-wise,

they are usually effective for noisy greyscale images. Smee speed is a critical

factor in the selection of an algorithm, a simpler and much faster technique

was designed using a matrix of size l x 8. A ma1or speed gain is accrued smce

only one complete raster access is needed. An additional level of simplification

has oeen introduced: the well detmed (elliptical) nature of the blobs being

analysed allows the matrix scanning to take place at exact byte boundaries,

obviating the need to address pixels within image bytes. Thus the number of

image memory accesses is reduced significantly. In general problems with this

technique are its insensitivity to very thin vertical lines (less than 2 pixels wide)

and also to objects spanning byte boundaries which might get inadvertently

removed if they were less than 3 pixels wide. Experiments nave been

5.3


performed to determine the effectiveness of the algorithm in this application.

Details appear in chapter 6.

The thinning process is similar to noise reduction, and has been

combined with it. The frame buffer is scanned to cneck tor norizontal lmes.

When encountered, they are replaced by single pixels at the centre of the line.

Hence a blob, which is essentially a set of norizontal lines, is reduced to an

approximately vertical line. This line is further reduced to a single centroid

point by another pass over the raster, this time cnecking for vertical Imes.

Hence, on exit from this process, all blobs on the raster are thinned down to

single pixels close to, or exactly on, their centroids. This 'centroid' intormation

is stored in a CG file for subsequent use by the tracking and code output

subroutines.

5.3 Determination of encircling keys ltracking)

The next task logically is to determine which key encircles each

finger blob centroid. A solution can be attempted by tnree general tech~iques:

connectivity analysis, blob tracking, and finger-key centroid linking.

Connectivity Analysis of the image is a general blob analysis

technique which accumulates blob statistics over a single raster scan 03). The

statistics include hierarchical and sequential blob links, e.g. 'parent', 'child',

'sibling' blob pointers as well as 'up', 'down', 'left' and 'right' pointers. If the

overlay image (without any fingers) were analyzed once, tnen keys encircling

finger blob centroids could easily be determined by manipulating the parent

5.4


pointers. The disadvantage of this method is computation time and immense

storage requirements.

Blob Tracking is another general blob analysis technique which does

not utilize complete raster passes U3J. Starting at the finger centroid, the

overlay image is scanned until a blob is encountered. Then the algorithm

attempts to 'track' the outline and tests to see it it closes on itsel! and around

the centroid. The disadvantage of this method is of computation time since the

frame buffer must appear to be random-pixel accessible.

Finger Centroid-Key Centre Linking (FCKCL) is a new technique

designed for this application. During the power-up and initialization phase, an

image of the overlay only (without finger blobs) is stored in a buffer. During

this time, the user is asked to define each key by specifying the code to be

output when it is found pressed later during use. To point to the key itself, the

user places a finger within the key and proceeds with the definition. Since the

finger is known to be within the key, it's centroid is stored in a key attribute

tile. During normal operation, the attribute file is accessed sequentially. The

line containing the finger centroid coordinates and the key "centroid"

coordinates is tested to see if it contains any dark pixels. I! it does not, then

there is an uninterrupted path between the finger and the key attribute. This

implies that the finger must lie within that key.

Of the three techniques described, Connectivity Analysis is the most

powerful since it calculates extensive blob statistics. However, the FCKCL

technique was used because it was the most economical in terms of

5.5


computation time and storage requirements. Within it, the test of the line

containing the finger centroid and key "centroid'' coordinates results in features

that are very useful in this application. The test can be modified to

accomodate open key shapes on the overlay - i.e. shapes not bounded on all

sides. This is done by aborting the test if the line length exceeds a threshold.

This artificially introduces a boundary around the key centroid, so that the

outline itself could be as incomplete as desired, or maybe even non-existent!

The second feature is programmable depressed-key search time. By controlling

the range of line shapes and lengths allowed in the test, the maximum time

taken to determine whether or not a key is pressed, can be controlled. A

straight line with a predetermined line length limit implies that the absolute

maximum time to determine wnetner the key is pressed equals the time to test

if that line is all white. Allowing curved and/or longer lines implies a greater

search time per key but has a higher probability of fmding a depressed one.

However used, this ability to limit the search is invaluable in ensuring a

consistent human machine interface response time.*

In the implemented system, the test line shape was straight, a~d the

line length was llmited to the maximum key size on the overlay.

5.4 Code search and output

Once encircling key attributes are determined, the appropriate

ASCII codes have to be transmitted. The attribute forms a part of an ASCII

code file that was generated during the initialization/definition phase. A single

record in this tile is lJ. bytes long: the first two are the key attribute, and the

* 5.6 For a 256 x 64 pixel resolution, an alternative technique would be to map every pixel to a coordinate number. Thus, connected pixels could be determined easily. However, the technique rapidly becomes impractical for larger spatial resolutions.


next two correspond to a 15 bit address pointer indicating either a location to

which control can be transferred to, or the ASCII word to be transmitted

whenever the key is pressed. This list is searched until the key in question is

found and then the ASCH code is transmitted. This is repeated tor the entire

depressed-key list. This action complete one system cycle allowing another

frame acquisition to begin.

5.5 Condusions

The interface operation trom a software standpoint is summarized

here in order that the algorithms used may be reviewed. For picture

digitization, a single threshold scheme was selected due to its simplicity and

low storage requirement. After digitization, the image is subtracted from that

of the overlay alone resulting in an image of only blobs corresponding to

fingers. Noise due to inaccurate picture registration is removed by scanning a 1

x 8 matrix over the raster. This operation also thins the blobs. The result is a

picture buff er containing only vertical lines, one line per blob. These Imes are

compressed to find the centroids of the fingers pressing the overlay. A Finger

Centroid-Key Centre Linking technique is used to determine which keys

encircle the fingers. This information is used to output codes corresponding to

the keys being 'pressed', thus completing the cycle.

The algorithms worked well without exception and were easy to

implement.

5.7


Chapter VI : System Integration

6.0 Introduction

In this chapter, all the components of the system just described are

brought together to form the Flexible Human Machme Interface. Section 6.1

outlines the steps involved in useage the use the prototype system. Section 6.2

describes 'spatial resolution' tests performed on the interface, while o.3

describes 'time performance' tests performed. The results of these tests are

important because they provide insight mto the creation of useable and

effective overlays as well as into the behavior of the interface under use. The

last section describes typical applications of the mtertace.

6.1 Use of the prototype system

Use of the system involves three stages: creation of an overlay,

association of codes with keys drawn on the overlay, and tmally actual data

input. An overlay is simply a drawing of key outlines on any transparent sheet.

When this overlay is placed on the glass sheet top of the mterface, it

automatically forms the working surface, i.e. the keyboard. The true flexibility

of the mterface accrues from the ability to draw keys of virtually any closed

shape or size as well as the ability to change overlays rapidly at will. All

shapes on an overlay, whether positive or negative, must adhere to the

thickness constraints described in the following section.

6.1


The definition phase consists of placing a finger within any closed

outline on the overlay intended to be used as a key, and subsequently

associating a code with the outline. This is done by :tirst placing a finger on it.

When the gesture is 'seen', the system acknowledges it by issuing a prompt to

the system console. At this time, any key on the system console can be hit

causing that code to be associated with the overlay outlme just pressed. This

process is repeated for all outlines on the overlay intended to be used.

Alternatively, a previously entered 'code definition file' could be retrieved Irom

a mass storage unit such as a floppy disk. Hence frequently used overlays need

not be repeatedly defined.

Use of the system can commence once the definition is complete.

The user has to place a finger within any key outline on the overlay to cause the

associated code to be transmitted. Keyboard characteristics such as 'auto

repeat' (repeated code output while the key is 'held down') or 'once-only',

multiple key rollover etc. are specified during the definition phase. The next

two sections describe the experiments that were performed on the implemented

system to determine various response-related parameters.

6.2 Spatial Resolution testing

The term 'Spatial Resolution' refers to the resolution of the

mterface when processing outlines of shapes drawn on an overlay. This section

describes tests that were performed to determine the minimum key outline

thickness necessary on an overlay for successful operation. The minimum

thickness of the space within keys as well as between keys was also determined.

6.2


Two other parameters were measured: the first was the minimum distance

between a finger and the overlay that would not cause a key on the overlay to

get triggered and the second was a value for the working black-white

digitization threshold.

The testing was done in a small room wherein the ambient light

conditions could be controlled. For inclusion in this thesis, an image-access

and-print routine was added to the interface testing software. It was used to

create the images that follow. These images are of a test overlay and were

printed using an 80 column dot matrix printer. Due to paper size constraints of

8Y2 x 11 inches, only one half of the overlay printout could be included here.

Recognizing this, all testing was done on that side of the overlay. All outlines

* on the test overlay were 3-lj. mm. thick.

There are two distinct phases of the interface that contribute to its

spatial resolution. The first is the image acquisition phase wnose output is a

binary image of the overlay. A well contrasted image could lead to easy

recognition and processing of the overlay outlines. Consequently, tne outllnes

need not be as thick as they would have to be with a poorly defined image.

Results o:t this phase are affected by the brightness of the image as well as the

black-white decision taken on the video signal. Two tests were performed to

test how sensitive the interface was to these aspects: a) ambient light

sensitivity and, b) digitization threshold sensitivity.

a) Ambient light sensitivity: One of the factors that affects the

interface operation is the ambient light intensity. This is apparent immediately

* 6.3 All diagrams within this chapter are 'reverse images'-i.e., light outlines on a dark background. Also, due to image reduction constraints, the horizontal scans of the camera appear as columns in the diagrams, not rows.


when it is recalled that the image acquisition takes place on an image seen

through a plastic film whose transmission characteristics are a non-linear

function of the incident light. The latter also implies that the brightness of the

viewed image cannot be expected to mcrease lmearly with mcreased mcident

light. Figures 6.0, 6.1 and 6.2 are copies of printouts taken with a 100-watt

incandescent lamp shining directly on the keyboard, normal overhead lighting

and finally, all overhead light switched off. The features of interest are the

thickness of the key outlines and the contiguity of the space within the outlme.

Figure 6.2 was created under very low ambient light conditions.

When compared with Figure 6.1, the average mcrease in outline thickness is

seen to be l pixel. This comparison establishes the minimum space between

mner edges of outlines !orming a key. Smee with outlines thic1<ened by low

ambient light, the reduced space must equal the minimum acceptable by the

blob reduction algorithm (2 pixels). The reduced space can be computed to be

equal to l (Left outline thickness increase) + l (Right outline thickness

increase) + 2 (Minimum width allowed by blob reduction algorithm) = 4 pixels.

When divided into the physical width of the overlay, this requirement implies a

spacing of 6.2 mm. lt may be prudent to remind the reader that the lighting

conditions under which this test was performed are definitely not ones under

which a user is likely to work.

Figure 6.0 was taken under high ambient light conditions. The result

is an apparent decrease in outline thickness of about l pixel when compared

with Figure 6.1 (taken under 'normal' light conditions). This comparison

establishes the minimum key outline thickness since the reduced outline

6.4


•••••••••••••• •••••••••••••• • • ............. • • • •••••••• •••••• aanaaaaa•••• ~••• ••• 14 """ ii•••na#•••u.nu•• ••n •• •• aa•aaaaaaa••a aaaaaaaaaaaaaaaa aaaa#1;a•ttJ1J•••a'taaauaaaMa&aa11·1

••••••••••••• •••••••••••••••• •••••••••••••• •••n,.·r•n••••• ••••••••••••• ••••••••••••• ••••••••••saa aaaan11awaaaaa

• ••••••••••••••• ·········••tt••• ••••••••••••••• aaaaaaaaaauaaaa

•••••u•••••••• au!1aauu••••ua

···········~·· ···~···~"·~·· aaaaa'taaaaaaa1a aU111ta•••··~ ::a

••••••••••••• ••••a•••••••• ••••••••••••• ••••••••••••• ••••••••••••• ••••••••••••• ••••••••••••• .•..........•

••••••••••••••• ••••••••••••••• • •••••••••••••• • •••••••••••••• • ••••••••••••••• • ••••••••••••••• • ••••••••••••••• •••••••••••••••• ••••••••••••••••

....•....•.... ····~~······· •••••••••••••• •• ~••~•irii•an• •••••••••••=•• ••••uau•••••• •••••••••••••• aii:n•••••••nH1 •••4••• ••••uaa aJ:: :a a ;i ¥a<i w •••• • ••••• ,, ..•••••• ··········••it.• •••••••••••••:r ••••••au.:>taaaa aaaaanaauaaaaa asaaa•aa•#aHaa

••••••••••••• ••••••••••••• •••••••••aauaa •••••••••••••• •••••••••••••• an••••••••~•• • • • • •••••••••••••••• • • • ••••••••••••••••••••••••••••••••••••

••••••••••••••••••••••••••••••••••••• ••••••a•••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• •••••••••••••••••••••••••anaaaaaaaaaa ••••••••••••••••••••••••••••••••••••• .......................... # ••••••••••

·····························~······· ••••••••••••••••••••••••••••••••••••• aaaaaaaaaaaaaaaaaaaaaaaaaaaanaaaaaaa

•••••••••••••••••••••••••••••••••••• ..................................... ••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• .................................. ••••••••••••••••••••••••••••• ····················-·· ··-

• • • • • • •••ii• ;s::= a•••• ii• :a•., •••u• •••••aim•• If :ta••••••• i: •• •••••na#•••••~U••~•••••

•••••••tt•••1t••••••••ll•• aauaa •••aaa•aaa~#aQaPa •• •• !t•••••••• u 1a

•....• •#••·····"··~ ••••••''" •......... ~······=···· ....•.• •••••••n••a~• aaa~••

•.••.......•• ~····· aft•••••••••»• ••H 11aa •...•....•. ". . ..... •••••••n•attna

•••a•••.:t•••• ••• • •

····=·· • ••••• aaaanaa

• •••••••a••••• ·················••ti .• •••it:a.ii.:••••••#a:•ttlf:a••• ••••Jt•••••:ii:sia••••au::aa

••••••••••••••••••••• a ••••••••••• ••••u•••~•••••»••*••••• •••••••••••••••••••• •••••n•••••••• anaaaan•••tt:iaaaweaiaau aa aaaaa#aft•aaaaaaan:aaaa aaa u:aaaaaaaaaa aaaa<L•a: :s:uaaa :tlJ#WIJ•#aaaa •••••••••••tt••••••••• ••••••••••••••• •••••••••••it••••••••uaa ••••••••n•••••••••••••••••it ••••••••••• ••••=•1auui.iau.1aaUlf•••1t# •••••••••••••••••••• •••••••••••••••• •••••••1t••w:~aaaauaaa1;aa .....•.............. •...........•........ •••••••••••••• aaaaan•u•U••=;J;•••••••••

•••••••••• ••••••••••••ttu•••••••w••• ••••••••••••••••••••••• ••••••••••••••••••••••••• ••••••••u•••••••••••••••••• •••• •••••••••.:••••##a:; ·inaa;;a••••

aaaa'taaaaaaanoiaatt!tatla##&aai:aa aaaaaa##aa.aaaaaaaaa###&S###•#n

••••••••••••••••••••.tt•••••••••••••••••tt=••••••ttn••u !tani.11:x11aa" •• ••••••"=••••••••••••••••••r.•••••••aa.;unaaaaa.:•u••••••#•a=u•aaa# aaaaaaaaaaaaaaaaaaaa•aaaaaaa1t'>aaaaaaaattWaaaaa##.1CSai;rir2f#ltA.tiaM#•8 ••••••••••••••••••••••••••"•••••••••=•••••••••••••n•••••••••••~

Figure 6.0 - Overlay image taken 1.11der bright ambient light. Note the generally thin (and even incomplete) outline thiclcne:sses compared with printouts taken under higher ambient light (the next two figures).

----1 I

••••••n•••••••• aaa11aaa•••••••• •

• •••••••••••••• ............... •••••••••••••• •tt•n••••••••••

. ............. . ••··••·····•··• ••••:i•••••••r.aa

• ·············• ·············· • • .......... ,,.

············ ······s;••=·~·

·····=······· ····~, ......... . ··············· ··········••ii·· ··············· ·········•••it••

•·············· aaa:naaaaaaa:s ••

····=·········· ·············· ••••ic•a•:saaua ······•····· ······•••11 '1• ............. aaaw11 .. 11••••1taa aaaasaaaaaaaaa ············• •••a:u1aiiaa•::a •••••••snUl"f#

·········••:i•• •••••?S•••it•••• •·············· aaa;r:aa:aaaaaa;:a• ············· •••iiria••••••• ··•·····•··· ••••••••••a•

••-•tt•u::t•••?i•• ••••••••••ti• !ta aaaUl:!!1tnl!!t1l~=ta ··············· ·····-···••ti•• ...............

············· ••••H•••••a;i:a ············ ············ .............. ••••!1a11••••••• ...............

············· ••••:t•••••••u. ........•.. , .. .......•..... ••••u•••••••=•• . ............. . ··•·········· ••••:i••it••••• ········••ti•• ............. • ••••••••••••• •••••tuiaattauaa ·•··········· • ••••••••••••• ••ll ••••• .. .. ••••.a••······· •••••-=u=•:a::aaa•••••• aaa a •• a

····=································· •••aa::aatts111taaaaaaaitaaa•••••:-.aa;s••••••

····~:••······························· ••••u•tt••••waa;i•t: ••!tn•a•••••• :a:::aaaaaar.. •••*•••••=~•eua••••••••••••••••••••••• aaaaa;~a::u~aaaaaaaauaaaaa:ar.:aauaaaa~

aaaasaa••••••••••••••••••••••••••••••• •••••~•~s:;2aaa•••••=••••••a~2••••••• •................................... •••••••i:t'l! tt" ·1 U••••••=••••••wu:r:;a=••••• aaaa:ta•s••••••••••••••••••••••••••••• aaaaa~:waaauaaaaaaaaaaaaaaa::~aaauaaa

····~································ ••••iu;a;:;s;::;aaaaaaa;:11aaaaaaa w •••••••• aa•at: Utauaaaaaaaaaaaaaaaaaaaaaaaaaaaa ••••H :: •tt• !t••••••=•••uaa•••••= ..:aoi••••• ····~························~····· aua:uaaaaaa:a:waaa:~u••••=••u

···~~····#···~•«••········ ••

•• .. .. . ... ....................... ·=···••t•••tt•••••=•=.n .•• ••··•···•·····•·•·•··• ••••••••••••~•••••n~•• ... ·••···•····••• ······•••li••• ········ ·········• ••••••: ... ............

•tt•••.au;::aa ·a

•·••·•·····•· •••aana:i za~·. =• ·•·•••······· •ts•••••t:•••• • •••••

····· •11 .•• ····· •tt••• ····· anuaa • •••• aaaa#rtsiaa . .•..............

·-•:-•= =·= ==·····=· '!•• ....................... aaa .. u•;s••••••••••••t:.tta •••••••• attaaaaat:aai. =••••:r••nnaa •••••••w••••••••••••• aaaauaa•••••• ••••••••••••••••••••••• ••••u:1 ;r11 1!!!•iaaaua1ur:i:u •••au ••••••••11 anaaa:a.r•••••aua•ttn11·i•a ·····~·"············· •.•........••. . ..........•......•.... •••=·:u:;: ~. :s:: a ·1 aaaaaaaittla aaa:;a-:. .i aa •••••# ··=··=···#··········· ······~········ •••• :·====:a•••

••••••n••= 2zuu1::;::z;.a:::·aa ...................... aaaaa:•s•:.- :axattaatt1 ·•na ···#===·~-~ :••••••tt~•

•••••tt•••~n•••••••••• •••::;:;Q:..•n 11 ·;aaaaanani:•••

........ • ••• ··············•••#•••··· anaase •a-.n:ur: ••tr 1• att ttfl "11 '"tf; ·nia

~···~···················· .•.................•....•...

•••~u1:~;=1 1111·1aa••••••U•#••tt• ••aau,,ua;1•••aan••t'~•n"a•1•n•r••

•••••:•••"••••••••••••••••••••n•a••••••••••••••••••••••••••••••• ##Ut!:i =-•Z:=••=•aaaaaa::t•••••••· . .:a.::;.uaaa~ ~=iu:raaa:ut"'l ••n ::•=•:i:uraw::aa

uaaa::aan••••••••••••••••••••••••••••••••••••••••••••••••••••••• =••••=*•••••::s:;;:;s.•••ui.aaaaaaaz: :a•••••aa:.aai:aaaaacaais 1•==tt=••nt;aa

Figure 6.1 -Overlay image taken under normal overhead light. Note that the outline thicknesses have increased, and all key :ihapes are now dosed. The latter is a prerequisite for suc:cesful interface operation.

••••••••••••• aaaaaaasaaaa ........ . .. .............. •.........

••••:•••····· ... . ... "

•••••••••••• •••••••••••• •••••••••••• •••••ti•••••• •...........

·········~···· ·····~······· •••••••••••••• aaaaau~••••••

"II '4'1Jau:aaaan a.irnruia11a••• :''It: a••••••u aaa:usa;;nau::aa aaaaaaaaaaaaaa aaB#1S#~•#a#•S ••............ . ••.••....•.. tt -t;r;1•••••••• :i.:u;:r:i1i:::i11:ita• 1: ta•••u•••··a =!;2pr;::us.:s:ai1:~•

• •.•• !' ••••t: , .. l aaaaa.aa:iunaa it ·upiaaaaaa·11 aaa:i:ta•1••••• =-~····••••1:• ...... -.aittiaana

aaaatuu11.1•••• •••••••••••• .............

;iaaa•••••••••• aa:in.ttsu:•••••• ••......•..... ·····~······· ............... IUlUtf##a•x•••• ............. -············ ••tt ... !lrflttl!t••• •••••••••••• •n••••••••=••• •••••#•••••••

•••••••••••• •••••••••••••• aaaar••••••• •••••••••••• ...•.••....... . ..•........ •••••••••••• ••••••••••••

.............. ·····~······· ..•...•....... •············ •••••••••••• ••·•·········· ····~······~

uiix::s••••••~• ........ . .. •••••••••••• •••......•.... ············ •••••••••••• • • ••••••••••••• .................................. •••••••~•••••••••••••••••aux•••••••• ••.••...............•............... •••••••~••••••••n••••••••••••••••••• •••••••••••••••••••••••••••••••••••• aaaa••••waaaaaaaaaaaaaaaaa1~u••••••••

···········=························ ··············••1t••••••••t1 •:i:t•tt:•••••• ••••••••••••••••••an••••••••=•••••• •••••••n•••••••••••••••••#•••=••#•• .................... ~··············· aaaatia•:i.•••••••a•1aua••••••:~ -iat! ••uaaa

···········~·················~~····· •••••= • ·i=••••••ic :1 ••••••••ir aaa-n•••••• •.•••..............•....•........... ·······~··················~········· ·····························~····· •••••nauaaaaaaauaaa••••••••••• •• ••••••••••••••••••••••••••• aaaaa##Uaaaaaaa•&aaaaa .....•......•...... #&••••·=··········· •••••••••••s••••••• ••••~::11~c••••••••••

• • •• an=•••it:••

au :n•:ia#~:••• ••••••.sJ•••••• a#ii ;a.;a:a-s;a•;a•••

• •••n •••~• ••••• ••• ••••••==~•====•:=•#•n~~ ••••••r r•;,r ·:111wnan-n1111ra ••••.n•=••••n-:uia:q; nuww• aa#S CttaP!ltt•iftt•~~~,,~W

••::;;u•a•:•••••• ••• •••"••n1;1•

a •=••• n.na:::::::naa1! ••••••n-'t:t•ll #tl:llllf

~-~~·t•:·~-#~· :1"Qlt~

••••11 1·~··t:1P · 1u1·•at1 iu11:;;··-;;:: :·~;s## tSIUft!lt

aaa••Ur"Ul&I, Qllll ::I

u••~~-~F#tt:# a•~:1a

a•••••h••• ai;-:,:;. • •• .;:1an•••

n• »•••••!{•• ••:a.zirt#:s::::::ui:i=•••••

••••U~ft~M~·J=a11~•~»r••

~-~-~~~~:::u~~~~~•••#••••

aaaasn:. ••~:. ·~1"n•••~:~• ::;u::;: •. -; :::-::::;-:i.ot~Jt#l"I'#••••

aa:.is .... ;. . .i: . iii::aww;; ·:;a ••••:.c~===~~=~~,~n~•~~!l

•••••••••••=••••••• •••••• ""••••• ••a••••!l:t::Sr. ,i;·i:; ·; ;;;, : 1; ·r:i;: •••••u•~••#••~••••• aausaanaaaaaa =•~~=~==::::::i 4~==~n•••

•••••••••as:.11••••••• •••-=•••it•• ir~ua.c·: ·11111: ·1:1;:e,n·· -:~ •••••.si !::' ·1:uc::!ta:tun ::aaa aa:e:•••••• n.#••:i:;.aa::::c::r.==i::i::i:i:;; ::.aur:I n ••••••••••• >:i•••••••u• a •••••.ttu• flit i; 111• 1; :: ·t:t:tt;i;;z i! :;·;i; ••••ttnau1 •••••••• .; :ui.a••••• air•••••••• :i.:t•.: ·:~ ~:t!::t!;<;>:::;.; ;:.aan ~ aaaaaa.aa•aa.n••••••••••••= a:raaaa:i••••••#r! 1a!1ot· :l'l»#•~:nttt··io;:i

•••tt• tt• ••••••uaa11 • ••••• 't:t.· .. . 1 ~:u:.i;i •• u•: .i1· ;1 A; itiis; ·1 tr : ::::t ·i ·; : i,:i. n :J#IUf • aaaaaaaaaaa:zaaaaaaaaaaaa•:uu:::i:;!aat:##a••~'a:::a:w.JS••" 1;~:· •:'l•n•Jra.l: :;;i

uaaa•u•a~aaAa#aaa•a•••••••~~•tc•tt&##l~~•=#~••~· =;;~~r~,~~-~·i••nq

•••••••••••••••••••=•••••••••Jt•aauaaaa:o:ao;s:::aaai:a::u :i::it :in•un" ~• ••••••••••••••••K~•tt•••••u~ua·;aa#a••~·~~~~;~ ::;.· ;~~ ;uP~aa !~••••

Fig1re 6.2 - Overlay image taken under low ambient light. Note. t!": increased ~~~~-t!Ji:~:- Inaeased thicknesses reduce interface :sens1tiv1ty as


thickness must be 2 pixels, single pixels being removed during noise reduction.

The minimum drawing-outline thickness can be computed to be equal to 1

{decrease in thickness due to increase in ambient light) + 2 (Minimum thickness

needed by the noise reduction algorithm) = 3 pixels. When divided mto the

physical width of the overlay, this requirement implies a minimum line width of

f.1..6 mm. These results are tabulated below:

Operating Light cond1t1ons

Very low Normal bright

Minimum key drawing outline thickness 3.1 mm

Minimum inter-outline space 6.2 mm.

(within keys)

3.1 mm.

3.1 mm.

4.6 mm.

3.1 mm.

These results can be used as rules when creating overlays. If the

system is going to be used in a well-lit room, the numbers in the centre column

should be used. However, an overlay which would be useable in extreme

circumstances can be created by using the bold-faced numbers.

b) Digitization threshold sensitivity: The video signal is digitized as

black or white using a single threshold. The thre.shold itself is set by the host

computer with a resolution of l part in 256. This subsection contains prmtouts

showing the effect of variation of this threshold. Based on these results, a

working value is determmed.

Although printouts were taken at nearly all 256 levels, only four

6.5


significant ones are included here. Figures 6.3 - 6.6 show images of the overlay

taken at thresholds of 6, 15, 32 and 39 respectively. The features of interest

are the thickness and completeness of outlines on the overlay. A reference

level range of 15 to 32 was found to create the best outlmes. Below this range

the result is incomplete outlines, and above the result is excessively thick lines.

(An excessively thick line is one that is more than 2 pixels thicker than

expected from theoretical calculations). Hence this range was adopted as the

working reference range. lt was found that the per!ormance of the mterface

was relatively invariant within this range. However, the black-white decision

level can be changed frequently if ambient cond1t1ons are variable. This

feature was not implemented in the prototype because the working conditions

were fairly stable.

The other important phase is the blob detection and reduction

process. It will be recalled that the ability to accurately compute the centroid

of a finger blob within the binary image is a prerequisite to the determination

of 'pressed' keys, and therefore, the operation of the mterface. The centroid

computation process can be expected to work well only within the constraints

of a minimum finger blob size and of a minimum separation between two

fingers simultaneously placed on the overlay. In terms of overlay-creation,

these constraints contribute to the spatial resolution of the FHMI. Two tests

were performed to study these characteristics: a) Finger - Overlay Proximity

Sensitivity and b) Finger - Finger Separation Sensitivity.

a) Finger - Overlay Proximity Threshold: An important response

related factor of the mterface is how close a finger can be to the overlay

6.6


••••••••••• 4•• ·········=··· ••••••••••••••••••••••••••••••••••••••ti••••••=n•••~••••••s••~•R

••••••••~•••• ••••••••••••n•••• ••••••••••••••••••••u=#•~~••••• aaaaaaaasaa#a aaaaaaaaaaaaaaaaa aasaaattaaaaaaaaaaatr naawaaata J< ;: •a aaaaa~sa•aaaa aaaaaaaaaa#aaaaaa aaaaaa#•aaaaaaa aaau~aaR#a#•h

•••••••••••is:••••••••••••••••• aatta••'l•••••unu ••:s••••s:••wnll •••••a••••••• •••••••••••••••• ••••••••••••••• ••~•~••~=•#•• •••••••••••u•••••••u•••••••••• ••••••••••••••.n x11nffu•u••••n• ••••M••••••••••••••••••••••••• •••~••••••••••~ a#####•=uaaaa •••••••••••••••••••••••••••••• ••••••n •••••••• au 'taa#a.a• •tt ;: • •••••••••••••••••••••••••••••• aaaaaau#aaaauau aaasa#awttaaaa ••••••••••••••••••••••••••••••• •••••a1:1.ta•••a~u 1ttaa1an11••uu• •••••u••••••••••••••••••••U•••• ••••••••••••••= ••aa#aa#••••• ••••••••••••••••••••••••••••••• ••••••••••••••• wr;~~~ttna•nan• aaaaaaauxaaaaaaan•••••••••~•••• •••"••••••••••• aa"tt••••••••••• •••••••••••••••••••••-•••••••• ••••••••••••••• aa#·1n11ttnaaawuaa •........•... .•........•..••• ............. •••••••••:••• ••••••••••••• • • •••••••••••••••••••• • • • • • •••••••••••••••••••••••••••••••••••• ••••••it •••'1 !. •tiruaa1tn11 :: na •••..•••.••.••.................•...... ·····#········#·······~· ••••••••••••••••••••••••••••••••••••• ••••••••t1••.._ tiaauiaa•tt• aaa .. ••••••••••••••••••••••••••••••••••• •••••#a#nnauaaw#an~nnann

••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••• ..................................... ••••••••••••••••••••••••••••••••••••• ···························••tt•••···· •••••••••••••••••••••••••••••••••••• •••••••••••••••••••••••••••••••••••• ··························~·········· ••••••••••••••••••••••••••••••••••••• ·························•:i••••••*••• aaaaaaaaaaaaaaaaaaaaaaaaaaaaauaaaaaaa

·····························~······· ••••••••••••••••••••••••••••auaaaaaaa ················#················· •••••••••••••••••••••••••••• ••••••••••••••••••••••• •••••

aaaaaaattMaatl!lUIW•tt•ttatA•a ••• aanaaaaaaw.aan<r

••n~n ••••~nnau11aa

••••=••••• n;;aa :••••i< ts aaaaau M :ol•tt :1 :t•••• ·: •• aa#~anan•aaaaa ##a~~n

•••••:~u•n:•••~·tx ••••a:tn;auwaaaa

ttUl!Ulttl

atHUUt# aaaaaaa11:1nnr1::u awu1;ar. ~an••a•••••••• ~#••~11

••••••ial!·iua • •••<"•• ••••••• a ••••n••

• JUt•w n•••1' I' 't•

•~•"••••••~#aq#ttwuua

••••••1111ttau1t••~••••a•1••

•••••n••n••••uiiuHan•••n• ••..•••••.•••••••.••. ••••...•••.. ·········•••it••······•: :1tt ••••a •u•••••••••••••• aaw••••••••••• •••••u•n•• a" iiaJ· • a;Ht" aauti

••••••••••••••••••••••••••••••••••••• ••••••••tr••••••••!:•• au •u ••••••••u•••••••••••••••••#••••••••••• aaaaasa#aauuis u;:;; •~:;nu.aan?t •••••••••••n•••••••••••••••••••••n•••• •••••••••••:st :.an•tus»tl u 11 aa •••••••••••••••••••••••••• ;iu•••••••••• ••••#a••a####:'Or.a.:q SS:•••• •••••••••••••••••••• ••••••r ••••••• ••••••• ••• n ;1aua ttnau#:; 11tsa aaaaaa#aa••••••••••••• •••••••••• •••••••aa;11:aasaa:t#a::u;;.anan •••••••••••••••••••••••• ••••• ••••••••••••••an!~•••••••:w;:aw

•••••••u•••••••••••••••••• ••••••••••••••••••••••••••••

·····••#•••=•#••••#=.::••"· ..... 1, ••••••u••••~••a11

11••~·r•••auvau!tnw

•••••••11•••••••••••••••••'*U••••••••11 wnJSiusuuau•w••aa.i;;:i tt#a;i11 riaaatn -••••••••••••••••••••••••••••••••••••.t1 'tair•=nn u1:nnw 11a1u•tt••~ !1t1a ••••••••••••••••••••••••an•••••••••1; n 1u1.un1u11i.:#••naat111 u P•«•uaana ••••••••••••••••••••••••••••••••••••••••n•tcaaa.tiiiaau !tX•UN 1:1wauti: ••

Figure 6.3 - Overlay Image taken at a digitization threshold of 6. Note the thin outline thicknesses resulting in incomplete shapes.

•••tut•••:ru•• ·····~·s:•••• ••••ia••••st•• as11aaa:asa1Ul:f ••••••••i:n.sue ••••=••t:••ttn ••••••••nn#a ••••••n•••• •••••isaa•tt• •••.ti••••••un aaaau••••a~i•

······=···=~i •.•........• aaaaa::su::i:;:; •••••••aaaaa •••••••••••n ·······••1l•• •.......... ,. •........... aaaa:tlUtaaanti

···=········

•• •• • • •............. ...•..••...... •••1tt:••······ ............. ··=·········· ............. •••i;~········ ••••••••••••• ··=•h•••••••• . ....•....... ............. ............. ............. ......•...... ···········••st • •••••••••••••

• •• • •••• ••••:i••••••• •........... ••••n••••••• .............

eaaaan#a••••• •............ aaaaiu;att•••• ............ aaaaria••=••• ............ •............ ............. ••••u••••••• ............ •••is.itaa••••• • •••••

. ....... . ······=••#11'. . ..•..•.••. ·~·····••#tt . ......... . •••••••••ttn . .•••..•... ............ , .. ............ •••••aa:iuut"• . ...•.••.•.• •••a111aaaiiaaa •••••••••••• aisaaaaaaa#tt• ....•....... ······••JJ••• • •• • •

aaaa~~#aaauaattll~R tt!l~ea a a a a aa#t~=saaaaaaU#•aaaa•aaaaaaaaa•aaaaa

••••ttti••••• ··••••••=u~ ••••••••i:•••••• ••••1*•••&••••;!••···················· •••••••••••n••••=•••:•••••••~~•••••• .•.......................•..........

a • •• a;r • • • a •.....................

••••••#aaai:a••==••• •• •••••••.n• •••••• ································•··· ·····••::••~··==~=·~=········~····· ··············~···················· •••• :: .. n .. •~•••••c•#•"! :~••••it•au aaaa=ra .................................... •••ttff•••••x•••••~uttueaaaa••••~••••••

••••••••••••••a••••••••••••••••••••• aaaaaauu~a#uaaaaaaa~:••••••••~••••••

I ········••tt••······················· •••••*•••·················•••:•••••• •••••••••••••••••••••••••••••• •••• n~u: ••••••••••••uuaaaa •

••••ff•••·············

••••••••uisi;aaaaaaaa!:a• ·····=······=·········· a aaaaaaaaraaaa;;;;aa .............

a aaaaasaMnaa ....... . ...... . ••••• .. ••••• ••n•• ............ . .... aaaata:r;:ra:u.i . .......... . ............ ., •........... ••••••• •

airnca . .... ••t:aa ••••• a:a·faa

• ••••• aaaaaunaa

• •••••••••••••••• .. •••n•••~•••••••anaa ....•..................

••••••it •••••••caaau ••=••• ••••••••••n ··•••••••sutaa ·····=·=·••••tt••··· •••..•••.••. • •••..•.•.....••.•....• •••••===i::••~•••c••• aaaat:t••••aaa a••aau#!taas:i :a:irai::aaau ··aa

·············~····· ••..•.•••••. ··················=···= ••aw;·.,!: wuac.:••••••• •••••r. ~•••••• aaaaaisnaaa:.aaaaaaa: 1;a• ··=~·~······••*•••• •••••••.••••• ···················•## •aaua.nt111aaa;: ••=••• ••••:1~1•••••• aaa#atsttaaai:•••••••• ''aft ··~·=~············=· ....... . ...................... . •••••a.:aaii:: ;sa'l;':r••tt•tt• a a •aai:. ••=•n ·1•== ·ia#wssa•• :: •• ··=·=·····==··········· ••••.....•.......•.....•...• •••••=i:.n:s;o. =•••t;•=••~•••• ••••• !~•••••••• ·iaa:s aaaa ••••• .;aa

····~·=····················· ·····················=··········· aa.11.1.;:u;iaaan:i•a•s•••••••••••na:::saau.aaaaii:r:=•••is:r.uaai:: iaaaaiuiai.. aa

••c••=•=••••••••••••••••••••••••••••••••••••••••••••••••••••••• ••••*:•ir;11nu1 :i•••=••••• •••••=• ·a~::nnuraaaa:'! :••••••• J:aaa .iiaa••••:nsi:aa Figure 6.~ - Overlay image taken at a digitization threshold of 32. Outline thicknesses have inaeased, however are still acceptable. This threshold wa.s deemed to be the upper limit of the range of acceptable thresholds.

.•.•........•. •............. .. . .... . •............. . ............ . ••••••=•n••••• ••••••••••is•••• •••••••••••••• • •••••••••••••• ••••••••••••• ••••••••••••••• ••••••••••••• •••n••••••••••• ........ ,..... . ........•.....

••••••• ••••«••······ a21 '!tlllit l'l1fRSaaaa aaaautisn•••tt••

au a a aa 1t#U2UUtaaua•e • 1u"1ua;ui:1ttaaa •tt••tt••••tt i:a aa;u~.i;orti•••••

••••w=•»ntt ;·a •••iura•aiaaaa • 1t•••••uiiau a a;uuiuu°'uauaa :SQ#SiUiiaiUJINIJa

··········••i:s ••••n•••••••• . ........... . •...........• •••:••········· •••••••••••••• ••••••••••••••• •••••••••••••• • ••••••••••••••

aaaauaaaaaaaa . ........... . . ........... . .••........•.. . ............. . . ........... . ······••ai••• •••it.it••······ .............. . ...•..•......• ••••H•••••-naa •••••••••••••• • •••••••••••••• •••••••••••••• ••••••••••••••• •••••••••••••• • •••••••••••••• .............. . ..••.•...•...• ··"·········· -••tt••······ •••••••••••••

·•··•······· attaa••••itWHa• •••2usuw••••••

••••••••u•••• ••••••••••••• ••••••••••••• •••••••••••••• • •••••••••••••• .............. . ......... . • •••••••••••• • • • • • • •• • • • ••••••••••••••• ••...........•..................• . .

•••••••••••••••••••••••••••••••••••••• ....................................... ·················#···················· ···········#·························· ··························#··········· ·················••tt••················ ········"··················•••rt••····· . ................................... . aaaaaaa•aaaaaaaa•aaaaaaaausaaaaaaaaa ••••••••••••••••••••••••••••••••••••• ••••••••tt••••••••tt••••••••••••••#•#"a ••••••••••••••••••••u•••••••••••••••• ••••••••••••••••••••••••••••••••••••• ···········~························· ••••••••••••••••••••••••••••••••••••• ••.......................•..........• ••••••••~••••••a•s••••••••••••••• • .•..........•................ . ········=··············· . . ..• ••••••••••••••••••••• • •••••••••• ..................... ••••••••••••••••••••• ••··················· ••••••••••••••••••••• ••••••••••••••••••••• ••••••••••••••••••••

. ..........•. •............. .............. . ............. . . ............ . • •••••••••••

• •••• •••••• • ••aw • •••=••••=••~=•;••#aana

••••••••••n••••••ii:t;aaa •••u••=•••••n•••••••"• •••• ••••~ 11•••••11aanaa •• ••••••••u~aau~

an ••••••••"''~• •#•••• •••~~•••an •••••iiit••• • ••11 ;; : .•• •••it•••····· • ........ It ••

f'JP1U11JUl~Jf#1'#a

•t!••n•••••n !:t.n

.... jl#·······

!l• i!•• •••ittt ••naa .... #1: .. "•• ............ p

••••••• ••••tsw ••n•:J••tiutinaw

• ••tt~••~a~=u~~••••• ••••ana1n;·11w"aaaaiiai=a• •••=•••••=•=~~•11iiuaanu

•••••••••:i tsaaaaar.aunaa aaauau~aaua~;:=u~it•••"a

•••••••••i11.1•Ant;a1:nr•A•• aaa#aaa#####ttUan#•aaw • •••••.ii•aa.u:iaa1uia••••••

············=···=~···· •••••n#awttn•••••••A iiwu .........•......•.... ••••••••••••••••••••••• ········~ ··~······=·•••*•·········

•~tvaana•U•••••1•n•n••••••••a • • •• ····=·~·················· • ••••••••••=••••tta#au•tttta#•••• •••••••••••••••••••••••••••• •••••••••••••••ttn•att ·1•nu 1ttt•>rati1aa •••••=••n•••••••••••••••••••••••••••••••=•••~•u•an"••nnan#waaau •••••••••••••••••••••••••••••••••••••••••uaaaa ti :tatt1! itVauai •tta rt ii•• ••••••••••••••••••••••••••~•••••••••••••••••~n•~•••••u••=•••••tt •••••••••~•••••••••••••••••••11•••••••••••••••••:a••~ tt•=••11••ir i;aa Figure 6.4 - Overlay image taken at a digitization threshold of J,, Due to the improved (and satisfactory) outlines recorded, this threshold was considered to be the lower limit of a range of acceptable thresholds.

aa•aaauw.:a.:i 11 a.##1Uf##U:SSaa ••••••is-•=••:s aa•s:;.;;.:.waau

······:::..-•;&.~ aaau;iu~•=•• ........... aa&~-'*+.~ .iWJ:#

•••••J::..Z:.#a::i.. ••••siu••••• ······=···· SUSl:Ufntt•aaa IUU~#aaaa••a ······#···· . ......... . tt••••······ •••••••1nt•• ••••••••••• ······;;=••= •••••••••••

•••••••uii is

•••tt••••••aasz ••=•=•= uiuu;;r;:: aaau•••u1ucaan ••• ,. •••••• .#~# ............. ,; = naaa• :utaau a aaiai.: :sa••••••• '1&tuu:.aa4.;u • .i..i..i ···#········· •••• ,, •••••• #. ............. ":"t•········=· ·······•••••tt ···=········· •aanuaa•aaaaa • • • • •

un;.; ... :.~it•u U:#-:i:&t:iisaaaJU: aasin A:iaaaa •~.:ti::~naaaa:ss::

••••••••aaa#u •~un: . t- • • •••• nniiaaJ:faaaaa

tta~••'!t·••••• 11::.in·.i~••••=

aaaaa:. ll:JUttzaa

a·1un••••••• ••••~::at••• ······=···· aaaau••axaa

• • •

' . 1::·:. : ~ ;;:,i ~ •.

n; ··•· •~ a;1ut.u i: .u •• .i ii::: t:: .tt:::1.:.::iuiiu1ti:.u ·· :::iun:::i;;11: 4 :i..::il::. I< -'·UU

Ii :.; •• .i •• ;.:,;.,,.

""" .1:1 i; ~anaa

•n••••au:aus !

an:s<.;.;l:f f.:JlHt.O

i;; iuia;i:;;::;·

•=-u"'f••"···· ~ -t-=:::taaaaana •··········· ········••h•

·······~······ ·········=· .... ••=•=#······························ aa=.a.aus••••u a•••••••t1 ia••••••r.:iaaaaaa

•••• ii • • •

•••#••······························ aaa:•u1twaaatnaaaaaaa• :iaaaaaaauttaaaaaa S#a•aaaaaaanaaa•aaaaaaaaaaaaaaaaaa•a ••••••~=s•••••••••••t••••••••••••••• ................................... ••••uatt••••••••••••••••••••••=••••• ··-······························· •••tt•a•••••~•••••••ua•••••••~•••••• .................•....•..........•.. ·················••ii; ............•... ••••n#=••••••••••••••••••••••••••••• aaa:ra.a"1nsait n••••••• ~••••••••t:••••=••

•:•=·=······························

···············•······ aaaaa•Ua•a1JaaM#a•aa 'iaa ..... ················· •• ••11 ••••••••• 11 •• •············ • • •••• ······=J•• .......•....

aaaaau.i:iaau 1

·········•·· ••••••u1:••i1 J ........... ••••• •

······· ... ······· .. ,, .. ••••• •• naa ····· ••nea ····· •u.aaa ····· att•••an••••~u:aaaaaa:::.a•••••:ra i:i:asaa a •••••I' ..••

n•••n•••••••••••••••••••••••• • a ••••••••••••••• aaaaS#~#tSa~aaaaaaaa=••• aaaati::S#ftaLaaaaaa•a~aa

••..•.•••••••••••••• • ····=···········••#•••• aa•aaau::aW' :a&•Saa a aaaaaaa aaaaatrutt.aa:r;,uaaaaa••;i aa aa~~•••••••••••••• •••••••••••• ••••••••••••••••••••••• aaa::;:;;au:::: :.aa•••• •••• aitaaaaaa ••••••A •••J:: nua••••=ui. -=• #wr•a•••••••••••••• ••••••••••••• ••••••••••••••••••••aua •••==~;;.=i:aut:a••••• ••••=n ·i•••••• •••••= iaa# ti•••••••ii• •• ···=·············· ............ ······················ ••••=•••aau~•••••• •••••••=• •••••=~••••••••ua~tt ·-• ••auu•••••••••••••• ••••• •••••••••••••••••••••••• a••••a•e::::=ian;aaaann. a.;saaa:aaaan•••• .. ca•••••• 11 :sa ••••=•=••n•1t••••••••••• •••••••••••••••••••••••~•••• aaa:a;;uaaaz: ::sa••=•••1t•••••• •••••'~" ••••••=• ••~ ;;aaaa%aitit tiaa

·=·························· . ································ •• a:u:in:••••::••••••••• •••••••ii ••aai:aaaa aaaa ••••••••:s:: =-••••••••. •• •••·•·••··•••··••··••·••····•····•··•·•··•••••·•······•·•···•·· •••••• t:1uutt:••••••••is•:a••••••ll •••••••••ii11••••an.ai. ••• R•••saaa.a.:aa c•u•••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• Figure 6.6 - Overlay image taken at a digitization threshold of 39. The image is considered excessively dark because the outlines are 2 pixels or more wider than expected from theoretical calculations.


without activating the key below it. Ideally, this distance would be zero

implying that a key would have to be pressed down firmly to activate it. This

would eliminate the chance of activating a key when a finger was only close to

it since it could be en route to an adjacent key. This parameter depends on the

noise reduction and centroid-finding algorithms used. Figures 6.7 - 6.14 are of

a lcm. x lcm. square metal piece held at distances of u,I,2 and 3 mm. above

the overlay. There are two printouts associated with each distance: one of the

metal piece -overlay combmation and the other atter the overlay was

subtracted out, noise reduction performed, and horizontal lines compressed to

form a single vertical column of pixels. This also serves to illustrate the

centroid finding process. It will be observed that a Imm. separation causes the

image of the metal piece to be reduced to a single pomt. The vertical lme

compression routine will only accept single dark pixels if they are separated

from another dark pixel(s) by at most one white pixel. Hence a finger Imm.

above the overlay would be acceptable. However, 2 mm. separation would not

cause the key below it to be activated. This parameter was deemed

satisfactory since a larger distance threshold could cause accidental key

closures.

b) Finger - Finger Separation Threshold: The last parameter

determined was the minimum distance allowed between two :tingers

simultaneously placed on an overlay before they were mistaken for a single

finger. Three printouts were obtamed: one with two 1 cm. diameter circular

metal pieces separated by 1 cm., one with them touching each other, and the

third with them overlapping by U.3 cm. The results are shown m Figures b.15 -

6.20. A separation was deemed acceptable if the blob reduction process could

6.7


•••••••••••••• •• •• •• • • ••••••••••••• •••••••••••••••• •........•.•• . ..............• •............ . .............. . .•..........•• . .............. . .•........•..• . ............. . ............• . ............. . ••••••••••••• • •••••••••••••• •.....•.••... . ............. . ••••••••••••• • •••••••••••••• ............................. •••••••••••••• •••••••••••••••• ••••••••••••••••••••••••••••••• .................... rt••········ •••••••••••••••••••••••••••••••

•••••••• • •• •• • .••............ ··~·········· aeaaaaaaaaaaaae aaaaaqanQ~a•a ••••••••••••••• ••~•--~••••a• aa•aaaaaaaaaaa aaa#SUai;aaaaa aeaaaa•aaaaaa• •MatW•••aa"a ·····-··· •• • ........... u ........ ·········"'·· ........ . .... ~., ..•... ......... .. . ........... .. ........ ..a••• ·····#······ ••aa111u1a .. aa#n a 11;a••••n•" ua •••••••••••a•n ••••••••••••• aaaaaua.aaaa•a aa.aaaaaaaa•aa •••••••••••••• •••••••••••••• ..... ••••••••••••••••••••••••••• ••••••st••a••••• aa;raa••••••1taa

•••••••••••••••••••••••••••••••••••~••••••••••n •••••••••••••• ................................ ·····~···---· .•........... •••••••••••••••••• •• •• •• • • • • ....................... .. . .. .. . .. .. . . . . . . •••••••••••••a••••••••••••••••••••••• •••••••••••••••n•••••• ••••••••••••aa•••aa#•••••••••••••a•a•• ••••••••••••n1a##aaa•11aa ••••"•••••••••••tt••••••••o•••••••••nwa •••••••••••••~~#•••••••• -•-••••••••••••••••••••••••••••••aa• •••••s=•••n-aaa•••••••ttaa •••••••••••••••••••••••••n•••n•••••n#• ••••••• --•••••••••••••• ....................................... . . aaaa••••a•••••••••••••••••n••••••••••• •••••

aw11 •••••••wa;iaa ••••••••••••

·····················~·············--· ••••W••n••••••••uaaa•••••u••••••••••• ..................................... aaaasa!laa •••••••n•a

aaaa1i•n•••••• ••••••• •••••••n••••• •••#••

•••••••nn•••••••~•••••••••~•••••••••• •••••••••••••• aaaaaaaaaaaaa•aa-aaa#aaaaaaaaaaaaaaaaa aaa:ea•aH•l•J!'aaa •••••• •••••• • ••••• ··························~·········· •...•...•.•.•• 888888aaaaa•aaaaaaaa•aaaaaaaeaaaaa:eaa aaa•aaaaaasaAa aaaas• ••••»•••••••••••••••••a••~••••••••••• ••••••••••••• •••••• ··-················"················ •••• • ••••••t••• •••••••tt••••••••~••••••••n~•••aaaaaa• n ·aa 1aa1n•••••• •••-••••••••••••••••••••••••••• a a aaaa••••n••••••••n•• ····=#····················· • • ..................... . ·················••tt•••• ••••••.• • ................... =·· ••••••tt•••••••••••••• ••~•••••••••• ••••n••~••••••••n•••••• •••• .. •••••••••••••• •••••••••••••• •••••••••••~••tt~••••naa •.......•........... ···········••tt ~"··~·~······~·~······· •••••••••••rr•••••uaa1iaa ................... = ............. . ••....•..•..•.•..•.• ····~········~· •••••••--••u•••aa•••••~

••••••n••••r.••••••••r.•• •••••aaaaaaaa•aaaanaaa• ..................... . ............ .. aa•••••~•••••#•••••• •••••••••••••• aa ... aaaeaaaaaaaaaa• a aaaaaaa•aa aaaaaaaaa~•••R••••r.•aaw .. •••••••••••••u•~••••••• a a• ff~•••••••"••••••~••••••••••• ••••••••••••••••••••••••• aaaaaiatl!11auaaau11rr;as;;aaa••••••1t•a aaaaa#au'*•••••••••••••=••••• a ••••11 ••••••••tt:tN iJJJ••••n:a1H•••••• •••••••••••••••••••••••••••••••••tt••••1t•••••••••••tt •ta••nnaaanaa aaaa•Haiiaaaaaaaaaa:a:aaaaa•tt tt•aaaaaaantt JlaWaWSW!I :f'W JJ•lfit#lfU!l811a#aaaa ••••••••••••••••••••••••.,!'•••••••••••••••••••••:t iiawn :ur•••••••n~•~.

Figure 6.7 - Overlay image taken with a metal piece placed on the overlay. The metal piece shadow can be seen near the top. This piece .. overlay separation of 0 mm. will be deemed acceptable if the centroid of the shadow can be computed successfully.

............. ••••••••uaiuta

.. .. .. . . .••.•...••...• ............. . .............. . •••.....•...• ··~············· ••........... . .............. . •••••••••a•~• ••••••"••••••••• •••••••••auaa •••••••••••••••• ···········~· =·····~········· •·•••••••••·• ••2••··········· ••••••u•aa:aa uas:auua•••••••• ...•....•.....•.............•. ···········=·······~=·········

. .. .. . . ... . . .... ,, ................... . •..••....••...• . .......... . aaaaa•1•aaaaaaa aa•aaaauaaan .............. . .......... . ······~···· . . •......... ..•...•.• . .......•... ·=········ •. . .......•.. •............. . ........... . •#••••~••••••• ••••••n•••••• ..•..••.•..•.. . ........... . ••••••n••••••• ••••••••••••• •............•................ ........•..... . .........•..

aaaaaaa•za:aaaa•aaauaaa•aa•a#&8 aaaaa•aaaaaaaa aaa•aaaaaaaaa •••••••••••••••n••••••••••••••••••••••••••••••• •••••••••••••• •••••••••••••~••••••~•••~•••~•••••••••••••••••• aaa•aaaan .. aaa ...................... ~........................ . ............ . ••••#••••••••••~••••••••••••••• •••••••••••••• ••••••••n•••• .............••.......... .. . . . . ••••••••tt#•#••••••••:•••••• •• •• • •• -~ •• •• •• •• •• ······································ .•••...•.......••....•• ••••~••••••aaaaaca••=•••••••••••••••••~r••••••a•'t ••••••=•ii•••••• aaeasa:uaaaaa~as:aaaaaaaaaaaaa.aaaaaaa•aaaa•aaaaa•aaaaaaaaaaa.aaa

•••••••••••••••••n•• !taa:aa«aaa:a••••••••t•••••••• .. ••••••••a•aauaa ............................................................... •••••#•••~•••••••••u~••••••••••aa#aaaa • •• ••••••••••••~•• aaasaaa•••••2••••••••••••••••••••••••• •••._. ••••••••••••• ••••••• tta an •••••••=~=••••••s:;t "*•••=••• ••••••• tt.aa ••••••• aaa aaaaaua•••••••••••••••••••••••••••••• •••••••••••• a ••••••• •••••na••••••••••=•=::••••••=•••••••• ••••••••n••••• •••••• •••n••=•••••••••••••••••••••••••••••• •••••••••••••• •••••• ···················=•:ta••············ ······••r.•••2• ••••••• ..................................... •...•........• . ..... . ••••••••••• n•••••s:::a-::aanaaau•••••••• ••••aa:sa:: •••• ••••ua• ·············#······················· .......... .• . •..... ···············~•t•==················ ..•• .. • ...... .. •••••aaaaa~a••••••••••••••••••••••••• •• •••••••••••••• . ••••si~n•••=•••••••••••••••aa::::aaaaa ••••••tt•#·······~·=········· • ••••~•••••uaa•#aas•=~• a •• aaaaa •••"•••••••••••aanaa ••••••••w••

•. ···············••tt•• ....................... ········~··············· ........................

aaaan:a:<:a:;ia:::aaa• ••••••= ••••:r:i•- aaaaaaaa?r••••••••••••••• aasaaa~•••••t~•••ft•• •••••=••••••••• aaaaaaaaaaaa••••••••• .. • ••••• ani::tw:ti••tttt?t'tt:=• ••••st-;: :;::aaaaaa ••••••••a••=••••••••a:aa ••••=•••••a••~•••••# •••••••••••••••• ••• .. ••••••••••••••••••• ••••:U.) u 1t1u. ~ u•••c ~•• anaaaa!':a:aaa:aa•• ••••••••••=•••••'!"••• :~•• ····=·············· ......••...... . •..................... ••••tn:ot•:t.:•••==a:;;a;i ••••• •aaaa ••••••••a:;ta::t"=••••••••te•• .... ............................. •••======:=:~=•••==~~•=• • aaaaaau•••••au#•••=•••••~•:•n•• aatataaaattttUSS11SSt*aaaaa#t8 aaa.aaaaaaaa&a#aaaaaaaa••ataaaaa ••••n•=•=•.i.: 111a:=:1u1::airttaaaaas:::2:r.a.au:saaa:1 s•••••a.u ·· ••• -i••••l:•aa 1r•• aaaaa#aS:a:aaaaaat•aaaaaaS•&aa•aaaaaaaaaaaaaaaaa:aaa#aaaaaaaaaaa •••••ii-auu;:;;;;:;r:r.::t:a::;::a1111aa=:r:•••=••••••••11••••••a.i:•taaauaaaax•••i.:aa aaaaaasaaa:aaaz~~2••••••••••••••••••••••••••••••••••••••••••••••

•••••tt Ii ana.una••••••:a ~••••••;: ::=•••=••••-= ,••••••• ·1••• :ta•••• ••!t:t•• Figure 6.9 - Overlay image with a metal piece held 1 mm. above the overlay. The metal piece shadow can be seen near the top. This piece - overlay separation of I mm. will be deemed acceptable if the centroid of the shadow can be successfully computed.

.i

-------· --·-------·--------------------------aaa•••••••••eaaaaaaaaaaaaa••aaaaa•&aaa#•aaaaaaaaaaan••••••••G•a •••••••••••••n•••••••••••••••••••••••••••••••••••••••a••••••••• ······································~·····················"·· •••••••••••••••••••••••••••••••••••••••••••••••••••a•uua~•n••••

··························································••11•• •••••••••••••••••••••••••••••••••••••••••••••••••••••=••••••••w •....•.•........•.........•........................ ., ........... . ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••n aaaaaaaaaaaaaaaa•••s•a•aaaaa•aaaaaaaa•~•aaaaaanP•aa.,aaaa•aaa~aa

·········~··································"·#················ •••••••••••••••••••••••••••••••••••••••••••••••n••••••••n•••••• . ••..••••........•........•...................................• •••••••••••••••••••••••••••••••••••-•••••••••a11wau1••••w•••a11 aa a•••••••••aaaaa••aaaaaaaauaaaaaaaaauaaaaa . .•.•••..•....••. ••••••••••••••••••••••••••••••••••••••w••••••••1t•aau••••••••«•• ••••••••••••••••a••••••••••••••••••~••••••••••~••••••••••••••••

··························································••11•• ················~·····································=~t•••••• •••••••••••••••••••••••••••••••••••••••••••••••••••~••••••••n•• •••••••••••••••••••••••••••••••••••••••••••••••••••••••••a••••• . .................................................. ~·······"~·· •••••••••••••••-x•••••••••••••ria•n••;.a:•? tt••••••••••••••••••••••••• ·-································••rt.!l•:t•••••••••ll••••····,.·· ·······················••tt••························•lt•'f••••••• ••••••••••••••••••••••••••••••-•••aa••••••••••:,•ttttaaaaaaaan-n•• aaaawaaaa•••••••••••••••••••••••••••••••••••u••••••••~#••••••••

••••••••••••-~•••••••••••••••••••-•••••••••1tw=1aaa1ia•••••"•••• •••••••••••••••••••••••••"t•••••••aalf.••••••••;ta•n••••n•:i•JJ•••••• ........................................ 1!••••••••1t••••••••••••1t•• ••••tt••••••••••••••••••••r•••••••••"••••••••••••••••••••••n•••• ••-•••••••••••••••••••••••••••••••••••••••••••••••~•••••••nr.•• .................•........•..............................•...•. •••••••••••••••••••••••••••••••••••••••••••-••••••••••Jt•••••:Jta ••••••••••••••••••••••••••••••••••••n•••••••~•••••••••••3•••••• ••••••••••••••••••••••••••••••••••••••tr•••••••n,..•eu·:w••••n•#~•• aaaaaaasaaaaaaaUDaaaaaa•a•aaaaaaaaaaaaa•aaaaaaaaaaaaaaaaaa•aaa• aaa•••••••••••••••••••••••••••••••••••naaaaa••••••ana•••R••nnaa •••••••••••••••n••••••••n•••••••••••••••••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••••ft••••••••••••••••n••••~••

•••••••••n••••••"•••••••••••••••••••••••••••n•n••••••U••••••••• •••-••••••••••••••••••••••••••••••••••••••-••••••:t•••w-t1att•n•• ····················································•=1••!S•11•••• •••••••••••=••••••••••••••••••••••••••.M••••••••ir•••#•••ff••••n•• ••••••tt••••••••••••u•••••••••••••~•u•»•••••••••••••••••••~•••••

••••-•••••••••••••••-••••••••••••-••••••••••~•ua;1J1••••••f1~:1• •••••••••••••••••••••••••• i.a•••••••11 •••••••••••••••••n••••••••• •aaaaaaaaaaaaaaaaaaa•aaaaaaaaaaaaaaaaaftaaaaaaaa:aaa•'l'a•a1'jflt•l!•8 ••••••ft•••······,·············································· • ................................................ "·~·••»••··~--•••••U••••••••••n••••••••n••·······~·······•w••··~···#~········ ••••••••••aaa•••••••ts••••••••••••••••••s•••••••••••n••~n1tr.1trr•r•a •••••n••••••••••Jt••••••••o '" ••••••• 1u:1" ••1t••1111 a1: r a11: tt• wrtu:ufafl•••• aaaaaaaaaaaaa•aaaaauaaa•aaaaa•aaaaaaaaa•aaaasa1,lllW1! '1Jt1fa Jt•Jt• ll' !Jae ••••s••••••••••is ••••••••• !r ••••••••a:t••••••••n n•••••n••'*••w••••• ••••!.!!:l!Ullll!'~·····~~!1!.!1••••• ..................... I?&• ff# •••••••

Figure 6.& - Attempt to compute tne centroid of the metal piece shadow in Figure 6.7. Shown, is the shadow of the metal piece reduced to a single line after the overlay details have been subtracted out. Thi.> line can successfully be reduced to a single point to find the centre of the metal piece.

•••••••••••~•••••~4n~a••••••••••••lr.3'---••••••••••••~......n• ... ••n•• aaaaaaaa2uauaa•••••••••••••••••••••••••••••••••••••••#aaa•••••• •••••••U•::ta!t••••••=•#aaa••••=••••••••••• ... •••••:: aaa:sa;saa••••••• •····•········•····•·········································•· •••••u••••••••an:aaui:suaaaa•••••••••••••1t••••••••• •••tt••••=•••tt•• • •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• aau•a•••aaaaastaa••=~aaaa•a•aa•••aaa .. aaca .. a aaa•aaaaaaa•aaaaa .............................•....................•......•..... ··=············~···~= .. ·····=·········~·················=······ ............•........•...•...............•.....••.......•...... ••••••n•••n•••••••••••••••••••••••••••i""•••••••=x••••••••••••n•• . ............................................................. . ••~•tt•=•=•=~••••••••••••••••••••••••••n••••••••~•••••••••••••••

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••••••••a•aa:aauunaa•••••a•••••••••=•••••••••••••••=•••••••• •.•.................................•.......................... aaan.uauaaa:aua:twaaaaa.aaaaaaaaaaaaaaaaa•:;aaaaaaa••••••••••••••••• ··········"**···················································· ••••:t•=•••••••••••••~•••••••••••aaaa••~••••••••n•••*••••••••=•• ~as#aaaaaaaaaaaaaaaaaaaaaaaaaa••••••••••••••••••••••••aaaaaaaa•

··········~··==··=··=·················=········~···············

=··=······=•••••2••••·········································· aaaa::iauaa:tu n•••=c:i:a:r :r•••••••••••••••••r •••••••• :taaaaaaaa:iaaa1t•• nua••••••~••••••••••••••••••• .. •••••••••••••aa11•••••••••••••••• a•••••••••tt•••••a~u~~--•••~aaa•--•••••~••••••••=aaa2••••••aanaa

••••••=•••a•••••••••••••••••••••••••••••••••••••••••••••••••••• aaa!te-tt111u1•istt .. •••••tt• .. •••••••••••••••r.••••••••·taaa•••••tt•••••• ~ ....................................................... #•••••• aaa .. iaa•••=:a:••••••=it:;aa••••••••••••••••••••••••11aaa•••••=•••••• ··········=··~····· .. ·•············•··············•········· ... aaaaaa;sa••••••~•aaaucaa•••••••••••••••~•••••••••••••••••••••,••

·············~················································· aaanaaaaa:;;:aaa:aaa:a:u:::aaaaaaa.a ••aaaaaaais aaaaaaa•1: ••••••••=•••••• ••••••a•aaaas•••••••••••••aaa:aa .. aa•••••••••••••••••••••••••••• ••••••=••••••=••••••••••••••••••••••••••••••aaaaaaa••a#aa•••••• aaaaaaaaaauaaaaaaaaaaaaaaaaaa•••••••••••••••••••••••••••••••••• aaaaaaa•n•t: :1.au:aaaaa• •aaaaaaa:11••••••••1! eaaaaaaa·taaa:iaaaa11 •••••• aa•=•••••••#tt•••••••••••••••••••••••••••••••••••••••••••••••••• •••~•••••••z••:=~:==~••••••••••••••••airaaaaaaa•••••••••a#aaanaa

·············=················································· aaaa1ut••••ttii•••:a:a-::s::::.caaaaa:ia:raaaaaaaara•••••••1raaa:aaaaaa•••••• ··········~···················································· ········~··••:1te11:#•·~·················~·············••:=~···,·· ··········=···················· ................................ . aaaaaaaaa•••••••••••#sa•••••1t•••••••••><••••••••n••••••••••••••• a2a:aaa•••••••••••••••••••••••••••••••••••••••••••••••••••••••• ·········••tt••···············=········~························ •••:••························································· ••••••••••a#•••••••••••••••••••aau•••••••••••••naaa•••••=•••••• ................................................................ •••••::tt••••••••aaaan#aaaaa:saa"t••••••••t:••••••••n•a••••••••••tt•• =··························· .. ···•·········•··•··•···········•· •••n•~n•••••••••••••••••#===••••••••••=•••••••=•••••••••••aa2aa a•aaaaaas••••#aa•a•a••••••••••••••••••••••••••••••••••••••••••• a•a:1 n:::~ nu• tt••••••••-st :•••••••==•••••••••=••• .... •a:::•••••••••••••••• ·~··~•tta&a#•aUaaaaaaaaaa•aaa•••aaa•~aaaaaaaaaaaaaaaaaaaaaaaaaaa

Figure 6.10 - Attempt to compute the centroid of the metal piece shadow in Figure 6.9. Shown, is the shadow of the metal piece reduced to a two pixeb after the overlay details have been subtracted out. As explained in the text, this case will be successfully reduced to a single point corresponding to the centroid.


............... ····•••&iiliiai9aii ·············-~ ••3·········· ••••••••••••• •••••••••••••••• ••••••••••••••• •••••••n••••• aaaaa•aaaaaaa aaaaaaaaaaaaaaaa aaaaaa•a••••••a aa!f.aaaaa1aa11a •••••••••••• •••••••••••=•••• ••••••••••••••n ••••••••••••• •.••........• ·•·············• •.•........... ·~!••········ ••••••••••••• •••••••••••••••• ••••••••••••• • •••••••••• •.......•...• .•............•• •..•.....•... ··········"· ••••••••••••• •••••••••••••••• •••••••••••••• • ••••••••••• aaaaea•aaaaaaaacaaaaa8eaaaaaa• aa .. aaaaaaaaaa aanaaa•a•••na •••••••••••••••••••••••••••••• •••••••••••••• • •••••••••••• •••••••••••••••••••••••••••••• •••••••••••••• ••n••••••••»• ••••••••••••••••••••••••••••••• •••••••••••••• • •••••••••••• ••••••••••••••••••••••••••••••• ••••••tt••••••• au••••••••••• ••········•········•··········· ...•.•................•.....• ••••••••••••••••••••••••••••••••••••••••••••••• ••••t••••••••n • •••••••••••••••••••••••••••••••• •••••••••••••• •••••••••• •• ••.••..•......•..••.... ·~ .. . . . •••••••••••••••••••••• ••••• •• •• • •• •• •• • •• •• • • •••••••••••••••••••...••••••••.•••••• ······••••tt•lt••••······· ···············~·····································#········· •••••••••••••••••••••••••••••••••••••••••••••••o•••**••••••••••• aaaaaaaaa•••o•••••••••••••••••••••••••••••••••=••ua•auaauaaaaaa •••••••••••it•••••••••••••••••••••••••••••••• •n•attit••••••••••• ...•.................................. .. . ...........•.•. .•........................•..........• ••••--• ••••••••a•n••

·························~··········· ······#··· . ··~······ ..••.•......•.•.........•.....•..... ·········~••It •••.... •••••••••••••••••••••••••••••••••••• •••••••••••••• • ••••• •••••••••••o••••••••••••••••••••••••• •••••••• .. ••~• aa~n•• •..........•.•.•••..................• ·········"···· ;•••••• ••••••••••••••••••••••••••••••••••••• •••••••••••••• naw•••• •••••••••••••••••••••••••3••••••••••• •••••••••••••• v•••••• aaaa•aa•aa#naaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaa aaaaaaa ••••••••••••••••••••••••••••••••••••• ••••• •••••••••• •••••••••••••••••••••••••••••••••••• a ••• ••••••••»uaa ······················~········· • ····••••#••··········· •••••••••••••••••••••••••• •• a .. ••••••••annaaaaaaanaa ....•......•.........•. ....... . ...................... . .•••••..•......•..•• .............. ············#··········· •••••••••••••••••••• ••u••••••••••• •••••#•••••••••••••••••• e•aaaaaaaaaaaaaaaeaa aaaaaaaaeaaaa•a aaaaaa•aaaa•·~ttaaa•aaal!•a

•••••••••••••••••••• • .,.. • .,••••••••••• aaait 1rn :s::a,aaaai:iaa:;-; aaaa# aaeaaaaaaauaaaaaaaa# aaeaaaaaaaaaaa• aaaaaaa11!1a•U:tnaa#l!W••!•aa •••••••••••••••••••• ••••n•••••••••• ••••=xnaa#a=•a#aaa#aaaa ••••••••••••••••••• •••••••••••••• ••••••n~n••'I ·~••••••••it•• ••••••••••••••••-•••• u••••••••• ••••••••••••••••••••n•••• ···········=·········· ••... ···············•'t••••1t1t1J.ttliJS• ·············~··········· ···~·················#········· ••••••••••••••••••••"•••••• ••••••••••~••••••#•••~··•~••~•n•n•• a••••••••••••••••a•••••••n •••••••••••n••aaaan:r 1' :;u • n•ntt•••11 a•••tt aaaaaaaaaaaaaaaaaaaanaaaaaaaaaaaattaaaa•saaria••MOl•H!:f.'1 •itaaa•tt*lil lflt ••••••itaaao:•=-••• n•••••••• ii••••••=••"' ••••••••n••• •an•• :i in u;i naanit aaaaa•a•aaau:a:a:aaaaaaaaaaaaa•a•tsaaaaaaauaaaaaaaanaaanaaaM ••it• 11 • •

a•asuaaa••••••••!t••••••aaatn•••••••••• lf••••••tl,!f • :iiun.n"una,i;i !1aaan •••••••••••••••••••••••••••••naaauaaaattaair••••••••11 :: lfttn•••w1u1an ··············•• 1t••.!9-!.!.'!!'''····· ••••lfzt#U ••••• • n• JUttt1Ul111$ n•1'_iiu:•··~

Figure 6.11 ... Overlay image taken with the metal piece held 2 ~m: above the overlay. The metal piece shadow can be seen near the top. This piece ... overlay separation of 2 mm. will be deemed acceptable if the centroid of the shadow can be computed successfully.

••••••~.::: «#t:;aa ······=······ .... .. .. • •• .. . . . . • • ·····=··=~tt•• ······=••tt••••• ······~·······= ······••2••• aaasaaaaaua•a •aa•aaaaaa••••aa saaaaa•a•aSaa•a aaa•aasaaaaaa •••11=ruit:s•~•n• ••••••~••••••=•• ••••••i< •••••a1u: •• •i;:s:ra••••• ······•••tt••• •••••••••••••••• •••••••••••••• • ••••••••••• •••••=~==:~=• ••••••=••••••••• ••••••~••#•••• ••••••••••n .•..•.•...... ................ .............. . ......... . •••••=ua112~~• •••••••••••••••• ••••••P.•••••• •••••••~••• ······~··~·=· ...••..••.•.. #.. ..•........... . ........... . ••••#~=••===• aa::sau•••••••#aa •••••••••••••• •••••••••••tt• .............................. .............. . ........... . ••••••••=•=:::u1a••••ii••••••••• • ••••••J1

••••••• •• •••••••••t1 • ·········~···················· •..•.....••... . ...•......•. =•••••=:=#~==••#•:S••········~·· ······~·····=· ···~·········· ·········~#···················· ............................ . ·······===•tt2=••·=·········••tt••······~·-······ ···~·==······· ............................................................. ···==•••=:;~#··=···=~·········· ............. . ............ .••.....••..•••...•.•. .. .. . . . ··········~~~--~·=······ .. •• •• • • . .. . . . .. . ··········~··························· ...................... . ••••••a ::a:::; :-:a:n::s:=sawc•••••••U .nuuaa••••1; •=••••••:ta a.it it••••••••••• ·············"················································· •=•••••••r::::-:~•=••••• :suuu.aa:1• •••a•••a.au • iu1;.:aaaau1 a air 't••••~•••H•• ·······==~···························· •.••.••.•.........•••..• ••••••••••~~••••••••~••••=••~=•••••••• a a• a•2•••n~••••••

··································~··· .•.....•...•••.••• :taa2aaaaaauQ:;r;ui::::cr.•aaa•:SSUa'l•aaaaaa aa•aaaa·Js# aaaaa•a .. aa asaac•••••=•••:•••••••••••••••••••••• •••••••••••• ••••••• ::.:icl:••••••==•=====•iutaa•••••11aaaau•• •••••••.uti•.a.:ctu• •••••• •=••••••••==•=••=•••••••••••••••••••• uas••••••u•••• •••••• =•••#••••••~•••=••~tt=•••••••••••••••• ••••••n=~as~ia •••••• ••••••••••~•••••••••••••••••••••••••• •••••••••••••• •••••n •••••~==##n=••===•••••••••••~•••#~••• •••••••r.•••=~• •••tt•• ··········=·························· •.•...... .. . ..••. ••• aa.iu;aaa-it:iaaiun:••= :3aa•••=• 11a•••••• •••• •• •••••• :uta

=······~····························· aaa a a au ::uunnt•• •a••• a :a;ucaaaau;ia a• aaaauuao2•=•••••••••••••• •• •

.•............. •••• t~•••ts••••11 ••• &tta •••• , •••••••••••••• =••#

•••••::r•• ·ia:ut :nz:ura;:#a#:aa

······=············· ..........• ·······················# annsraau':IS::r::::##St•itit at:n•••t?::••••••• ••••••=rt~•••tuun:nc;;•:1t•••

••••••••tt•~•••e••••• ••••••••••••••• •••••••••••••••••••••••• ::::u2:atr. ~= i:.:t •• 1u:tu.acarut• aaa:::r::;a::a:caaa •••••••:i-:a:u: xa:ur=t•••• '!11•

·=········="········ ••••••••••••••• • ••••••••••••••••••••••• ••aa.ani::.::r=a~.:•=-====• ••••••=••==•••• ••••:i:r;i 1aa.i:naai:::;r~a••••• ~•=•:u:a:=:===••:i:=:1usa a •••••••• ••'f••••saar. :r::-=:s::sauua•·~•• #aaaa#••~~=•••••••••••• a •• ••••••••n•••••••••••••••••••• a#aaaaaa=~~e=••••••aitaasaas ••••••••••••••••••==••••••=••••• u:auiaa::a.au-::a:.a:::au-ttaa:iis.••:u i:aaa :iaa:.=is a:ui:uasia: naai= :iaaa:aiia.:c;•n•• •=aaa#==~~##a#•aaaaastaat••••aaaa•Uaaa#aaaaaaaa:aaaaa:aaaaaaaaaaaa

.an ti••••==•= ::a:a~;:::::ait:;a.u:sa:a.•a.a = ••:t•aaaau 11.r;a•••a• JI••• 't••=••••i;u•• •••••t=•=•#•••••·#~~········~····················~~~-

Figure &.13 ... Overlay image ta.ken with a metal piece held 3 mm. above the overlay. The metal piece shadow can be seen near the top. This piece .. overlay separation of 3 mm. will be deemed acceptable if the centroid of the shadow can be computed succ=fully.

••••~•w••~~••••===•~••aaauauaaa••••••••••••••~•il'I'a?r ... •.-W••••••• •••=••~•••••••••••n•~•aaaa•••••••••••=~3aa•••••~e••~••••••••n••

·········:····················································· ••••••un11 s u rt:••••••n »1· ••••••••t: •••••••••• ••••••••~•• ::••••••••• •• ••••:i~•·=······················································ aaaa:##";;:;;OJSS:aaaaaaa;;:; ~aeaaaaa•taaaaaaaa.-a•aS11aa:1":aaa• .. a.aaaaa•11aa

•••••#•••······················································ aaaaaaaa:a•aUaaaaa:a:ra•#•&Sa•aaaaa•aa•~aaeaaaaatt••=~••a•saaaaaa

•••••••••••••••••••u••••••••••••••••••••••••••••••••••••••••••• ••••••1ttts~•••••••••t: ec••••••• •••••••••::•••••••• 3••••••••••••••• ······•••tt•••··········~······································· •••••••••••tt•••••••ir .·••n••••••••••••••n••••••••••••~•••••••• ~ •• ·········~····················································· •••••~•ii••==••••••• c •!t••••••••••••••••n•••••• •1~•••.ii•••••••• Paa •••••••••••••••••##saaauaa••••••••••••••••••••••••••••••••••••• uaaaaaua•••~••••••••~••••••••••••••••••••••••••2•••~••••••••r•• aaaaauaaa•aaaaa•JaNaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa w••••••ua•U•••••••==~••••••••••••••••••••••••••••••=••••••••tt••

•••••#••······················································· ~••••u•2••=tt••••auatt.aa!tU••••=•••a•••••••••••••••aa2••••••••~••

••••••••••••auaaa#••••••••••••••a•••••••••••••••••••••••••••••• aaaa:aa#::::~•sa:a:au~=•=aaaat••aawuaaaa•aaaaeaa~aa.~aaa#u:aaa~•a

••••=•~saaaaaaau•n=••~•••••••••••••••••••••••••••••••••••••••••

•••••=••!:::::1u:aa:o;::;:; ~; a;;::a::••••••••••••t< •••••••• ::t••• ::•••••••~n •• aa•aaaaaaU##aUaat1 ;t• ft#•aaaaa:raaaaaaaaaasaaaaaaaaaaaaaaaa•aaaaaa1t a•••n•••W:::t:##a••••is~:::;a:eaa••••••••••• ••••••••-= ·:aa::- ·i••••o••• ·· •• ················~····=·=······································· ••••:i:t:.ft~san :~••tt"St#!ttr I"".,,,. !!'•11••••••••••atii: •••••••~ '•••""taauar.an•••• ···==••••••••••:t••~········· .. ••••••••••••••••••••••••••••••••• aautaai: ·· :::: :ur.:•••••=== . ~•::i••••• aiaaaH•••• :t•••••••• '1aa;;;:a•••=•••••• •••••'1.i:c::;=•••••:~•uct•n•••••••••••••••••••••••••••••••••••••••••• :!•••ittttttl lt~.t:rt#aaaaaa#•••a:raaa::ttaaaa•••• !'f:••••••=~'1#\lr:s"'lt::ut:t•••• •t•• ·=····~··#·········~····························••tt••••········ ==••••••••f! ••••a3tt##: .•a attuai:aiiw•••tt•••=s:=••••••::;::saa:t••••••••v •• ······~························································ ==••••~::a•aan##a•~r.~•~===••~••~#==••#~~=•••••••••~~••••••••tt••

aaaaa#c~:2aaa••••••~•••a•••••••••••••••••••••••••••••••••••••••

=••atuu:~ca:t::••.tuu1::iat:; •:t=••••:i:a::i:s;:saaa:.-a••••aaaaaaa:usaa•••••tt•• ••••=•~=::s••••••••••••••••••=•••~•n•••••••••••••••w••••••••••••• ••••a:an:a::s•iia:iaa::::::·;a:===•=••••••=••uuiaaa• .. •••••a·iaa:•=•=•••••••~•=•~••••••••=3••••••••••••••••••••••••••••••••••••••••••• =••••u•~•~~~•••••=•=~3•=•=••r.••••••••••••••••••=•••~••=••=••~••

~··==·==··········~············································ =•=••~xi: u oitt::ts::::::1;::~ ua::::a::raanaaaauaaaa:;aaa•••••:iaaa;taa:::na•tt:: •• ·=··==~::#==~··················································· ·; c;1ua•aaa2;: ••=•===•• ~•••••••••••••••••~: s::::a::a: :aa: =••••a••• uaa ............................................................... =•••• •uaw:utnaaaaaaaii =•==•••••••••c•••••••••••a#-..aa•a=••aaaaua •• ••••••••••••••••saana•••••••••••••••••••••••••••••••••••••••••• ···--=•2•=,~·····~·~=~==·=··~=-·······~········=•••2•••········ a••••••••••••••••=•~•=•#••••••••••••••••••••••••••••••••••••••• =:aaa#tt"S":t:a: ::a;;;:::===~:.=•==•••• 1t•aatt••••u••••••••::aas:1aaaaaaa#!! •• •••••===•=a:a::a:aaa••••••••••••••••••••••••••••••••••••••••••••••• ••••acu:1zsa ::aaa.;:a:a:t:;:a•~•••••••••:i.•••••••••••aa#aaa -iaaaaaaa•c••

················~#=············································ =••••:Juau•st•••••••••ia;aa:aaaau iaaau•aaa!!aauaaaaaaaua:ta••••••••••

·················~-~l•••········································ Figure 6.12 ... Attempt to compute the centroid of the metal piece shadow in Figure 6.11. Shown, is the shadow reduced to a single line l pixel wide after the overlay details hav'e been subtracted out. This line width is too small for reliable centroid computation.

••••••• • ••• a:aa••••un••••••••~••••••••••1 ••siiiiif ;;:sa:i ;JiiQ~;:i~'::iiaa

····===·~=··································~·····=············ •••••••••••••••••••••••••••a:z••••••••~•==••••a:aa#••~••••••~••

····~·························································· •••ttttu n 1t 111111 =1•••••=•U":t•••=•zaii a:a••=• .. ••raat••••= ·:aa= :i=••===•="•• ••••u•n•n•••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••=••••.=•••••••••••=•••••• •••••••••:: ac•••••= =~•••::a••••••••••• •••••=•••••••••••••••••••••••••••••••••••••••••••n••••••••••••• ••••••a#aa:.:ia:saaaa11u~•••••••••••••••aau11••••••if•••••••••:ia••tt••

••••tt••························································ aaaa.:r====~==••=•••••=•••••••••••••a•a•1=••••••••1•a••••••=••=~•• aass=##SA####aaaaaaaaaaaaaaaaaaaaaaa•aaaa#aaaaaaa•aaaa#aaaaaaaa •••••===.:r=:a••••••t•#••••••••••••••••••n••••••••1taas::s:r::.att:ii•••!t•• •••••••a•u:••••••••••••••••••••••••••••••••••••aaaa••=•••••••••• aass:11ui::c:uut:u••••••••=a#:a•••:t•••••••••r. sa••••••n••• ·ur•••••••n•a ••••••••••.•••••..•.••.•..•...•.....•...........•....•......••• ••••=~=;:;;:a#uaaaa#uaaa:r•••&••••••••••••••#:uu1:aaa2aaaa••••~••

•••••••••••••••••••••••••••••••••••••••••••••••••u••••••••••••• aaaaa=:=:au:taa:a:aaanaaa#a&#~aasaa:aaa##a.aaaaa#:faaa,aa#a#aWattaa

····=······=··················································· ••••~•~••••••==•••••••••••••~•••••••••••••••••w~aas2•==•~••••••

•••••aaau•••••••••••••••••••••••••••••••••••••••••••••••••••••• =•••=":;"t;;;a:;;::a•=•=••=•••••••••= •••••••••14 •••••••n::•••'4••••itas••••

····~=·=······················································· =•••= uu.as:!t;#Ui:a•••===••==i•:ts :& ••••=•••at~-•••••••#·· aa•#•t•# 't:aa:-:aa ••• ro :i•••••uaaaaa•••••••a•••••••••••••••••••••••••n••••••••••••• !:!:aSt##i~tt ~ •ut tt u•a••=•s =••===== :t•••••••••= 2a••••==r ••= •••::• .,. •• -r•• ••••=u••••••••••••••••na••••••••••••••••••••••••••••••••••••••• •••••=::2====t:t:taaaaau••••••~••••••••:a••••••••••·: ••• aaau;t:u:=•••• ••••u:us=tt:t••=••••••••••••••••••••••••••••••••••••••••••••••••• • ua•a::::.:===:::.iaasa••#~::saa•••ar:staaaaa.aaa;: •••••••= ·~•••.a•=••n•••••• ••••=•#•~•=n•••••••••••=••••••••••••••••••••••••••=•••••••••••• ::n:aa:ui: =•=~=•===••=-=:i•••tt••• ~a:•••• ••aauaa••••=u ._ aaa11ta:ia·1•:•1:1ta ••=••au~=••=••••••••••••••=•••••••••••••••••••••••••••••••••••• ••••=•=•:-•:,11:aw!l:aa:ait••••=====si:1t•aa:a::aa:::u::z;a:saaa:i" •••=••••!t••• .:aa •••••3#n••n•••••••••••••••••••••••••••••••••••••••••••••••••••• ••-•••~••#•••••••••~u•••#•===~••===•••~a•••••:ta:saaaaaaaaaaartaa

····~1:•~·················••*••································· n••••=•s••••••••••=•=•=•#=t•~=••••=•••u:•a•aaaa#•aa==••••~•n•~••

=•••:1 tl:#!~rt:••········································=•••••a:••••• •••:a.;;$;.;:sa;;:: :ia:a:auaa:n::;aaa••=== ~=•••••aaua•zt••••= ·; aa= -•w;;:;uiuus:iaa •••••:uaaaau••••••••••••••••D••••••••••••••••••••=u•••••••••••• ••••========2•••••=::~•••=•••··=•••=••.:.unaa••:aa.:::aai::•••••.,•••1:.aa a:::u:!:~ : :::r : •••••••••••••-ii•:r••it•••=••••••••••••••••:uia11ui::asaans2aa aua::s:::::====••u••••tttt••••=#::::':<ita::;aa:ur:=s-:• w;n::a::; ·· WR# 11utaa "'t#tuuaa ••••~ n1: •tt nnisaaa11••••••••'4•••a•••••••••=*•••••••••••tt••tu••••siaau :a::a;~:===b•n=a•an#:2a:=a:~•~••••=•••••••••••••• •••~••••ra•n"••

····=:itt!C"t••············#••it••·································· ~···~~~~==~~==~···==;••=====~··=·~=·········#==···=··==·~···~··

····~===:r~=················=•t•••···········~······••tt••"······ :r:=• 1:s:: !!lUt-:t= ••tt•••••cuaa.::a:::ii :ia::ai:aaaa:: aauaaaua .. ••= -:•====•==·=•• ••••==#•~~~=••aaaaaa•••••••••••=••••=••••••a••••••••~•=••=•••=• aaaaasu:::a:t: ;;u:aas••==•••=•=••••=.u.u1aa.au1a:aaaaaa..&1. ·aa.i:·.a~••sttaau:·•• •#••==•••••••••••••••••••••a•••••••=•••••••••••••••••=••••#•••• ••••::a=::..:2:~ ·.:::ta.:aaa •••••••a• aaa .. .=:i::.aaa ff &a••••=.u:· •:r::::.iau=li•a•u ~-•• ····=~-~~=···································#················· auaa:s::.n:uns::-:t•••'tl.UJtl••••••••.:t -i.a:uuta•a•t••••••ouui ;aa.:::••••=•=•nn•a:

Figure 6.llf ... Attempt to compute the centroid of the metal piece in Figure 6.13. The shadow of the metal piece could not be reduced to a single line at all.


successfully resolve the blobs within the scene. It can be seen that the blob

reduction algorithm works outstandingly, with the ability to resolve and process

even overlapping finger images. In terms of actual use, this implies that two

fingers, tightly pressed against each other as well as against the overlay can be

recognized as two separated fingers and processed correctly. To conclude,

Minimum distance between adjacent key ectges = O mm

6.3 Interface response time-delay

'Response delay' is the time interval from the moment a finger is

placed on the overlay to the actual code output. The measurement of the

interface response time was done using a Hewlett Packard logic analyzer to

monitor address traces on the host computer actctress bus. Since the program

addresses of various phases were known, the analyzer was used to determine the

number of computer instruction cycles before a phase was completed.

Execution times of four distinct program phases were determined. Each phase

displayed a range of times needed to execute that phase. The four phases were:

Phase l) Picture access and frame digitization

Phase 2) Subtraction of overlay details, noise reduction and

reduction of finger-image blobs to vertical lines {as part of the

centroid-finding process)

Phase 3) Completion of the centroid-finding process, and

6.8


....... •••••••••••••• ••••••••••n•••• •••a.1aa•~•••~•K •~•a~~•~•••u•

-••••••••••••••••••••••••••••• ••••••:: •••••aa:sa •a:ia.caaaaaa11a ••••u•=•••••• •=-=-•••••••••••••• ••••••s•saai;n••• •••:i .1aaua•••• aatt•••••••••• ••••••• .. •••••••• aaaa••••••••••• ••••••••••••• -••••••••••• •••=••••••••••••• •••••:i••••••ttau wuaanoa<1a•••• ••••••••••••• ••••••••••••••••• aaaaaa a ••••• a*t11:•••••••••• ·······--·····=················ ...• ................................ ..... .... ···#~=·=·····

•• ···········=· ... •••••••••••••••••••••••••••• •••••Jt••••••••n ••••n••rl•••••••••••••••--•••••aaaaa••••••• ••••••••••••••• aau..aaaaaaa•n• ••••••••••••••••••••••••••••••• •••••• •• ••••• ••••••••••n•• ............................... ..... . ............. . ....................... ~······· ..... •• • ••••••••••••• ............................... .............. . ••....••..... •••••••••••aaaaaaaaaaaaaauaaaaaaaaaa••••••••••• aaaa;a:aaaaaaaaa ............................................... .............. .

\ ................................. .............. . ............ . .......................... .. . . . . ·········••#••··········· •• •. •• • •• •• •• • • •• •• ......................................................... u •••••• aaaaaaaaaaaaaaaaaaaaa•aaa~•aaaaaaaaaaaaaaaaaaaaaaa&$1Maa#aaaaaa

••••••••••••-•••••••••••••••aaaaaaaaaa••••••a.tinaa.w:aa•••••••••• •••••#•••••••••••••••••••tt•••••••••n••••••••os:i;s,;iaa.,aa.uaau1••••• •••••••••a.a•••••••••••••••••••••••••••n••••• •~•••~••••••••n•• •••••.•.•......•......•....•...•.•.•••.•• . .. ··········~··· ....................................... . ••..•..••......•..• •••••••••••••••••••••••••••••••••••••• •••••••••• • •••••••••• •-•••••••••-•••••••aaa••••••••••••• •••••••••••1t• •••••n•• ················#·········~·········· ·····==·==···· ••••••••••••••••--••••••••••••••••••• •••••••#aaa~xu ... aaaaaacaaaaaaaaaaaaa~•n•••••••••••• aaaaaaa#aaaaaa aaaaaaaa&11aaaaaaaa•:•as••••••••••••••• aaaaaaauaaaaua ••••••#a•••••••••••••=••••••••••••~•• ~~•••#•••••••• aaa•••••aa•••••••••••••••••••••••a.uau •n•&&inu:: 1:•1 :t

.....................................

•••••:t •••n•• •••••• •••••• •••••• •••11••

••••••n•n••••••••••••••••••••••• aa •••ff••:J•ti#tla#••••••••• aaaaaaaaaaaa ... aaaaacaaaaaaaaa a a •••t•a•~••tt•auaar.aa#•a~

•••••~••~•••••••~•••••• •• •••••••n •U•••=#=;a#~a•~•••••*a&• •••••••••••••••••••# aaaaaaaaaaa:• ••uaaau~•aa~a•-•••••••••

••••••••••••••••~••• ••~••••••••••# a#~•••u~n#••••••••••U*•• •••••••aaaaas:saaaaaaa aaaaaaaaaaaaaaa ••••••~#naa•-a~••a•••••u

•••••=••••••••••••••••••••••••a.a#-...JaJ• .au.••••••;••••;sa;;aaaa••••• ••••••••••••••••••••aaaaaaaaaaaaaa#a& ••••••au•aauaaaaaaaa•••• ~••••uaaaaaaaa•••••• aaaauaaauaau••• •••••••••==a••••••••••• aaaaaaaasaaaaaa ... a• aaaaaaa•aaa••aa aoa•aaaax&aa#~aaaaaaaa&al .... ##&•••••••••••••• •~••••••=•~ •••n•••~••#Ua~a•••••••••••

••-••••••••--••••••u:aaa a a au ••.an•••••••••••• aa.;;;aaua••••• •••••#••J&•••••••••••••••• a aaun••a:iwiiaa.;;aa••••••oi••••••••• ............................ ·······••11••••••••4••"~··········· •••••~••••••••••••aaaaaaaunaaaa••••w••••••••••••••••••••••••••• •• .. •••aa:aaaaa .. aaaa .. aa•••••••••••••t1••••••••••••M••••••••••• ................................... ~.#·"=······················· ••-•••••••••••••••••-•••••aaaaaaaaaat1aaaaaaaa•••••••••••••••• ................................... ¢············~#·~··········· •••••••••••••••......_•••••••..!.!.!••••••n•••••••••••.uu_~••••••...!.IL..

Figure 6.15 - Overlay image taken with two metal pieces placed on the ovei:1ay and separated from each other. This separation will be deemed acceptable if the centroids of the two pieces can be computed successfully.

............... . ......... . •a Ult Mit •U W .. .. -.. ····~~········· ········~······ ·······#······· ···~············ .............•. =··············· •.••.••........•••..•.......•••

·········~·~4# ft##. ••3•••••• aaaans••• .. ••~• •••=•:•n••••• ••••••••••••••• •••tt••~•#••=•

aaaan••••aaututn u"11:•w•n••~1::ua

aaaaii:i•aua:aaa•• aaaaaaau•=•=•••• •••••••••••aan.n #aa:~as:.ian•••• ·········••ts••• ................. ············•lt ·······•••:i•• •••••a#aaa1u•••• ••••••••••;saaaaa ass•• a 1 •* • a;sa:·~aJJ••••••

••••••#•••••••• •••.ti••·········· ··········=;·· ..... ... aaaaauaw•••••••~•••••••••••••••• ••=l:f;••~•••••~ ••••• ••• #

•••••••••••••••••••••••••••••••• • :iaa:uinfttt1':tna -••iJ• 1:1•J:

••••~u•••••••••••••••••••••••••• :t;XP :tt~~·~•••u •••••• • a::a ••••••••••cua•••••••••••••••••aa• n-i:•••ti••"r. ·;aw .... •• ••••iuniassaa~u:ri;aa•••••••••••=••••• •#i:&n:i.a;rnttaaa~: an:i•• • #••• ••••••••••••••••••••••••••••:&:•••• aaaaiiaaaaaaau·r u 1•aait•1;•••11•JJ aaa•11u•aaa'&aaa:stt1:ta#aaa•atSUtttt•a#aaaa#aa•aaaaa•fatctt:i:s #1t•t:v .. a#:a•Wllff!

aa•••••••••u••••••••=•••••••=••••••••••••••••art!'• tt itt7t)tttrt~ttn·t•t ••••••n•a:euia•• aaa•-tt11aTW#S.##a•Ma#8S#S#11#tUasaa •••••••••••u••=•••••••• •• • • aa•••=•••ussaaatauaaa•••~tt~R#asuaaa tta

• • • :f

tll:f; it• ## ~ • •• ••••••••••aca••••••••••••••ttnaaaau.-.aa:tt • aaaaat'lt: 11••:t•Rr,,.wana iiaa ••••tt• •••••••••••••••••••*' nti••=••••:i nan••••• o: #J! • = ~1'Jta·: !Ja··;.=•••nn ••••••ir••••n••••••••••••••••••••=•••••.c •••••••11 ·:••~' .,••Tt tt '?I!•••· a11 •••••n••aaa11assaa:iuwaawait•11aaauu-auaa:1 ust;H1tl :s••· :•;i, oiaaaa:;:;':u:a11"11•••:' aa••••••••it••••••••••••••••••tt••••••••n•••••r.&tr' >t"1:a•••nisa11•11A• •••••••••••••••••••••••••u••••••=••n•st•••••• s •3••~••••n•••• aaaaaucraaaaaaaaaaaaauaaa••••••••••••••r '1 1aar!a • ., •1a1ta.awua• aaaa#ua••••••••••••••••w•~••••••~•••••• •••~•••=•#a ••••••••n ............................ #••······· ••••=••~•••••••••••••••••n•••••#•M•n••u

·································=•••#~ ······=···················~········=·"·

······································ft ••••n•a••••••••••••••••••u•••••••••·:-.an aa•••••••••=••••••••••••••••••••••••••u •..•....•.........•....................

8aaaattr:a•aua• •aa'°v.aaaissaaaa ········•••tttl.118 •=••·:wJJ1tacl:f;•••••

••••••n::•anM:•1u1 ;iaaanasaaaaaaaa aa•aaaun•aac 1ui

••••••• • •

• •••tt•• ••••• •n.••• ••••• ••••• ••••• ••oi•• ........

•••••••••••••••••••••••••••••••••••••• • •••••••••••••• •................••................ ··~·················~ ·=·········· .. ·····•···•····• .• • •....... u,, ••••••••••• ••••••••••••••••••••••• a •••••••••• ••••••••••~u•••••••••• •••••••••••••••••••••• ••••••••••••• aaaaaauav.u:tuan•••••n•• aaaaauaaaaaaaaaanaaaaa aa•aaaaaaaa•aaa aaauaaaaa:a•uaaa&##&&a a·aaaa:an•••••••••••••• •••••n••••••••• aaaan1·:,11aa ·:aanJ11•••..,::1a •••••tt•=••••••••=••••• ••=••••••••••••# •••~••~••#••~w•=•=•••u aaaaaaaaaaan•••••••••• •••••••••••••••• •••a.nun•au;.i !C•aa:~••n ;:•• ••••ii••••••••••••••••• a:":•••••••••ca••• •••~•tt••••aaOf.1:1a;iJn1uaaa aaaaaaaaa.aaaaa11aaaa#aa •••••••••••••• ••••n•l;!a•r. ·11•••11 irna11wa ••••#=•••••••=•••••••••• •••••••• •••w••~••••••••~••~•u#aaa

••••••••••••••••••••••••• a •• •••••u••••••• ': oi:ia.111.a•tt•''""*'';•• •••••••••••••••••••••••••*' •• •• =r•• n • •••• 1t.n•;uuu1~•n.••:1irna1a • •••••••••••••••••••••••••••••• •tt•••iT!'l•ttn••• tt•••N ·:wa:ran••it <t•• ••••••••••••••••• ••••••••• ri•••••*l••••.u n•• •ii aa iia;ra11aa:a:a;;.a1'1a:iaaan •••••••••••••••••••••••••••••tt••••••••tt ••••••• :J ·~ttur.1:aa••••••r- aa •••••=•••••••••••••••••••••=••==•na·•an 1111u1aanwn•••=••!taa1tn.i;aa•n aaaaaaau•••••••••••••••••••••••aau:u•••n taanaaaan••nttnwaan•••••• •••••••••••••••••••••••••,•••••••••~•nnan•••=1aa2aaaua~••••••••• ••••••••••••••••••••••••••••••••••••••.naaaaaaaa11111:n·ra•••a••~.1!.••

Figure 6.17 - Overlay image taken wit~ two metal pi:ces placed on the overJa~ and touching each other. This separation of 0 mm. will be deemed acceptable if the centroids of the two shadows can be successfully computed.

aaa-aaaa;saa:aaa .. aS·a11w••••••$.•••••••••••••••••••• •••• uana••••• ••• ············•······•···········•······•···············•········ ............................................... k .............. .

••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••••••••••••••••••-•••••••••••••aau•••••••••••••••••••••••• ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••••sa••••--•••••• .. •••••• ... •••• .. •••••••••••••••••••n•••••• ............................................................... ................................................................. ····#······· .. •··•···•·••·····••·•··•····· .. ·········•····•···· •••••••••••••••aaaaatta:••••••••••••••aa .... •••#••••••••••••• ............................................................... • ..................................... rt••···················-·· ............................................................... •••••••#•••••aa• .. •••••••••••••••••••••••• .... •~••••••••••••~•• ..•...•.....•••.•..•...••.....•....• .. ................ .. ............................................................... ·························•1••··················--·············· aaaaaua••••••aaaa••••••••••••••••••••••••••••••tt•aaaaaa•••••••• ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• ••••••••••••aaaaaaaa•••••••••-•••••••w••••••••n-aaaua•••••••••• •·········•·•·•···•···•······•······•···•······················ •••••••••••••••••••••••••••••••••••••••aaaaaaaacaa••••••u•••"•• .....•...........••.••.................•.....................•. ···································•••tt••······················ ............................................................... ............................................................... . ............................................................. . •••••••••••uaaaaaaaaaaaa•••=•••••••••••••aaaaas•••••••t•••••tt .. •• .. o•••••••••••••••••••••••••••jf•••••••••••••••••••••••••••••• aaaaaaaaaa;:i#a-aaaaau .. a••••••••••••••• •••••aau:r•••••••••••• rt•• ............................................................... . ............................................................ .. • ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••

' ........................ =······································ ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• : ••••••••••att1taaaaaaaaaaaaaaaaa••••aa•••••••••••naaanaa••••••••• . ............................................................. . ........... " .................................. #~···~········---••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• ····················~·········· .. ··········--········ .. ········ ............................................................... •••••cc~•••=-•••••••••••••••••••••••••naaaaaaauuaaaaaaaaaaaa"8a

au••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• aaaaa#••••••••••aaaaaaaaaaaaaa••••••••••••••••••••••••••••••naa ····~·······························--·····--·········· .. ······ •••••••••••tt••••••••••••••••••••••••••n•••••t••~•••••••••••••••

····~·················································· .. ······ •••••~•••aa#aaa•••••tt•••••••••••••••••=••••••••:•aas••~•~•••"••

=······························································ ••aaaaaSau:•••••••aaa:i; ••••••• n•••••••••it 11•••••••1,aaa::s•••••••• !fas ····=··················=······································· aaaaaur.••••••••••••#=•••••••~•••••••••~••=•••••••2••••••••••••• aaaaanaa••••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••••••••ttaa••••••••••••••2:aa••••••a••••••••=•••~••••••••••• ................................................................ aaaascaaaaaaaaaaaaa•••••••••••••••••••••••••••••••••••••aa••~••

Figure 6.16 - Attempt to ~pute the centroids of the two metal pieces in ---FiglD"e 6.15. As can be seen, the two shadows were successfully reduced to two separated lines.. The midpoints of these Jines are the centroids of the two shadows.

C•Sa.-.;;;;caa•SWaaa#aaa##a&aa••••••••u~~u~nYaxr.•r:n~~-:--::r~r.-:nnr~.-.

aaa•••••••••••••••••••••••#aa••••••n ••••••••unw#aa;;..w•~ ••••••••• •••••••••••••••••••••••••••••a:aaaaaaaa • ••••••••1'.awuuaun•••••••• aaaaaaaaaaaaaaaaaaa:aaaaaauaaaa••••aaaaaaa•aaaa#aaaaa#&aua•aaaa eaaaaaaaaa••-•••••••••••••••••••••••tt!i •.:t•••••tt!:aatt u••••••••naa aaasuaaa•••••••••••••••••••••••••••t1aaaaaaaaaaaa•••a##-:a•••••••• ................................................................ ••••••••••••••••••••••••••••••••••••••••:&J•••·1•a#aa•a•-in•a••1'••• •••••••••••••••••••••-•••••••••••••••~••••••••n•••~••••tt•••••• ••••••••••••••••••••••U•••••••••••••• ••• ••••n•••~•••••••••••• aaaa .. ••••••••••••••••aa•••••••••••••• a ••anttaaanii••••••woaa •••••=•••••••••••••••••••••••••••••n••••••••~••••••n•~•••n••••• aaaaawaaaaaaaaaaaaaaaaaaaaaa••••:r:••••••n•aaaauuaa•tt••ii•••••tt•• ••••••••••••••••••••••••••=•••••••••• ••••••••aaaaaa##aaw•••••• ...................................... * ·····~········~······ •.....•••.••...•.••••••.•••..•••...•..•.• ··········#········· •••••••••••ac•••••••••••••••••••••••••••••••••~1:aaan••••tt••••••

·····················································=······••tt ••••••••••• .. ••••••••••••••••••••••••••••••••••H•••u••••n•••~•• ............................................................... ••••••••••••••••••••••••••••••••••••••~••••••••"••••••••n•••••• ••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••• •••••••••••aaaa•••••••••••••••••••••••M 1••••••11 :~aaiiuuaa;ia•••t1•• ••••••••••••••aanaaaaaaaa##a•••••••••••••••••••••••••~••#•••••• ••••••••••••••••••••••••••••••••••••••••••••aau:;aa••n•••••1•11a• ••••••••••••••••:i••••••••t1•••••••••1;.w#1Cn:r:a•ii·t••••••••ttaa.auaasan aaa .. aaaaaaaaaasaasa•aaassaaaaaaaaaaaa:U)Jttaaaa•ttU!' tla•.u•1•:i·1•u11 IJla

•••••=•••••••••••n•••••••~•#•••••••••#aa~•••~••=••n•••~4u••••~# ••-••••••••••••••••=••••••••s••••••••tt "'•••••tt 1t 11 a•n aaa.iurn,uaa na• •••••••••••••••••••••••••#•••••P••••n~=•••••=••••••••~•••#~a•nn aaaaaacu1aaa••••••••••aa•••••••••••••attl' 1t•n•••Mn"1anu :~it•ii•n••n ••• aaaaic:a:a;r.aa••••••••••••• iia :: i:••••••••U•••••••••uu1;;a;u:•• ;f; •••• •••• n ••••••••••••••••••••••••••••••••••••••11•1•••••i.1n•anll :1nn.- lilf••n•t aa•n••••••••••••~••••••u•~•~••••••••••••••••~•••••••••••~=••••n ••••••••••••--••••••••••••2••te••••••••:a •••••itoiua•••u•••i:iaaac 11•• aaaaaaa•••••••••••••••••••l••••••=••~=•••=~••~•waanna•"••••••••• ···········a:••••·············=···············••1:••*f•••••••••11•• •••••••••••••••••••••••••#••••••••••n•=•••••n••••ft•••••••••••~n aaaaa#aaaaaaaaa•aaaaaaaaaaaaaaaaaaaaaatt1aa1aaa"11aaattaa1R11•aaa1a aaaaau•••••••••••••••••~•• =saaait1ana.~u11uaa••••••••••••~: ••tt•••••n •••••••••••=••••••••••••••aaarea•au•••• •i ••••••au 11•• u•t••••• ••M tJll •••••••••••aa#aaawaaaaaaanaaaa••••••~••••••••t••,••n••~••••#••••

aaaaa:aa•••••••••••••••••••••••••••••••~~•••••nun•suna#a•~••~n•~ aaaa•#•••aaaaaaaaa•••••••~•::ta#a#a 11a:."t11 ;s:ua.n:3s;:;uui ;;ais;;u 11 ~.;n:aa ~3•1'111

aaaaaaasa::aaaa••••••••••••••••••••••••r. ~••••••a ::iaaanawn ;uu1ai.-nna aaaann.a••••••••••••••••tt••'t#•••••a•••n1t1tt•:t•=•ii:;ic.#wawi=.rta1uaa11att aaaaaaa••••••aa2 aa••••••••••••••••••••naaaaa1t1t!l ::a:tu1t••1nx111a""' aaaauaa~•••••••••••••••••~••••••••••••••••••:••••••••~#•••••••• aaasaaa•••••••••••••••••••••••••••••••• •••••••H ·:•anaattannwaan.aa aaaa#aaaaaaaaaaaa•aaaaaaasnaaa•aaaw=~·=•#tS•#aaa•aaaS~#aaaaaaa~

•••••••••••a:i•••••••••••••••••••••=••••:!•••••ttstn= a'Jf111••r~!1r.saanaa aaa•a-:sia:a Raaaaaaa#aaa•aa•t1•:aaaa•aaa~1211c:aa••••aaaaatt##l•ll#fr1l#&aaa

·········••U••·····················•••nJt•••••nH l#A•H•••ttfit••••• ••••••••••••••••••••••••aauaa•• n rt••• ac na11 ••••w••••••••nii1nw.aaa~a ••••••••=••=•••••••••••••••••~••••••••!'! ra1J1aaan•1•at1·•••••••.aa1111 ····=•••tt••···············~:••n•••••••••••••=u~UUH••·~~·••R••·~

Figure 6.13 .. Attempt to compute the centroids of the two metal pieces m ---Figure 6.17. As can be seen, the shadows were successfully separated. The centroid of each of the shadow can now be computed.


••••••••••••••••••••n••••••••• •••••u••••••••• v••1.•~•==•••u• ····=···········=••#••········ •·•··•••·•••••• ·····=········ ••••••••••••••••••au.a-aaaaaaaaa •.tt•••••••••••n riiutst•:r•••u•;r.tt• •••••n••••••••aaeaa3••=••st•••• ••••••• •••••• D#a~n•~•"••••• aaaaaa•aaaaaa aaaaauaaaaaaaaaa a•aaaa ea aa••a•Q'UD&S#I& ••••••••••••• •••••••••••••••• •••naaa •••••••••••••••••••••••••••••• • ••••• ft

······#·········=············· •••• •••••••••••••••••••••••••••••• •

aa aaaa•uaa•#aaaa •••• ···;·•••u..••••t1• •••• •••••~•••u••••

••• ••*••···=····· •••a3aaana•••••••••n•••••••••• ···········=··················

•• •••&a aa:iua#21aa+u•••• aat:rt1aaaai11aa1a aaaai.:.uaua•••• •n•-" ••••••• ······················=········ ....•...•..••. •••••••••••••••••••=••••••••••• ••••••*I••••••• anu~a•aaaaa#aa

•=••#••••••••••••n•••••=••#•••••••••••••#••••• anaaas#a:aa••• aeaaaaaaaaauaaaaaaaaaaaaaaaaaw•aauaeaaaaaaa.aaa aa#Uaa.s.aaita#R8

••••#••··········#············· ....•......... ··~·········· ·················••:•••• .. . . . . ••••••••••••••••••••••• •• •• •• •• • •• • • • • • • •••••••••••••••••••••••••••••••••••••• ••••••••••••naa#•••••••• ·····~···=······················•••tt••·············=····~·#•••• ••••••••••=•••••••••=•••••••••••••••aau••••••••»•auxaanii:i•a•uaa aaaaaaau•aaaaaaanuaa••••••••••••••••••••••••=••••••••••••=••••• aaaaaa•aaa•#aaaaata#aaaaaaaaaaaaaaaaaa•aaa#aaa~UaUW•aa#w•aattn .. .......................... rt••··········· •• ·····=··~······· •••••••••••=•••••••••••••••••••aauaaa• ••••• •~"""~••a#aaa ••••••=••=••••••v••••••••n••••••••••• •••••=••••• ••annaa11a ••••••••••=••••••••tt•••••••••••••=•• ••••••ru111•a• 'l'f auattnaa aaaaa#aaaaaa•aaa#a~aaaaa#aaaaaaaaaaaa •aaaaaaaaaaaaa #a#88R

••••••••••••••••aaaanaauaaaaaaaaaaaaa •••••••naa1aaa •••••• •••••••••••••••••••••••••n••••••••••• •••••••~•••••• ••••••• •••••••••••••••••••••••caaaauaaaauaaa aaaaeaaaaaa#na eaaanaa ••••••n•••••••••••••••••••••••••••••• •••••U••••••• ••••••• •••••••••••••••••••c••••••••••~•••••• •••••••unaaaa R•an••• ••••••tt••••••••#•••••••#•#··········· ••• ••• ··#····· aaaaaa•aaaaaaaaaaaaaaaaaaaaaaaaaa#88 a aauaaa••aa#aaa ••••••••••••••••n••••••••••••=••• •••••••a:saa;;•ati•#••.u.• ••••••••••••••••••••••••••• a ••••••aauaa••'**'"•aaa#ioaa ••••••n••••••••••••••••• • •••••• ••••••••••#•u•=••••••••• •••••••••••••••••••na ••••• •••••••• •••••••••:tau•••• 1i•it•• uaa ....•.••....•••...•• .•.•..•.......• ··············~·····"··· •••••••••••••••••••• ••••••••••••••• ••••••nr n"l•tt 'l•••un•t:ttuaa ••••••••••a••••••••• •••atta#a•••••••• ••••••••••••••#n:~aitaaaa ••••••••••••••••••••Q••=•••••~••••••• •••••••a•aauDaauuuaattn•• •••aan#aaaaaauaaasaaa •••=••••••••••• ~•••=•=auau:#~•=•••••••

..•••.............. •••....•.....• ···········=········=· .. ·=···=·············· ..•.•.•. . ...••......•.•.•.......••• •••••••••••••aaaaaaaaaa • ••••• ••••?t•••••••MH••••••annaaaaa• •••••n•••=••••••••••••••• ••=••••••••••••=••••~"•••it•••• •••••••••••••••••••#•••••••• •••••••at1 =•••••••••••••••••nae u•• •••••••••••••••• n••••••••n •••••••••.Ht• .-a n.na•n •u•aaa:a.a:a;u•••••w• ••••••••••••••••••••••••••••••••••••••ft••••••••K~••~uaaacaaaaaa •••••••••••••••••••••••••n•••••••tta.J•rt•••••••••••••st•••••••=••• ••••••••••••••••••••••••••••••••••••aauawaaaaa:1 ti ••n Ll•••nn-aa•••• •aaaa#a•#aaaaaaa.aaaaaa••a•aa•aaaa•:lltR~Wn•a#Aa#XaaSaaaaaaa.aa•a

··············!~~!...•!_!_••··············· n.•••••••• 11••• tt• ·= ;i•••• .... Figure 6.19 ... Overlay image of two metal pieces placed on the overlay and overlapping each other. This overlap will be considered acceptable in normal use if the centroids of each of the shadows can be computed successfully.

aaa#n:s1:usn.••••••••:i•••••••a11na••=••••·~an••••••aa• ::u:u:;; .i.1,u• oin•••• ••••••••••=•••••••••tt•••=•••••••••••aattn •••••••a•au 1· azra;;:::sa#n:aa •••••=•••••••••••••••••••••••••••••U••••••••=•••••••u~•••••••#• ····················•••U••••u••········ !UI IUUl!ttl '· ttttaf'·~.,1t1• ••ltllW#lfn• s;a•••••U:••••••••n••••••••..i.ttuaaaaaaaaw4aa;s•a•n•.a:alf:#J;#J1Uli•.?#1'H•tttt ••••••••••=-•••••••••••••••••*•••••••aa:11•a•••••t1 tt••'t! ota• 1tn•wis111rna •••••••• •••••-"••• •••••••waii.aa•••••••ll••••••nn P.ft r: • i1ntt• ii w•::i:•n•••• ••••••••••#f••••••••••••••••••••••-"'••••••••••••n•••r. i?rtU••••an" JU'

••••••••n•a••tt••t•••••••«•••••••••••••••••••••••••#•~:t••~•••••• •••••••••••••••••••••••••aaai:aaaaaaaaatt•••••••ta nanrtttwf!attu1att•;a1t

uaaan•••••n•••••P••••••••••••••• ••• • • ••it•~•it••~•••=••••~• •••••••••••••••••••••••••••••••••••••• •• ••••t' •••fl na•ttn•••n :: rte aaa•a~anaaaaa##~~aaaaaaattwaaaa•aaaa~ftW#aa•aaaa~•at•U#att#:#•###S#

•••••••••••••••••••ti••••••••••••••••••t"•••••••••••• 11 •.ua:awan# !ta• •••••••n•iuraaatta:rttaaaaintaa:tua••••••••••• a ttJll..ti'"l•t:i••••••:z,,.na••• ••••••••••••••••••••:~••••••••n•••••••••Ra'aaa•••••••u•••••••n :·a• •••••#••••••••••••••••••••~•••=••~••~•••••••#••~•~·~•u•~n"•~~•n •••••••••••••••aaaa#aeaaaaaaa•••••••••• ~•11••••" 11•a:. 1ut•"•n••tt•n uaaats:ita•••••••••P••••••••n•••••••aa:iasan•••• "";s •if •=••••:J:•tt•r IUtata •••••••••••••••••••=••••••••••••••••••••••••••~ a•ttu 11unauwa•n rtntt ••••••• n••••••••"••••••••u 1uuraaanrra1: n •11 •n•••=-••:t1nu: ••t:11•••••n•" tt:aaaaauaaai:•••••••••tt•••••••"••••••••.ar ~••nt1 "an Ji lla.P :r;ir:np;r:ftnr.:· ••

••••a-~•••••=••••••aa#aan•~•••••••••••••it••••#~n•••••~~·~·•~••=•

•••••••••••••••••••#••••••••u.•••••••••~ ••••••tt•H•!tff 'tR;;aitu••ir-11aa ••••tt•••n••••••••••••••••naaa:aa•••••••••••••••••••••~••••#aaaw

•••••••••••a:•••••••••••••••••n••••••••t< •••••••• tt••• i! u rt••••• n:; •• aaaa#U•~aaaaaaaaaaaaaaa#an#aaaaa•tta#aaaaaasaaaa#tta•••=aa•aa•a•#

•••••••••••••••••••tt••••••••••••••••••• 11aa.ua• iii:i n•arr tinrra.w.na•t! r~a• aaa•a"aaaaaaa##•a:aa•a##aaa#aaaa#aa•aaaaaaaaaaaaaaaaa•~a~tt~aaau

aaaaaaaaa:aaaaa•aaaaaaaa•aaaa:•••aaaaaaattuaaaaa#a#&a:I! :ia••U'f"nr: 'lP•

•••••••c•••••••••••••••ft•~'••••anaaattn•••••••••••••••~••~•••n;1~

•••••••••••••••••••••••••••••••••=••••~~•••••••Hnau~un~nwaann••

•••••••••••••••••••••••••n•••••••••••••••••••••••4••••••3=••••• •••••••••••••••••••=• .. •••••~•••••••••tt•••• ••ar1•11~"•~•n~nn~•• ••••u.••••••••••• 1t•••••••• 111uta••••••••••nnn•a~#• :iftn.ti.#f t:#;:S#JJ#ann

•••••••••••••••aa•••••••••••••••••••aa••••••••• •••U :·n:r.•n•:r•r.r;K• ••••tt.naucaaaa#aa1S.asut••••a11 n•a•••••• •••••• aa111; rs•••U#1':fl :t•o::itt 1:;s Ka.a ••••••••••••••••••••••••••••••••••••••-" ••••••an1.anft :1nu?T:a•••tfttaft ••••u•••••••••••n••••aa#a;1•aas•a•••~••••••••n•~••••••#•••~•••n•1

••••••••••••••••••aua:••••••••••••••••aii 11a••••••~t•att:iau•na ••n 1: •• ••••waauna-tt•••••=•••####a1: ttitaa#ai.a#a If Q:#il an »Rt i= ;ar. :i 11•w.·1i;~rst:s:ai:•••• •••••••••••••••••••=-=•••••••n a:aa••••••!?W••~ 1.11111 ~-••n;tna;r1; Pft"nn :; aa aaa-au:urn•••••••naaau.#a#tiiut••=iucai! tt11· nu •11 n1t11,; :iu•ruuusu:aaa•tt#n lt!lrr •••=•••••••uaa:aaaaan:rt••••••••=••••••••t'l·tttttta•r !! lf1lni;: :· 1tt>N"itrwa11 :i aa aaa•att lttl •a~#att.aaaaaasas:aa-a~ !t aa#~tt• aa.i:: taaaaaaaaaaaaaa~#a'tC•tUta:t#aa fl •••••••a•aauaaaaaaaat• .. •••••••••••aaaatt11aaaaaaa1raat1 :••::n:i1;;~u;: ·· rsn: •••••==•••••••••~a•a~••n••~~•••••••••••••=n•~•=~•==••~••ft==••=~ ••••••••••••••••••••"••=•••••,. ••••••••t:••••••iir:; ~= u: •u=•=••tt!:::ta ••••••••••••••••••••••••••••ai1aaia:aa:usa•••ii•••••••••:!.a21:11::~::~! •n•stn aaaaaaaaaaattaaaaaaaaaaauaasaattuaaaaaaal"••••••Wt.i :raan 1:1:.::•:it1f2'1f r ~;;

•••••••u••••••••r.••••••••~itita••••••~••••••••4••••••••=•••=~#••~ •••••••••••••••••••c!taaaieaaaaaaaaaaaaa~ '*••••••" r: ltt11>l11•••r:t1r •·tt• ••••ttuaaaaaaa:aaaa ••••••••a·trt••••ft••ts a••= •••a~ua:::a1.ura:1 :i-a:ia11•a11u •••••••••••••aaaaaaiutaaaaa.aaar.•••••••attaaaaaaaa1ta•i2 aasaa:<.a•a1iaa •••••••• xaa•••••••••••••••~na•••••••:t:••••••• • nai:a••"lts.i:an•.tita#tt •••••••••••• .. •••••••••••••••••••••••cva.aaaaaaa11a•••·•~..!_~••1•••

Figure 6.20 - Attempt to compute centroids of the two shadows in Figure 6.19. Note that the two overlapping shadows were successfully separated.. 10e centroid of each of the shadows can now be computed.


determination of keys encircling such centroids

Phase 4) Code output corresponding to 'pressed' keys, and book

keeping.

Phases I and 2 are the same as those described in Section 6.2. The

overall response time is also a function of the number of fingers pressing keys

simultaneously as a result of the increased computational load on the host

processor.

Response time as a function of program phase:

Minimum Typical Maximum Objective

(all times in milliseconds)

Phase I 8 16 24

Phase 11 50 50 50

Phase Ill 16.5 16.5 42.5

Phase IV 8.1 8.1 36.5

Total 82.6 90.6 153.0 50-150

Response time as a function of number o:t fingers simultaneously used

No. of fingers

1

2

5

10

6.9

Minimum Typical Maximum

(all times in milliseconds)

82.6 90.6 153

86.0 97.0 181.2

101 149.5 306

137 229.5 500


An examination of the software shows that Phase III execution time

changes most with the number of fingers used simultaneously. All algorithms

that do not access the raster sequentially are executed here. (Phase Il, in

contrast, applies a transform on the entire raster, scanning left to right, top to

bottom, regardless of actual image content. This implies that Phase 11 will

execute in a relatively invariant amount of time, which it aoes). With an

increased number of fingers, the number of direct pixel accesses increases

dramatically. In fact, an analysis shows that within Phase 111, almost 4U% of

the execution time is expended in computing addresses and masks for direct

pixel access. Chapter VII shows how the overall interface response time can be

improved considerably with some change in hardware.

6.4 Typical Applications of the interface

The interface can be used in a variety of modes and appllcations. A

few specific examples are listed below:

Electronic Drafting

Hardware in current drafting systems comprises a video display,

keyboard, digitizer tablet and 'intelligence'. If the FHMI was used, the

keyboard and digitizer could be replaced with the advantages of multiple entry

of the keyboard and visual convenience of the digitizer. Different overlays !or

different classes of symbols (Civil Engineering, Chemical Engineering,

Electrical engineering etc.) would make it much simpler to do electronic

drafting because the operator would not have to keep alternating between the

6.10


digitizer and the keyboard for entering symbols and control functions. Also, he

would not have to memorize keyboard codes tor control functions oecause each

key would be directly labeled by its function.

Music

Another example of use is in a computer music system. Various

overlays sold along with the synthesizer could allow the Duyer to create a

variety of instrument keyboards. This would imply ease of learning and true

system flexibility for the buyer.

Control Panel Simulation

Human-engineered panels/keyboards can be designed using this

system. Key configurations of a panel area can be changed within mmutes Dy

simply redrawing or changing the overlay. This implies a 'paper design' of a

front panel or keyboard could be easily iterated several times to arrive at an

optimal solution. Therefore, an organization could create intelligently designed

and user tested panels at a fraction of the cost of the existing cut-and-drill

techniques.

Operator - Training

In order to teach a new operator the use of, for example, a

computerized industrial controller the FHMI could be used to advantage. A

'training kit' would consist of the interface, several overlays with key

6.11


configurations of differing complexity drawn on them, and a reference manual.

This way, a self-paced learning system could be instituted.

6.5 Conclusions

Tests done on the interface have been described. Results of these

tests have been interpreted and linked to a set of rules for creation of overlays:

the minimum thickness of any outline on the overlay as well as the minimum

dimension of a key is J.l mm. The minimum distance between two keys is 0

mm. The average time response of the interface was measured to be 80.6 ms.

for a single key closure. The digitization threshold adopted was 1.5.

Some specific applications of the interface have been identified. It

is also hoped that these examples of use will motivate the reader toward other

applications of the FHMI.

6.12


Chapter VII : Conclusions

7 .O Introduction

This chapter discusses possible enhancements to the interface and draws some

final conclusions.

7 .1 Possible enhancements:

The interface could be enhanced in three areas: introduction of

keyboard feedback, reduction of the key closure - to - output time delay and

improvement of the overlay -image resolution. This is not to imply that other

enhancements could not be made, only that these areas wer:e felt worthy of

attention. The other enhancements as envisaged by the author, were either

trivial and/or cosmetic in nature. Five students, chosen from the Department

of Electrical Engineering, were asked to use the interface while their reactions

were studied: they all essentially recommended the enhancements mentioned

above.

Keyboard feedback is essentially the process of letting the user

know that his action of 'pressing' a key has resulted in a key - closure. It is

important in improving user -confidence, since it allows a user to easily adapt

to the different cnaractenstics of d1!ferent systems. Feedback relying on the

sense of touch to communicate the closure information is referred to as 'tactile'

7.1


feedback. It is normally accomplished in keyboards by creating some resistance

to fingers pressing keys. Some ideas on creation of such feedback are laying a

transparent 'bubbly' surface between the keyboard and the overlay. This

transparent surface would contain air bubbles much like the common packing

material. Everytime the overlay is pressed, some resistance to the action

would be felt. Another idea which is similar consists of tightly stretching a

transparent membrane a few mm. above the keyboard. 'fhe overlay would be

placed over this membrane. The distance between the transparent membrane

and the keyboard top would be greater than the proximity threshold of 1 mm. so

that in normal use, the user could rest his fingers lightly on the overlay without

activating any keys. Pressmg down firmly would cause that key to be

activated. Audible feedback is also fairly effective. A synthesized 'click'

could be sounded everytime a key is pressed. This is akin to the sound that

some modern keyboards artificially generate to create a 'positive feel'.

The second enhancement recommended is reduction of the time

delay from a key depression to code output. The average time from depression

of up to five keys depressions to code output was within the 150 ms. upper time

limit goal of Chapter II. However, a speed improvement would be required if

more than five fingers are to be used simultaneously. This goal could be

achieved by a combined hardware and a so!tware re-design. The following

analysis outlines the speed improvement possible in each of the four operation

phases described in Chapter VI. The first phase (the image acqu1s1t1on phase) is

executed in an average time of 16 ms. A considerable speed improvement could

be achieved by speeding up the raster scan rate. The new CCD cameras now

available can achieve scan rates that are four times higher than the RS-330

7.2


video standard, or in other words, 4 ms. per frame. This implies that this phase

could be executed on the average in 6 ms. This would involve a re-design ot the

camera interface and the synchronization pulse stripper block. The second

operational phase, phase 11, in which Image subtract10n, noise reduction and

partial centroid computation is done is executed in 50 ms. Using the image

processor mstead of the host CPU to perform these functions would mean a

reduction in computation time to 4 ms. ThiJ> change involves an image

processor microcode re-design. The centroid finding process commenced in

Phase II is completed in Phase Ill. After that, the keys encircling these

centroids are determined. This phase takes 16.5 ms. to execute. It is in this

phase that most of the pseudo-random pixel access occurs. Using a faster and

more capable CPU, it is estimated that the time could be reduced to 3-4 ms.

This would involve a hardware re-design to interface to the new host as well as

re-implementation of the software to execute on the new host. Control passes

onto the last phase once the encircling keys are determined in Phase Ill. The

actual code associated with the 'pressed' key is transmitted and some book

keeping operations are also done in a total of 8.1 ms. Agam, with improved

computational facilities, this time could be reduced to 1.62 ms. This would

involve the same modifications as in the previous phase. In addition, Phases III

and IV could be executed in parallel w1th Phases I and II, reducing the average

response time to the greater of the sum of the times of Phases I & II and Phases

1ll & IV respectively. Thus, for single key depressions, the overall key-closure

to code output time could be reduced to l 0 ms.

The last enhancement suggested is improvement of the overlay

detail resolution capability of the FHMI. The current minimum acceptable

7.3


thickness of any outline on an overlay is 4.6 mm. under worst case conditions.

This could be considered excessive if a cluster of closely pos1t1oned keys was

needed. Fortunately, this improvement is relatively easy to implement by

increasing the sampling rate to 640 pixels per scan line. This would reduce the

minimum thickness requirement to 1.8 mm. However, a penalty would have to

be paid in terms of increased storage needs U 0 kbytes versus the current 4

kbytes) and computation time (roughly, a factor of 3). A similar case can be

made for the current minimum inter-outline space within keys. The

improvement suggested above would automatically improve this parameter

from 6.2 mm. (worst case) to 'L.4 mm., and from ..:S.l mm. (nommal) to 1.2 mm.

7 .2 Condusions

The interface could certainly be further enhanced along the lines

mentioned above, and perhaps in some functions not envisaged by the author.

However it is felt that the current implementation has been a success; it is

useable in any of the applications suggested in Chapter 6, as well as other

applications in which there are at most five simultaneous key depressions. This

automatically encompasses all keyboard-like uses and many other. What is

more important however, is the proof of viability of the concept of using a TV

camera to create a powerful human-machine interface. The concept opens an

immense number of avenues of application, which are likely to prove invaluable

in the next decade of increased computer use.

7.4


List of References

1) E.B. Montgomery, "Bringing Manual Input Into the 20th. Century: New

Keyboard Concepts," Computer, Vol. 15, No. 3, pp. 11-18, March 1982

2) S. Hull, "On Categorizing Input Devices," lecture notes in CSS 25225,

Department of Computer Science, University of Toronto, Spring Term 1980-81

3) N. Mehta, K.C. Smith and F.E. Holmes, "Feature Extraction as a Tool for

Computer Input," Proc. ICASSP 82, pp. 818-820, May 1982, Paris

4) K. Ramakrishna, "Wipe Keyboard Slick but not Practical," Computer, Vol. 15,

No. 8, pp. 97-99, August 1982

5) S.K. Card, W .K. English and B.J. Burr, "Evaluation of Mouse, Rate-Controlled

Isometric Joystick, Step Keys, and Text Keys for Text Selection on a CKT,"

Ergonomics, Vol. 21, No. 8, pp. 601-613, 1978

6) A. T. Welford, Fundamentals of Skill, London: Methuen, 1 ~68

7) D.B. Devoe, "Alternatives to handprinting in the manual entry of data," IEEE

Trans. on Human Factors in Electronics, Vol. Hf'E-8, No.I, pp. 21-31, 1~67

8) J. Mick and J. Brick, Bit Slice Microprocessor Design, New York: McGraw

Hill, 1980

9) N. Mehta, F.E. Holmes and K.C. Smith, "A Fast Image Processor," Proc. IEEE

Conf. & Exp., pp. 242-243, October 1981, Toronto

10) P. Stucki, Advances in Digital Image Processing, New York: Plenum Press,

1979

R.l


11) E. L. Hall, Computer Image Processing and Recognition, New York:

Academic Press, l 'i79

12) M. Brady, "Computational Approaches to Image Understanding," Computing

Surveys, Vol. 14, No. 1, pp. 2-71, March 1982

13) R. Cunningham, "Segmenting Binary Images," Robotics Age, pp. 4-19,

July/ August 1981

R.2


..

Appendix A

Calculation of the Minimum Microcycle Time of the Image Processor

A block diagram of the Image Processor is repeated here in Figure

A.O. It is easy to see that the minimum microcycle time is the greater of :

t1 = tcik(pr) + talu

or t2 = tcik(sr) + tmux + tseq + tacc(mem)

where

tcik(pr)

talu

tcik(sr)

tmux

tseq

is the clock-output time delay through the pipeline

register

is the control-output propagation delay through the alu

is the clock-output time delay through status register

is the input to output propagation time through the

condition code multiplexer

is the control-address output time delay through the

microprogram address sequencer

tacc(mem) is the access time of the control store

These values were taken from the 1'::182 AMO bit slice components data book,

and the two times t1 and t2 were derived to be

A.l


250C 0-500C

64

103.5

89 ns

125 ns

It can be inferred that a clock period of 150 ns. would result in reliable

operation over 0-500C.

A.2


... .....

MUX BUFF-ERS .... ~ ..

~ .. .... + SEQUENCER DEC . .. /-'<. •• A+ 1 \-I

~ ,.

-CONTROL STORE

l{A + 1) 140 PIPELINE REG ..... I.'

I l(A) I SYSTEM ~r BUS

.. .. TX REQ ~ ALU

SYNC+ I S(A) ... ... TX ACK

STATUS REG

I S(A - 1)

Figure A.O - Image Processor Block diagram

A.3


Appendix B

Details of Software Implementation and Tools

The software for the host processor was written in assembly

language to achieve the best speed possible. The development aids used

included an editor, assembler, loader and a primitive file manager. These tools

ran under a multi-tasking operating system environment (R T68MxJ.

The assembled code occupied 2380 bytes excluding all data storage.

The variables and scratchpad areas occupied 5U8 bytes. Three byte looKup

tables occupied 256 bytes each. The two frame buffers needed (one for the

overlay-only image and one for the image containing overlay details and finger

blobs) occupied 2K bytes each. Thus the total RAM area needed for all

functions was approximately 7K bytes.

The software comprises four main routines. Their names and

functions are described below:

Access: Causes the image processor to fill a predefined image

buffer area with a digitized image. The image size and starting

coordinates also have to defined prior to calling this routine.

Clexor: Subtracts the overlay-only image from the 'working image'

containing overlay details and finger blobs. It then shrinks all blobs

in the resulting image to vertical lines.

Picgrv: Tests the working image buffer to see whether or not any

A.4


fingers were found pressed. If so, it fills a list (CGLIST) with the

centroid of each finger.

Norflo: Access CGLIST sequentially and attempts to track to Key

centroid points as per the FCKCL technique. When an encircling

key is detected, the code corresponding to the key is output.

Under normal use, these routines are called in succession in the

order they are listed. They were designed to provide all parameters needed by

successive routines. However, priOr to invoking this 'infinite loop', certain data

structures (overlay definition codes, OMA controller parameters, Image

Processor microcode etc. ) must be created for use by these routines. That is

done by a routine called Define. Define also takes care of appropriate helpful

messages during the definition phase to aid the unfamiliar user.

The software had an additional mode of operation: Diagnostic mode.

This mode was created primarily for ease of development and debug. It allows

examination of the data structures after each major routine is called as well as

the ability to dump a binary image onto an output device for visual inspection.

A.5


Appendix B

Details of Software Implementation and Tools

The software for the host processor was written in assembly

language to achieve the best speed possible. The development aids used

included an editor, assembler, loader and a primitive file manager. These tools

ran under a multi-tasking operating system environment (R T68Mx).

The assembled code occupied i:'.380 bytes excluding all data storage.

The variables and scratchpad areas occupied 508 bytes. Three byte lookup

tables occupied 256 bytes each. The two frame bu:tf ers needed tone for the

overlay-only image and one for the image containing overlay details and finger

blobs) occupied 2K bytes each. Thus the total RAM area needed for all

functions was approximately 7K bytes.

The software comprises four main routines. Their names and

functions are described below:

Access: Causes the image processor to fill a predefined image

buffer area with a digitized image. The image size and starting

coordinates also have to defined prior to calling this routine.

Clexor: Subtracts the overlay-only image from the 'working image'

containing overlay details and finger blobs. It then shrinks all blobs

in the resulting image to vertical lines.

Picgrv: Tests the working image buffer to see whether or not any

A.4


fingers were found pressed. If so, it fills a list (CGLIST) with the

centroid of each finger.

Norflo: Access CGLlST sequentially and attempts to track to Key

centroid points as per the FCKCL technique. When an encircling

key is detected, the code corresponding to the key is output.

Under normal use, these routines are called in succession in the

order they are listed. They were designed to provide all parameters needed by

successive routines. However, prior to invoking this 'infinite loop', certain data

structures \overlay definition codes, OMA controller parameters, Image

Processor microcode etc. ) must be created for use by these routines. That is

done by a routine called Define. Define also takes care of appropriate helpful

messages during the definition phase to aid the unfamiliar user.

The software had an additional mode of operation: Diagnostic mode.

This mode was created primarily for ease of development and debug. It allows

examination of the data structures after each major routine is called as well as

the ability to dump a binary image onto an output device for visual inspection.

A.5


PLEASE DO NOT REMOVE

CARDS OR SLIPS FROM THIS POCKET

UNIVERSITY OF TORONTO LIBRARY

TH:ESIS

966 Mehta, Nimish ele A flexible human machine

interface MA.Sc., U. of T., Oct. '82

UNIVERSITY OF TORONTO ENGINEERING LIBRARY


BlackBerry Exhibit 1017, pg. 1...1.1 BlackBerry Exhibit 1017, pg. 12 ( Input to a computer can be...

Documents

Transcript of BlackBerry Exhibit 1017, pg. 1...1.1 BlackBerry Exhibit 1017, pg. 12 ( Input to a computer can be...