Using pictures in expert systems

Engng Applic. Artif. Intell. Vol. 5, No. 4, pp. 329-344, 1992 0952-1976192 $5.00 + 0.00 Printed in Great Britain. All rights reserved Copyright © 1992 Pergamon Press Ltd

Contributed Paper

Using Pictures in Expert Systems P H I L I P B A R K E R

Teesside Polytechnic, Middlesborough

The use of expert systems is now quite commonplace in virtually all areas of engineering and science. However, despite their significant utility many commercially available development shells constrain users to consultation and advice-giving dialogues that are primarily textual in nature. This paper discusses: (1) the need to incorporate pictorial forms into expert system dialogue; and (2) ways in which pictures and images can be embedded within such dialogues.

Keywords: Expert system, pictures, Prolog, image storage, image retrieval.

INTRODUCTION

The phrase "expert system" is often used as a generic term to describe that category of computer software which attempts to emulate human decision-making pro- cesses and problem-solving behaviour. Software of this type differs from conventional software in its ability to generate "new knowledge" through the application of various methods of inferencing. Many of the first expert systems were fabricated using artificial intelligence lan- guages such as Lisp or Prolog. Nowadays most expert systems are generated by means of "development shells" (for example, Crystal, Savoir, EasyExpert and ESP Advisor ~) or "knowledge engineering environments" (such as KnowledgePro, 2 KEE and ART3).

Although inferencing represents a powerful approach to automatic problem solving it is still one which, in many ways, is quite limited. A major limitation of most commercially available expert-system shells is the need to have access to knowledge engineers who are able to "draw up" and codify the basic rules and domain expertise that are needed in any specific problem-solving situation. 4 Automatic rule generation and codification of knowledge is at present beyond the power of most expert systems. Arguably, the incorporation of neural-network techniques into expert- system shells will eventually remove this limitation. 5 However, commercial development shells having this ability are still some way off.

Another important limitation of many currently available expert-system shells relates to the nature of the consultation and advice-giving dialogues that they permit) Most systems allow only textual exchange. The user communicates with the expert system by: (1)

Correspondence should be sent to: P. Barker, Interactive Systems Research Group, School of Computing and Mathematics, Teesside Polytechnic, Middlesborough, Cleveland TSI 3BA, U.K.

entering strings of characters through the keyboard; and (2) reading the textual responses produced by the system on the computer display screen. Sometimes the basic textual display of information is augmented by simple low-resolution diagrams that have been produced using the graphic character set that the display screen supports. 7

Despite their current limitations expert systems are widelyused as sources of advice within particular, well- defined problem domains, s-12 Typical application areas for expert systems include design, planning, system maintenance, fault diagnosis, trouble shooting and control. In each of these areas the ability to incorporate high-quality pictorial forms into the expert-system dialogue would be extremely advantageous.

In view of its importance, this paper addresses the issue of embedding high-quality pictorial images into commercially available expert-system shells. The work that has been undertaken is illustrated using a commercially available development shell called ESP Advisor. ~3 Two basic approaches are discussed; one involves the use of TV-quality images stored on video disk while the other is based on digital images that are stored on compact disk.

ESP ADVISOR AND PROLOG

A wide range of expert-system development shells exists commercially. The available product range reflects a spectrum of processing capability. Invariably, the nature of the facilities available within any given shell is often reflected in its purchase cost. Usually, less-expensive systems offer fewer facilities and are less versatile than those which require a substantial capital outlay.

ESP Advisor is a relatively low-cost development shell, the source code for which is written in Prolog. The system is designed for use with IBM PC microcom-

329

330 PHILIP BARKER: USING PICTURES IN EXPERT SYSTEMS

i T.** 1 .__1 Kno.,.d0. I Editor Engineer I"1 Domain

Expertise

I I P c,i. Handling

J ,o,oo Compiler v I System

I Complied Kno ,.:O.

Consultation Shell

-ql I Picture Display

Fig. 1. The ESP Advisor expert-system development shell.

puters (or compatibles) within the MS-DOS operating system. It is primarily intended for "advice giving" applications that are based upon the screen display of textural information. A typical ESP knowledge base would therefore consist of a collection of textual items. The conditions under which any particular item of text is displayed depend upon the values of parameters (or variables) that are set during the course of a consultation dialogue.

Because it is a relatively simple shell (both in concept and to use) it has a number of limitations and restric- tions. Fortunately, many of these can be overcome through the use of either external code routines (which may be written in FORTRAN, C, Assembler, and so on) or extensions that are written in Prolog. However, in order to use these features an appropriate version of Prolog must be made available within the environment in which ESP Advisor runs. The particular version needed for ESP is Prolog-2.14 This version of Prolog offers a rich environment in which to develop knowledge-based software.

The basic system that has been used for the work described in this paper is illustrated schematically in Fig. 1. Essentially, a knowledge engineer would use an interactive text editing system to create an ASCII text file containing the codified domain expertise relevant to the problem being solved. The domain expertise is coded in terms of the knowledge representation language (KRL) used by the ESP Advisor system--this is discussed below. The ESP KRL compiler then trans- lates the ASCII source file into an executable knowledge base which can be processed by the consultation shell. This shell is a generic "dialogue manager" that can be used to front-end any previously compiled ESP knowledge base.

As was mentioned above, the knowledge bases that ESP uses can contain references to external code and/ or Prolog routines. It is through such extensions that picture handling is achieved within the ESP system. As can be seen from Fig. 1, in this situation, an additional task that the knowledge engineer has to undertake is the specification of the pictorial material that is to be

PHILIP BARKER: USING PICTURES IN EXPERT SYSTEMS 331

] r e f e r e n o e s t a r t . {mater ia l - -wood end o o n d i t i o n e - - d r y } r e f e r e n o e s o r e w s . { m a t e r i e l - - p l a e t i o and o o n d i t i o n s - - d r y } r e f e r e n o e g l u e . ( m a t e r i a l = o c t a l and o o n d i t i o n 8 --wet} r e f e r e n o a r i v e t . {materlel--paper and conditions--dry} reference stapler. {materiel= stone and oonditlons--wst} referenoe oeaont. referenos finish.

II I I II I II

Fig. 2. UseoftheESP Advisor"re~rence"command.

used during a consultation dialogue. The pictures that are used in particular applications are brought together and organised in the form of either a segmented or a dedicated image store. A segmented image store is normally used in situations where a given application uses only a relatively small number of images.

An ESP knowledge base consists of two basic parts: a title and a body. The title is purely descriptive and serves as a means of identifying and describing the knowledge base during menu-driven dialogue. The body of the expert system embeds the domain knowledge. This knowledge is structured into "sections". Any given knowledge base will therefore consist of an obligatory unnamed "control section" and an optional set of named sections. When a knowledge base is invoked the control section is executed. Any other named sections in the knowledge base will only be executed if they are specifically referenced by means of a "reference" command. Such a command may have two basic forms:

(1) reference (section-name). (2) {(pre-condition)} reference (section-name).

The first of these forces the unconditional execution of the section whose name is (section-name) whereas the second form facilitates conditional execution. In template (2) the named section is executed only if the command's pre-condition is satisfied. The (pre- condition) that precedes the reference command is essentially a Boolean expression that evaluates to a value of true or false. If the expression equates to

"true" then the named section is executed, otherwise the command stem is ignored. Examples of the use of reference commands are illustrated in Fig. 2. Sectionalization is an important aspect of expert-system design and implementation. The appropriate use of sections allows a knowledge base to be developed using either a "top-down" or a "bottom-up" approach. Sectionalization also enables a knowledge base to be structured according to the particular needs of any given application.

Within any given section of an ESP expert system the embedded knowledge is usually structured into a series of paragraphs. Like sections, paragraphs are condi- tionally executed. The advice or commands contained within a paragraph are executed only if that paragraph's pre-condition is satisfied. The format of embedded advice and ESP commands within paragraphs is as follows:

(3) {(pre-condition)} (textual-advice). (4) {(pre-condition)} (command).

Some examples of simple ESP paragraphs are illustrated in Fig 3. Paragraph 1 is displayed on the display screen only if the value of the parameter AGE is less than 30 and the value of the parameter QUALIFICATION is BEng. Similarly, the textual advice embedded in paragraph 2 is executed only if the value of the parameter AGE is less than 40 and the value of the parameter QUALIFICATION is BSc. Paragraph 3 is executed if the value of QUALIFICATION is either BEng or BSc.

I! I

{age~30 and q u a l i f i c a t i o n = beng} / , Paragraph I */

'Young engineers should apply for ohartered status'& 'baoausa thl8 18 a reoognlsed qualifloation in' & 'many professional clrolee. If you ere eligible'& 'to apply please ask for FORM OEng.Reg.2990'.

{age~40 and quallfloatlon =bso} /* Paragraph 2 */

'Young solentiets need to have • background'& 'in statistios and elsotronio instrumentation.'& 'The ability tO use a ¢omputer 18 also useful.'.

{qualIEioation=bang or qualifioation=bso} /* Paragraph 3 */ quit.

{qualifioatlon--phd} /* Paragraph 4 */

rafarenoe phd_section. m

Fig. 3. Examples of ESP paragraphs.


s e o t i o n p r o l o g z ' Simple E x a s p l e ' . / * P r o l o g aode f o r i l l u s t r a t i o n */

q u a l i f i o a t i o n ( s m i t h , b s ¢ ) . q u a l i f t o a t i o n ( b a k e r e b s o ) . q u a l i f i c a t i o n ( b r o w n , b e n g ) . q u a l i f i o a t i o n ( J o n e s , n o n e ) . q u a l i f i o a t i o n ( t h o m a s , p h d ) . a g e ( s m i t h , 37 ) . age(brown,25). age(jones,52). age(thomas,29). age(baker,57). young_englneer(X) z-

age(X,Y), Y<30, quallfiuation(X,Z), Z=beng, set parameter(qualifled,true), wrlte(X), write(" is a young engineer.").

young_solentlst(X}t- age(X,Y), Y<40, quallfloatlon(X,Z), Z=bsoo set_parameter(qualifled,true), wrlte(X}, write(" is a young scientist").

unquallfled(X|t- wrlte(X), write (" is not qualified."}.

I | I I

Fig. 4. An ESP Advisor Prolog section.

Typical examples of ESP Advisor commands include: quit, XOR, do and reference. The reference command has already been described. The quit command allows an exit to be made from a given line of reasoning while the declarative XOR command allows an "exclusive or" relationship to be set up between a set of parameters. The do command is important because it facilitates the execution of Prolog goals. The full format of this command is as follows:

(5) {(0re-condition)} do (prolog-goal).

The (prolog-goal} used in template (5) consists of a standard Prolog predicate followed by an optional (parameter-list). That is:

(6) (prolog-goal):: =(predicate) ((parameter-list))

In order to use Prolog facilities within an ESP Advisor expert system, its associated knowledge base must include a special named section called "prolog". Such a section may contain any valid Prolog statements. An example of a simple Prolog section is illustrated in Fig. 4. This contains a collection of facts in the form of a database and two rules (or goals) that can be used to determine if a person is a young engineer or a young scientist. These Prolog goals can be called from within an ESP Advisor expert system in the following way:

(7) {test_needed} do young_engineer(X). (8) {validate_status} do young_scientist (NAME).

Of course, the parameters X and NAME would need to be instantiated prior to calling the Prolog goals.

Four basic types of parameter are used in ESP Advisor: fact; number; category; and phrase. Fact parameters represent the values of simple yes/no questions which can only evaluate to the Boolean values true or false. They correspond to facts that can be established by asking the user simple questions requir- ing a yes or no response. Number parameters are used to represent real numeric quantities; these may be used to ask numerical questions or to calculate values. Notice that in ESP Advisor there is no concept of " in teger"- -a l l quantities are real (a minor limitation). Category parameters may take on the value of one of a number of options that are contained in an enumerated list (for example, co lou r - code might be one of the pre- defined values red, yellow, green, white or black). Phrase parameters are used to represent short pieces of text such as a person's name or the title of a book, and SO o n .

The standard ESP template that is used to declare a parameter within a knowledge base takes the following form:

(9) (name): (description} (type)

explanation (explanatory-text) (range-or-option-list)

rules (rule-I),


(rule-2), (rule-3),

! ! !

(rule-N), askable

using (question).

In template (9) there are four keywords--

explanation, rules, askable and using. Many of the entries in the template are optional but some mechanism must be available to allow the assignment of a value to the parameter. Usually, this is accomplished using the rule-list or the askable. When the value of a parameter is needed, the rule-list is examined (if it is present) to see if a rule can be found which will allow the parameter's value to be ascertained. I f no rule is applicable (or if there are no rules to evaluate) then the question specified in the askable is posed to the user.

t i t l e ' F i g u r e S - D e n t i s t Expe r t S y s t e m . ' . ' T h i s e x p e r t system p r e s c r i b e s d i f f e r e n t cou rses ' & ' o f t r e a t m e n t f o r v a r i o u s c a t e g o r i e s o f p a t i e n t ' & ' d e p e n d i n g upon t h e number o f t e e t h t h a t t h e y h a v e ' & • and t h e i r s e n s i t i v i t y t o p a i n k i l l e r s . ' .

teeth_left = 'The patient has some tooth' fact askable using 'Does the patient have any teeth?'.

d e c a y e d _ t o o t h = 'An e x t r a o t i o n i s p r o b a b l y n e e d e d ' fact askable using 'Is the patient''e tooth badly decayed?'.

r e f e r e n c e i n i t i a l _ p a r t . /* g e t d e t a i l s o f p a t i e n t * / {not teeth_loft} reference dentures. {teeth_left and decayed tooth} reference extraction. {tooth left and not decayed_tooth} reference filling. reference records. reference billing. /* *I s e c t i o n i n i t i a l _ p a r t | ' T h i s s e c t i o n g e t s t h e i n i t i a l d a t a ' . 'We need the patlent''8 name in order to check whether or' & 'not he/she has had previous treatment at this surgery.'.

name z 'the patlent''s name' phrase askable 'What is the patient''s name?'.

rec sum = 'the patlent''s record numbers n~mher rule use cheek status(name,roe nun). 'I am testing to see if th~s is an established patient.'.

{roe_stun>0} 'This Is a an ESTABLISHED patient.', do fetch_record(name).

{tee_sum=0} 'Thls Is a NEW pc,lent.', do get_recnum, do make r e c o r d ( n a m e , r o e sum) .

s e c t i o n d e n t u r e s - = ' T h i s s e c t i o n - h a n d l e s t h e u s e o f d e n t u r e s ' . ' T h e p a t i e n t w i l l need t o h a v e h i s / h e r mouth and g u n s ' S ' m e a s u r e d i f i t i s i n t e n d e d t o f i t d e n t u r e s ' .

s e c t i o n e x t r a c t i o n s ' T h i s s e c t i o n h a n d l e s e x t r a c t i o n s ' . ' I t s e e m s a s t h o u g h t h e p a t i e n t w i l l n e e d t o h a v e a '& e t o o t h e x t r a c t e d , e . q u i t .

s e c t i o n f i l l i n g x ' T h i s s e c t i o n h a n d l e s f i l l i n g s ' . • The p a t i e n t w i l l need t o h a v e none t o o t h f i l l e d . ' .

s e c t i o n r a c o r d e = ' T h i s s e c t i o n h a n d l e s p a t i e n t r e c o r d s ' . ' T h e d a t a b a s e o f r e c o r d s I s now b e i n g u p d a t e d ' .

s e c t i o n b i l l i n g = ' T h i s s e c t i o n h a n d l e s t h e p a t t e n t ' ' s b i l l ' . ' B i l l i n g w i l l depend upon w h e t h e r t h e p a t i e n t i s a p r i v t t e ' a ' p a t i e n t o r one who t e e l i g i b l e f o r N a t i o n a l H e a l t h s u p p o r t '

I

Fig. 5. See caption overleaf.


/ * The P r o l o g S s e t i o n * /

s e a t i o n p r o l o g : ' t h e p r o l o g s e o t i o n ' . p a t i e n t ( h a n n a h , 1 2 3 ) . p a t i e n t ( g i l l e r , 4 5 2 ) . p a t i e n t ( l a m e n t , '/89 ) • p a t i e n t (man~ io 3.11 } . p a t l e n t (r i o h a r d e • 6 S 8 ) . c h e e k s t a t u s (XtY) z -

p a t i e n t (X,Y) • w r i t e ( " P a t i e n t f o u n d . - ) .

s h o c k s t a t u e (X•Y) : - w r ~ t e ( " P a t i e n t n o t f o u n d " ) , Y i s O .

make r e c o r d ( X e Z ) : - n~, write("Creatlng record for patient: ,'), write (X), nl, wrlte("Number Ist "}, wrlts (Z }.

fetch_reaord (X} : - nl, wrlts("Getting reoord for patient: "), patient (X, Z}, writs (X), nl, wrlto("Patlsnt number is: "), write (Z).

gst_reo_num: - nl, write("What is the patlent''s reeord number? ,,}, read (N), set_parameter (roe_nun, N) •

/* End o f D e n t i s t Ix]pert System */

u

Fig. 5. An ESP Advisor knowledge base with a Prolog section.

The nature of the rules used for value assignment depend on the nature of the parameter and are of the following general form:

(10) (Bool-exp) if (condition) (fact parameters)

(11) (arith-exp} if (condition) (number parameters)

(12) (option-name} if (condition) (category parameters)

(13) (phrase) if (condition) (phrase parameters)

(14) use (goal) if (condition) (call to Prolog)

In the above templates the condition following the if keyword is used to determine whether or not the rule will be used to evaluate the value for its parent parameter. In template (10) (Bool-exp) represents a Boolean expression that must evaluate to true or false. Similarly, in template (11), (arith-exp) is an arithmetic expression that evaluates to a real numeric quantity, and so on. Template (14) is used when a Prolog goal is

to be used to determine the value of a parameter. Figure 5 lists a simple contrived example of an ESP

Advisor knowledge base that illustrates the use of (1) sectionalization and (2) the Prolog interface. The Prolog section is primarily responsible for checking whether or not a dentist's client is an established patient by reference to a simple database of names. In the fully implemented system the Prolog section would also be responsible for the overall maintenance of the database as well as for the billing of clients.

USING PICTURES IN CONSULTATION DIALOGUES

The images that are used in consultation dialogues with expert systems can be of two basic types: analogue and digital. Analogue images are normally stored on video disk, whereas digital images may reside on either hard disk or a digital optical storage facility. Two types of digital storage may be used: re-writable and read- only. The basic arrangement of equipment used for the work described in this paper is illustrated schematically in Fig. 6.


The first diagram (Fig. 6A) shows the basic compo- nents used to construct an image store based on the use of a video disk. Figures 6(B) and (C) depict similar image stores based on the use of magneto-optical re- writable disk (MOROD) and compact disk read-only- memory (CDROM), respectively. The use of both the analogue and the digital approaches to image storage is

discussed in more detail in the remainder of this section.

Using video disk

The video disk player is connected to the computer by a serial communication link. It is controlled by means of command strings which embed the digital

| I I I I I

(A) Video Disk

[ ] / EXPERT SYSTEM

(B) Re.writable Optical Disk

I'--'1 RS-232-C

Image Display

Video Disk Image Store I

a ~ s l o g u e Image

I I I I

/[ J I

I., EXPERT SYSTEM

I I

4-----)

R5-232-C

Image Display

i MOROD Control

I I

SCSI

digital Image

MOROD I

(C) Compact Disk Read-Only-Memory

[{ I EXPERT

SYSTEM

}/ /{ it II II

Control

RS-232-0 High Sierra

digital Image

I CDROM ,, I

(D) Single-Screen Display

Video. Disk

[Imago Display '

II II

p" | image Processing ,, Digital

Optical Disk 1 Fig. 6. Workstations for picture display.


i i |i i

title 'Figure 7 - Image Retrieval by Number,. plenum : 'frame number selected by user'

number range 1 . . 5 4 0 0 0 askable 'What picture do you wish to see?'.

do s h o w p i o ( p i o n u ~ ) . 'The i m a g e now b e i n g d i s p l a y e d i s , . . Qpionun . . , . , .

section prolog z 'controls video disc image store'. showpic (x ) z-

t e l l ( p r i n t e r ) , wrlte('P'), wrlte(X), write('R'), nl, tell(user).

i i I II I l l I

Fig. 7. Direct image retrieval

II

using picture addresses.

address of the image to be retrieved. A typical video disk might contain up to 54,000 high-quality video images; each image is identified by an integer address. In order to display a particular image on the screen of the TV, a command of the following form has to be transferred to the video disk unit:

(retr ieval-command)::= P(digit) (digit>(digit)(digit)(digit)R,

where the digit string is the actual address of the picture to be retrieved and displayed.

Because the ESP Advisor system has no built-in

facilities for controlling an external image store, picture retrieval must be accomplished by means of interface software that is written either in Prolog or in some other appropriate language. Undoubtedly, the simplest approach to image retrieval is through the use of a simple Prolog goal which when invoked would pass image address details across to the image store. Suppose that a Prolog predicate "showpic" was available to perform this function. Within an ESP knowledge base references to the showpic code would be made in the following way:

{(pro-condition)} do showpic {N).

title 'Figure 8 - Image Retrieval by Menu'. picnum: 'frame number selected by user' number rules

12345 if brldge=stone, 24680 if bridge=metal, 34567 if bridge=wood, 47631 if bridge=rope.

bridge : 'the types of bridge available for study' category options

stone, metal, wood, rope askable

'What type of bridge would you like to examine?'.

'This expert system deals with design faults in bridges.'& 'There are four types of bridge - each has a major design'& 'fault.' , ,&

'You will be shown a picture illustrating this fault and'& 'then be shown how to test existing bridges for the fault.'.

do showplo(plontm). 'The image now being displayed is a good example'& 'of a ' .. Qbrldge .. ' bridge showing the design fault.,.

s e c t i o n p r o l o g z ' c o n t r o l s v i d e o d i s c imago s t o r e ' . / * T h i s i s t h e same a s i n f i g u r e 7 * /

I I I I I I II I I I I I ii i i i i i I

Fig. 8. Indirect image retrieval using a menu facility.


t i t l e ' F i g u r e 9 - Image R e t r i e v a l by t i t l e ' . p i o n a m e | ' t h e t i t l e o f t h e p i c t u r e t o be f o u n d •

phrase askable

'What is the title of the picture you wlsh to see? • . • Thle expert stern illustrates the use of simple'& 'picture retrieval by title.'& • •&

'Hero lea picture 0£ a motor car''s suspension system.', do flndpio('suspenslon'). 'Here is a picture of an exhaust system.', do flndpio('exhaust system'}. 'Now enter the title of a picture to be retrieved.', do flndpic(plcname).

s e c t i o n p r o l o g = ' c o n t r o l s v i d e o d i s c image s t o r e ' . p i c t u r e ( c a t • 2 2 3 0 0 ) . picture('sun and wind',24680). picture('christmas'•12380). pioture('Jack and Jill', 34120). picture{'exhaust system',27823|. picture(suspenslon,45679). findplc(X) z-

plcture(X,N), showplc(N}.

/* showplo is as defined in figure ? */

Fig. 9. Retrieval of images by title.

where N represents the address of the image to be retrieved. The definition of the showpic routine and the way in which it is used is illustrated in Figs 7-9.

Three different approaches to picture selection are presented. In Fig. 7 the user simply enters the five-digit address of the required image (notice the automatic range checking). In Fig. 8, during the course of the consultation dialogue, the user is given a list of pictures to choose from. Finally, Fig. 9 illustrates how simple titles might be used to retrieve pictures. This is accomplished by means of an additional routine "findpic" which uses a simple database in order to look up the frame number associated with a given title.

Notice how, in Fig. 7, the showpic routine sends the appropriate command string to the video disk via the standard stream "printer". In actual fact, the video disk is attached to the system by means of a standard serial communication link (COM1 or COM2 in MS-DOS machines). Therefore, in order to direct the output from the parallel printer port to COM1 a MS-DOS mode command (MODE LPT1 :=COM1 :) needs to be issued prior to invoking the ESP Advisor system. ~~ The code embedded in the showpic routine shown in Fig. 7 is highly machine-dependent; in practice it would only work with one particular type of video disk player (a Philips VP-705 LaserVision system). However, it would not be difficult to modify the code to work with any other video disk system. For example, for a Pioneer LD/V4100 video disk system the relevant definition of the showpic routine would be as follows:

showpic(X):- tell(printer), write(X), write('SE'), nl, tell(user).

Instead of using actual picture addresses or titles as a means of image retrieval, in many situations it might be more appropriate to user either picture attributes or keyword descriptions as a means of retrieving images. Suppose, for example, that the images in an image store had been indexed by means of a single keyword that named the principal object in the image. Such a database might take the following form:

picture(engine, 23516). picture('steam roller', 34125). picture('stone bridge', 40632), picture('stone bridge', 40633), picture('metal bridge', 51237). picture(engine, 37612).

In many ways the above database is similar to the titles database which was used in Fig. 9. However, in writing the search code for the example in this figure (the routine findpic), it was assumed that within the database any given picture title would be unique. As can be seen from the above database, when keywords or attributes are used for picture indexing it is quite common to have multiple entries associated with a given keyword descriptor; the entries for "stone bridge" and "engine" illustrate this. The image- retrieval software must obviously be capable of hand-


tltle 'Figure 10 - Image Retrieval by single keyword'. 'This expert system illustrates the use of simple'& 'ploture retrleval by keyword.'&

'It retrieves all plutures indexed by a given'& 'keyword or keyphrase.'. do query. 'End of Consultation.'.

s e e , i o n p ro log : ' ¢ o n t r o l s v ideo d iso image s t o r e ' . ? - o o n s u l t ( " p i o t u r e s . d a t " ) .

l i s t a l l ( X ) = - pic ture (X,¥ ) , assert (piofound(Y)) , n l , a d v i s e ( I ) , wrlte("Picture number is "), write(Y), showpi¢(Y), nextpi¢, nl, fail.

1istall(X):- predicate(plofound/1), retraotall(plofound/1), /* write("DB oleared."), */ nl.

1istall(X):- write("No piotures found."), nl.

query:- nl, nl, write("Whioh picture do you require? "), read(X), nl, 1is,all(X).

nextplc:- nl, wrlte("Next Piuture [y]: "), read(_), nl.

s h o w p i o ( X ) : - tell(prlnter), wrlte('P'), wrlte(X), wrlte('R'), nl, tell(user).

| I INN n i l

Fig. 10. Image retrieval using a single keyword descriptor.

ling this situation. The simple findpic routine used in Fig. 9 would therefore need to be modified in order to retrieve all objects satisfying a given retrieval criterion.

The way in which this might be done is illustrated in Fig. 10. The Prolog predicate query is responsible for asking the user for the keyword or keyphrase that best describes those pictures which are of interest within the context of the current consultation. The listall routine is then responsible for searching the database (PICTURES.DAT) for all the pictures that satisfy the user's search specification. Notice that there are two other listall routines in the Prolog section; one handles the situation in which no picture references are found, while the other "cleans up" the database after a suc-

cessful retrieval operation. Another important feature of the Prolog section shown in Fig. 10 is the inclusion of the nextpic predicate following the reference to showpic within the listall routine. This provides a paging mechanism to enable the user to browse through the relevant pictures one at a time. Obviously, it would be a simple matter to modify the consultation dialogue so that, after looking at a sequence of pictures, the user could enter the identity code of the one(s) most relevant to his/her current problem.

A natural extension of the use of single keywords (as depicted in Fig. 10) would be the use of multiple indexing terms or attributes. A typical database to handle this situation is presented below:


picture (22100, [cat, dog, fish, moon]). picture (31200, [lion, sheep, wolf, pig, zebra,

cow]). picture picture picture picture picture picture

(23300, [car, cat, star, moon]). (26400, [bicycle, boy, girl, cat]). (38500, [house, man, woman, dog]). (47600, [shop, car, lorry, train, boy]). (22700, [snow, lake, tree, zebra, deer]). (29800, [rocket, computer, sun, star]).

picture (17900, [train, house, car, girl, boy]). picture (31023, [cat, mouse, cheese, apple, fish]). picture (32657, [fish, bridge, 'Tom and Jerry',

tank]).

This database uses a predicate PICTURE (with arity 2) to store picture numbers (the first argument) and a list of keyword descriptors used to index a given picture (the second argument). Standard Prolog list notation is

t i t l e 'F igure 11 - Imago r e t r i e v a l by m u l t i p l e keywords'. p t c l t s t z ' t h e l i s t o f k e y w o r d s '

phrase askable 'Enter your keyword flat'.

'This expert system allows retrieval o f pictures'& 'based upon a llst of keyword descriptors.'& , ,&

'Enter your llst in the following forms'& ' A,B,C,D, ... '& 'where A, B, C, eto represent the keywords describing'& 'the ploture you wlsh to see.'. do s h o w p i o s ( p i o l i s t ) . 'End o f C o n s u l t a t i o n . ' .

s e c t i o n p r o l o g : ' c o n t r o l s v i d e o d i s c image s t o r e ' . ?-oonsult("plotures.dat"). / * Rules to test for llst membership * / amember(X, tXJ_]) a m e m b e r l X , [ _ J Y ] l ; -

amember(X,Y) .

all i n ( [ ] , ) a l l ~ i n l [ E J T l ~ Q l : -

amember (H, Q), all_In (T,Q).

/* R u l e s f o r s t r i n g t o l i s t c o n v e r s i o n c o m m a ( [ ] , 0 ) . comma{[441T],l). comma([HIT],N)=-

comma(T,NZ), N i s 1+N1.

b a g o f ( X , G o a l , X l i s t ) = - c a l l ( G o a l ) , asser ts (s tack(X) ) , fail; a s s e r t s ( s t a c k ( b o t t o m ) ) , c o l l e c t ( X l i s t ) .

c o l l e c t ( L ) : - r e t r a o t ( s t a c k ( X ) ) , l , ( X : : b o t t o m , l t L : [ ] ; L: [XJRest ] , co l lec t (Res t ) ) .

s t r i n g t o l t s t ( R , L ) z- X i s - - s t r i n g R t " " , l i s t ~ X , X ) , asse r t ( s ta r tS0 ) ) , conma(Y,E) , s t a r t ( S } , S i s s t r i n g s u b s t r t n g ( X , SwE-S-1) , W i s Z e t r t n g "keyword(" S Z & " ) . " , deoode(WeK), assor t (K) , r e t r a e t a l l ( s t a r t l l ) , a s s e r t ( s t a r t ( E ) ) , f a i l .

*/

Fig. 11. See caption overleaf.


|111 I

string to llst(R,L) z- retraotell(stert/1), bego£(X,keyword(X),L), retrecta11(keyword/1).

/ * Rule to llst all pictures containing a given object * / s h o w p i c s ( A ) : -

string_to_list(A,L), aesert(totplos(0)), flndall(L).

/ * R u l e s f o r search ing database * / f i n d a l l ( X ) z -

p i c t u r e ( P , Q ) , a l l i n ( X , Q ) , t o t p i c s ( S ) , r e t r a o t ( t o t p i o s ( N ) ) , S is I+N, a s s e r t ( t o t p i o s ( 8 ) ) , s h o w p i o ( P ) , advlse(1), nl, wrlte("Pioture number: ,,), wrlte(P), nextplc, fail.

flndall(X) z- totpics(N), No0, retract(totplos(0)), advlse(l}, wrlte('No plotures found.,), nl,wrlte('Press <I~TBR> to oontlnue ...'), get0(B), restore screen.

flndall(L~:- totpios(N), a d v i s e ( I ) , write('So more plotures found.'), nl, wrlte('Total number of pictures found: '), wrlte(N), nl, wrlte('Press <ImTBR> to oontlnue ...'), get0(B), restore screen, r e t r a o t ~ t o t p i o s ( N ) ) .

nextplc:- nl, wrlte("Next Picture [y]: "), read(), nl.

s h o v p i c ( X ) z - tell(prlnter), wrlte('P'), wrlte(X), wrlte('R'), nl, tell(user). I II I

Fig. ll.lmageretrievalusing multiplekeywords.

used to enumerate the indexing entries. In order to access a database of the form shown above an ESP Advisor expert system would need to pass across to the search software a list of keywords that describes the sought-after picture(s). For example, a typical query might be expressed in the form:

[bridge, road, tunnel, river]

which would be interpreted as being a reference to all pictures containing a bridge, a road, a tunnel and a river.

Unfortunately, ESP Advisor does not have any parameter type that is directly equivalent to a Prolog list structure. Consequently, a phrase parameter must be used to generate a character string that embeds the list to be used in searching the database. Appropriate code


within the Prolog section would then be used to decode the string and convert it into a proper Prolog list structure. The way in which this is done is illustrated in the simple expert system that is presented in Fig. 11.

In this example the ESP Advisor expert system invokes the Prolog section by means of a call to the goal showpics the argument of which (pielist--a phrase parameter) represents the list of keywords describing the required pictures. This takes the form of a comma list in which individual entries are separated by commas--for example, "bridge, road, river". Within the Prolog section the first action to be taken is the conversion of the input parameter from string form into a list structure. This is performed by the two rules list to string(A, L) and their supporting clauses-- c o m m a and bagof. Comma searches a list structure and locates the positions of the commas used to separate the keywords in a list. Bagof collects together objects that satisfy a common condition and puts them into a list structure. Notice that bagof occurs as a built- in predicate in many versions of Prolog. 14' 16 However, the Prolog-2 (version 1.22) supplied with the edition of ESP Advisor (2.04) used in this work did not provide bagof as a built-in predicate and so it had to be defined within the Prolog section of the expert system. 17 Once a keyword list has been generated the findall rules are used to locate appropriate picture addresses from within the database. Findall uses standard list processing techniques 16' 17 to check that every item in the search list is contained within a picture's list of indexing terms in order for the picture to qualify for retrieval. Two sets of important list-processing clauses used in this operation are a m e m b e r (which checks for membership of an item on a list) and all_in (which ensures that all items in one list occur as a subset of those in another).

Prolog could be used to augment the picture handling capabilities of ESP Advisor in many different ways. For example, it would be possible to implement retrieval methods based on: pictures of "nearest match"; the incorporation of not logic; pictures containing particular numbers of objects; and pictures in which objects bear particular spatial relationships to each other. Some of these possibilities are further discussed later in the paper.

Using digital optical storage Apart from the marked difference in picture quality

(see below), the major differences between the analogue and digital storage techniques depicted in Fig. 6 lie in: (1) the method of picture storage that each approach uses; and (2) the way in which individual images are addressed. Read-only video disk storage facilities (similar to that described in the previous section) store one high-quality image on each of the optical disk tracks. Any individual image can be retrieved by means of its unique five-digit address. In contrast, provided suitable interface software and hard- ware are used, the optical disks used to fabricate digital

image stores (similar to those shown in Figs 6B and C) may be regarded as either re-writable or read-only high-capacity hard disks. For the work described in this paper Microsoft's MS-DOS compact disk extensions 18 have been used. These enable individual digital pictures to be stored as standard MS-DOS disk files. In most of the work undertaken with ESP Advisor, ordin- ary uncompressed "PCX" files have been used.19 These have been used to store EGA quality images of size 640 x 350 pixels with a palette of sixteen basic colours. Depending upon the nature of the images involved, the file sizes of these pictures varies between about 11 and 49 kbytes. Given that a standard CDROM disk has a capacity of 650 Mbytes this means that such a disk could store in the region of 20,000 uncompressed digital PCX images--assuming an average file size of about 30 kbytes.

Three basic methods have been used here to generate digital images for incorporation into expert systems based upon the use of ESP Advisor. The first method depends upon the use of a graphic artist "drawing up" pictures (to specifications laid down by a knowledge engineer and/or domain expert) on the screen of a CRT using an electronic paint-package such as Microsoft's Paintbrush. 19 The second method of picture production involves the use of a digital image scanner to digitise existing paper-based artwork that is relevant to the creation of a multi-media consultation. The third method involves obtaining video images from a camera, video tape or video disk and digitising them. The digital images that are produced by this method are then "clipped" and/or compressed in order to conserve disk storage space.

Virtually all of the Prolog code presented in the previous section is directly applicable to the use of the digital optical storage devices illustrated in Figs 6(B) and (C). However, instead of using individual track addresses to reference image data, file names must be used instead. Therefore, in order to be able to use the Prolog code described above, the file-naming conven- tions used in the digital image stores must be made identical to the track addressing system used in the analogue image store. In other words, a file such as 23489.POX in the digital system would be used to store the digital counterpart of the image held on track 23489 within the analogue image store. The showpic code used in Fig. 7 (and elsewhere) would then need to be modified so as to send an appropriate file name across to the command processor within the digital image store. The following simple redefinition would accom- plish this:

showpic(X):- tell(printer), write(X), write('.PCX'), hi, tell(user).

Provided the file-naming convention described above is adopted, all the Prolog routines discussed previously


now become directly usable within expert systems that utilise digital image stores of the type illustrated in Figs 6(B) and (C).

In the author's work with digital images for expert systems both rewritable and read-only optical disks have been used for picture storage. The re-writable disk systems are the easiest to use but are far more costly to purchase. However, they do enable image store production to be undertaken totally in-house. In contrast, when conventional CDROM is used, the disks that are needed for any particular expert system have to be mastered and replicated externally.

EXAMPLES

The techniques described in the previous section have been used to create mixed-mode, multi-media expert-system consultation dialogues in a variety of different application areas such as electronics, instrumentation, mechanical engineering, chemical engineering, manufacturing, process control, medicine and selection interviewing. This section briefly outlines some examples of those applications which appear to have benefitted most from the use of pictorial augmentation.

Within almost every area of electronics, instrumentation and electrical engineering, diagrams and pictures are widely used in the representation and communication of knowledge. Because of the substantial utility of pictorial forms in these areas, the use of video disk imagery, digital graphics and image scanning have been explored as a means of obtaining pictures of electronic equipment, system block diagrams, line drawings, instrument front panels and electronic circuits for use in expert-system dialogues. To date the best results (in terms of image quality) have been obtained using video disk, digitised colour photographs and system block diagrams. Due to the limitations presented by the limited resolution offered by EGA computer screens, experience with digitised circuit diagrams (and those graphically produced) has been restricted to relatively simple circuits--in order to achieve reasonable legi- bility. However, the use of VGA and enhanced VGA screens should enable this limitation to be overcome.

Another major area in which the use of images has been explored is in consultations with medical expert systems. Some of this work, involving the use of the KnowledgePro knowledge processing environment has previously been described elsewhere, z Currently, three exploratory projects are being developed in this area. The first involves building a knowledge-based pictorial support system for the Oxford Textbook of Medicine (OTM) on compact disk. 2° OTM is essentially an electronic book that is based entirely on textual material. Users acess the contents of the CDROM by means of sophisticated and complex search-and-retrieval software that is, unfortunately, quite difficult to use. Investigations are therefore under way into the possi-

bility of augmenting the basic CDROM through the provision of an expert system facility that will: (a) make the system easier to use; and (b) provide access to a supporting image database. The second medical project is the creation of graphical interfaces to medical expert systems that are capable of providing multiple levels of explanation, depending upon the prior experience and knowledge of their user. The images used in this system are digital images which are stored on compact disk; the expert system is used to model the user and keep track of his/her level of knowledge about the various topics of consultation. Depending upon the degree of understanding shown by the user, the expert system determines an appropriate level of response and then composes suitable textual and pictorial advice. The third project (in the area of physiotherapy) is an expert system that gives textual advice about treatments for various types of injury, and augments this advice by means of appropriate images taken from a supporting video disk. It is anticipated that this system will form the basis of a multi-media "expert advisor" on treatments for sports-related injuries.

The third area where the use of pictorial support is being explored is process control. 51° This work has been investigating the various ways in which picture- driven multi-media dialogue within expert systems can be used to help process-plant operators make control decisions relating to the operation of complex process- plant equipment. Pictures have primarily have been used to depict: (1) data trends in graphic form; (2) patterns of indicator and status lights on process plant equipment; and (3) critical relationships that exist between different items of equipment within continuous-flow production lines. In this work the expert systems embedded within the software support facility are used to help human operators understand, interpret and react to messages produced by complex manufacturing equipment. The rationale underlying this work is based upon the high bandwidth of communication that can be achieved using pictorial communication techniques compared with the use of textual methods. Decision-making in environments of this sort is highly time-critical if high throughput and high levels of operating efficiency are to be achieved. It is believed that pictorial dialogue methods can help to achieve these goals.

DISCUSSION

The pictorial dimension is important in many expert- system applications--particularly, in the areas of design, process control, image analysis, medicine, security and surveillance. Of course, when considering the use of pictorial support within expert systems there are two important aspects that need to be taken into account.

First, it is important to consider the use of pictures within consultation dialogues in order to augment the


textual communication that takes place between the user and the system. For example, in input mode during a consultation, the user might be shown some pictures that are relevant to proving some goal; he/she might then be asked to decide which one most closely describes the situation in hand. Similarly, in advice- giving mode, an expert system might present a graphic image which describes a possible solution to a problem being solved--for example, a picture showing the best route between two locations, taking into account a given set of constraints.

Second, it is important to be able to develop methods of inferencing about the knowledge content of pictures or sequences of pictures. For example, if an object A is stationary and the distance from object A to object B is less in picture N than in picture N + 1 then infer that object B is moving away from object A. Similarly, if an image N contains a set of features F(X) that are constrained within a containment C(Y) such that the spatial arrangement S(X) holds, then infer that image N contains an object of class Q.

Although the author has been involved in work involving inferencing methods for pictures (see below), this paper has been primarily concerned with the first of the above two aspects of using pictures within expert systems, that is, pictorial support for consultation dialogues.

Many knowledge-engineering environments provide built-in facilities for displaying pictorial data during consultation dialogues. The KnowledgePro system, for example, provides commands such as PICTURE and VIDI=O_$EARCH to facilitate the display of digital and analogue images, respectively. 2 Similarly, the G2 system that is widely used for real-time applications, provides an object-oriented environment for the display of graphic data. 21 It can be used to facilitate the creation of system schematics, block diagrams, flow diagrams, and so on, that describe the application domain being studied. Unfortunately, most of the less- expensive expert-system shells do not provide facilities of this sort--in which case, the techniques described in this paper might prove useful.

There are three major limitations of the work that has been described in this paper. The first relates to the way in which pictorial material is displayed. The second is concerned with the reactivity of the images that are used in the consultation dialogues. The third limitation arises as a result of the method that used for image indexing. Each one of these limitations is briefly discussed in the remainder of this section.

As can be seen from Figs 6(A, B and C), each of the multi-media consultation systems that has been described uses two display screens for information display--one for text and another for pictures. This type of twin-screen arrangement appears both cumber- some and unnecessary. Investigation is therefore pro- ceeding into exploring ways of dispensing with the separate image display screen, by incorporating both

the text and pictures into a single windowed display similar to that depicted schematically in Fig. 6(D). A suitable interface board which has recently been pur- chased will allow this development to be undertaken. 22 This equipment should also allow several images to be displayed simultaneously--each within its own window area--as well as moving image sequences.

Unfortunately, the images used within the examples described earlier show no reactivity. In other words, users are not allowed to interact with them in order to specify data values or spatial relationships. Of course, this is a major limitation in situations where the user may wish to specify the relative or absolute position of objects in candidate pictures for display. 23 This limitation could be removed through the provision of either mouse support (for high-resolution pointing oper- ations) or a touch-screen (for low-resolution work). Some progress has been made with respect to the provision of peripheral device drivers to support these requirements 24 and it is hoped to include these extensions in future developments of this work.

The third major limitation that needs to be overcome is the way in which the pictures are currently indexed. At present, this is done entirely manually. A knowledge engineer or domain expert provides sets of keyword descriptors that are then coded into the types of relationship that were illustrated earlier. However, for large image stores, it would be useful if this could be done automatically. For this reason we have become interested in applying image-analysis techniques to digital images that have been produced by digitising pictures taken from video disk, video tape and a video camera. Various approaches to feature analysis in con- junction with shape grammars are being used, in order to identify objects in images which can then be used to index the pictures. 25' 26 Some success has been achieved with pictures containing simple geometrical shapes that do not overlap each other. However, the successful application of the method to the indexing of complex pictures is still some way off.

CONCLUSION

Expert systems are playing an increasingly important role in many areas of design, control and decision- making. The need for such software to accommodate multi-modal, multi-media consultation dialogues is now well-established. This paper has demonstrated how text-oriented expert systems that are produced using a low-cost development shell can be enhanced through the incorporation of pictorial forms that are retrieved from either analogue or digital image stores. Prolog code for retrieving images based on a variety of different techniques has been presented. Particular importance has been attached to the use of keyword descriptors. The limitations of the approach have been described and the need for the development of automatic picture-indexing methods has been briefly discussed.


REFERENCES

1. Barker P. G. Expert systems in engineering education. Engng Appfic. Arti]~ InteU. 1, 47-58 (1988).

2. Barker P. G. KnowledgePro--a review and assessment. Engng Appl. Artif. Intell. 2, 325-338 (1989).

3. Laurent J. P., Ayel J., Thome F. and Zeibelin D. Comparative evaluation of three expert system tools: KEE, knowledge craft, and ART. Knowledge Engng Rev. I, (4), 18-29 (1986).

4. Hart A. Knowledge Acquisition for Expert Systems. Kogan Page, London (1986).

5. Barker P. G. Some artificial intelligence techniques for the interpretation of experimental data. Paper presented at the Conference on Sensors and Sensing, Dublin City University, Ireland (1990),

6. Barker P. G. and Manji K. A. Pictorial knowledge bases. In People and computers" IlL Proc. Third Conf. British Computer Society Human-Computer Interaction Specialist Group, University of Exeter (Diaper D. and Winder R., Eds), pp. 161- 173. Cambridge University Press, Cambridge (1987).

7. Barker P. G. The HPLC doctor. Engng Applic. Artif. lntell. 3(3), 238-239 (1990).

8. Balagurusamy E. (Ed.) Artificial intelligence in industry and government. Proc. Int. Conf. on Artificial Intelligence in Industry and Government, Hyderabad, India. Macmillan India Limited (1989).

9. Gero J. S. (Ed.) Artificial intelligence in design. Proc. Fourth lnt. Conf. Appfication of Artificial Intelligence in Engineering. Springer, Cambridge (1989).

10. Barker P. G. Expert systems in process control. In Artificial intelligence in industry and government. Proc. Int. Conf. Artificial hztelligence in Industry and Government, Hyderabad, India (Balagurusamy E., Ed.), pp. 413-423. Macmillan India Limited (1989).

11. Barker P. G. Expert systems in analytical chemistry. Irish Chemical News, pp. 28-32. Autumn (1989).

12. Hohne B. A, and Pierce T. H. (Eds) Expert Systems in

Chemistry, ACS Symposium Series No 408. American Chemical Society, Washington, DC (1989).

13. Expert Systems International, ESP Advisor User Guide and Reference Manual (Issue 2), Oxford (1985).

14. Expert Systems International, Prolog-2 Technical Reference Manual (Issue 2), Oxford (1985).

15. Jones K. P. Expert systems as desk-top assistants. MSc Thesis, School of Information Engineering, Teesside Polytechnic, Cleveland, U.K. (1988).

16. Rogers J. B. A Prolog Primer. Addison-Wesley, Reading, MA (1986).

17. Bratko I. Prolog Programming it: Artificial Intelligence. Addison-Wesley, Reading, MA (1986),

18. Hitachi New Media, MS-DOS CD-ROM Extensions (Version 2.10), Software Interface Specification, Hitachi, Hayes, Middlesex (1989).

19. Microsoft Corporation, Paintbrush User's Guide, Microsoft Corporation, Redmond, WA, 1987.

20. Weatherall D. J., Ledingham J. G. G. and Warrell D. A. (Eds) Oxford Textbook of Medicine, Electronic Version. Oxford University Press, Oxford, 1989.

21. Moore R. L. G2: chemical process control. In Expert Systems in Chemistry, ACS Symposium Series No 408, pp. 169-179. (Hohne B. A. and Pierce T. H., Eds). American Chemical Society, Washington, DC (1989).

22. VideoLogic Ltd, MIC System (Version 3.3)--DVA Supplement, VideoLogic Ltd, Kings Langley, Hertfordshire (1990).

23. Charles S. A study of spatial data models and their application to selecting information from pictorial databases. Ph,D. Thesis, University of Loughborough (1990).

24. Lamont C. Pictorial interfaces to an expert system. GDSE Dissertation, Teesside Polytechnic, Cleveland (1989).

25. Chang S. K., Principles of Pictorial Information Systems Design. Prentice-Hall, Englewood Cliffs, NJ (1989).

26. Chellappa R. and Sawchuk A. A. (Eds) Digital Image Processing and Analysis', Volume 2: Digital Image Analysis. IEEE Computer Society Press, New York (1985).

Using pictures in expert systems

Documents

Transcript of Using pictures in expert systems