[Hayes-1997] IA What Works and What Doesn’t

7/28/2019 [Hayes-1997] IA What Works and What Doesnt

1/16

s AI has been well supported by government

research and development dollars for decades

now, and people are beginning to ask hard ques-

tions: What really works? What are the limits?What doesnt work as advertised? What isnt like-

ly to work? What isnt affordable?

This article holds a mirror up to the community,

both to provide feedback and stimulate more self-

assessment. The significant accomplishments and

strengths of th e field are hi ghlighted. Th e

research agenda, strategy, and heuristics are

reviewed, and a change of course is recommend-

ed to improve the fields ability to produce

reusable and interoperable components.

Ihave been invited to assess the status ofprogress in AI and, specifically, to address

the question of what works and what doesnot. This question is motivated by the beliefthat progress in the field has been uneven,that many of the objectives have beenachieved, but other aspirations remainunfulfilled. I think those of us whove been inthe field for some time realize what a chal-lenge it is to apply AI successfully to almostanything. The field is full of useful findingsand techniques; however, there are manychallenges that people have forecast the fieldwould have resolved or produced solutions toby now that have not been met.

Thus, the goals that I have set for this arti-cle are basically to encourage us to look inthe mirror and do a self-assessment. I haveto tell you that Im in many places right nowwhere people often jest about what a sorrystate the field of AI is in or what a failure itwas. And I dont think thats true at all. Idont think the people who have these opin-ions are very well informed, yet theres obvi-ously a germ of truth in all this. I want to talk

about some areas where I think the field actu-ally has some problems in the way it goesabout doing its work and try to build a shared

perception with you about what most of theareas of strength are.

I think there are some new opportunitiesowing to the fact that we have accomplisheda good deal collectively, and the key fundingorganizations, such as the Defense AdvancedResearch Projects Agency (DARPA), recognizethis. In addition, the Department of Defense(DOD) is increasingly relying on DARPA toproduce solutions to some challenging prob-lems that require AI technology. These sig-nificant problems create opportunities fortodays researchers and practitioners. If Icould stimulate you to focus some of your

energies on these new problem areas andthese new opportunities, I would be satisfied.

At the outset, I want to give a couple of dis-claimers. Im not pretending here to do acomprehensive survey of the field. I actuallyparticipated in such an effort recently, theresults of which were published in the Com-mun i cat i ons o f t he ACM(Hayes-Roth andJacobstein 1994). In that effort, I tried to beobjective and comprehensive. In this article,however, Im going to try to tell you candidlyand informally the way it looks to me, and Iwould entertain disagreement and discussiongladly. However, I think any kind of judg-ment is value laden. I do have some values.Theyre not necessarily the same as others,but I think they represent a pretty good cross-section of the viewpoints of many of the peo-ple in the field and many of the people whopatronize the field (in both senses).

My values are that a field ought to be ableto demonstrate incremental progress, not nec-essarily every day, every week, but over the

Articles

SUMMER 1997 99

Artificial IntelligenceWhat Works and Wh at Doesnt?

Frederick Hayes-Roth

Copyright 1997, American Association for Artificial Intelligence. All rights reserved. 0738-4602-1997 / $2.00

This paper is copyrighted by the American Association for Artificial Intelligence. All rights reserved.


2/16

have pointed out, you must be concernedthat the techniques we demonstrate in thelaboratory carry over and actually producematerial benefit; in short, do our techniquesscale up effectively?

What are the questions that I want to focuson? I tried to put myself in the position ofthe people that the conference chairs wereasking me to think about. Theyre largelygovernment research and development peo-ple, such as some of the DARPA programmanagers who come out of the AI communi-ty, go to DARPA, and immediately try todefend AI and promote it and keep it vital.And, I tell you, in these environments, peoplereally are asking what they consider to behard and pointed questions. You would thinkthat these people would have good answersreadily available. However, although manypeople have tried to address such questions,it appears difficult to boil down the answersto these questions.

What do we have that really works, andwhat are the constraints that determinewhether it works or whether it breaks? Whatare these false advertisements that peoplecontend that the field has been rife with?And are we willing to say that there are cer-tain things that are just beyond the limits ofwhat might be achieved? What are these?

These days, affordability is a big issue. TheUnited States, of course, is in a period ofadjustment with a very competitive globalenvironment. Every day, theres a trade-offbeing made between politically expedientcost reductions and long-term potential pay-

offs, and affordability is an extremely impor-tant concept these days, much more thanbefore. I think all this is an attempt on mypart to rephrase the debate because absentthis, many people in Washington will main-tain the position that, Well, AI, wasnt thatthe thing we talked about a decade ago, andif its not done, didnt it fail?

To a large extent, AIat least the symbolicpart of AIis built around the integration ofthree basic capabilities: How do you representthings so that you can do problem solving onthem, the problem solving being produced byinferences drawn by various kinds of engineapplied to various kinds of reasoning logicand then, because most problems are expo-nentially difficult, there has to be some con-trol, and control is often the differencebetween having something thats usable andhaving something thats not even interesting,in fact (figure 1).

To apply these simple elements, the fieldlearned quite awhile ago that you had to add

course of years and decades, certainly youought to be able to demonstrate that yourestanding at a higher level than you were yearsbefore. I think to a large extent that the fieldis subsidized as a darling of the domestic andmilitary research and development communi-ties, and we owe them something for all theinvestment. To the extent to which we cancontribute solutions to problems in thoseareas, I think thats good.

I have been an engineer, Ive been a tech-nologist, and Ive been a scientist. I thinkthese are different enterprises. Science, for me,is discovery of truth from observation. Engi-neeringis what we do when were asked tobuild something, even perhaps without cer-tainty that the methods we pursue will leadto a successful outcome and perhaps not evenknowing why it works when were done. Andtechnologyfeeds off these other two fields andfeeds into them by producing reusable com-ponents and processes. In many ways, civi-lization is an accumulation of these techno-logical advances, and I think we ought to beable to assess where we are in producingthese reusable technological elements.

Certainly, no matter which area youre in,you have to be interested in the repeatabilityof your results and whether they generalizefrom one situation to another. In addition, asseveral of the American Association forArtificial Intelligence (AAAI) and InnovativeApplications of Artificial Intelligence papers

Articles

100 AI MAGAZINE

Representation

Inference

Control

Figure 1. The Three Basic Element s of Symbol ic Reasoning.



3/16

something to this, and Ive called out herethree typical kinds knowledge that have to beadded to make this work (figure 2). First, con-sider domain knowledge. What is that about?Most often, its a description, a model, of theessential features of the otherwise arbitrarilycomplicated and continuous world. Whichthings are we going to talk about? Which arewe going to pay attention to? How are wegoing to choose a representation that affordsreasonable computation?

The difference between the knowledge anapprentice gets in a basic introductory hand-book to a field and that which you get byworking in the field for a long time is sub-stantial. Expert systems, for example, grew upon the idea that you could extract fromhumans who were really proficient in an areatheir expertise, and if you had this expertise,with almost any reasonable inference mecha-nism you could implement the expertise as aseries of deductive rules of one sort or anoth-

er. If people were already solving problemsmore or less consciously and symbolically,control was probably not extremely difficultbecause humans, of course, dont do expo-nential computing consciously. And so, withthe expertise that we had extracted viaknowledge engineering, we could implementa basic problem-solving approach. We wouldimplement the rules required to derive validanswers to given problems.

Finally, there are a number of areas whereproblems are of such complexity or where theextractable knowledge has limited ability todrive immediately to a solution that we

encounter a requirement for a kind of controlknowledge. Ill call this man agement know-how. Its knowledge about managing problem-solving resources, knowledge about how torecognize in a problem space which areasmight be most fruitful or least promising topursue first.

As people begin to implement applicationsystems, they take these various knowledgeelements and embed them in an application,which here Im just calling a probl em solver(figure 3). The problem solver in an integrat-ed application brings all these elementstogether. The basic elements of control, infer-ence, and representation form the founda-tion. The three kinds of knowledge are alsopresent. But something has been added toenable us to build an application. Here forlack of a better term, Ive labeled this a prob-lem-solvi ng archi tecture, by which I mean aframework for inserting these various ele-ments and having them all plug and playtogether. We have a lot of problem-solving

Figure 2. T he Sim ple Model M ust Be Elaborated to Show the

Ki nds of Know ledge Requi red to In form and Use It .

Articles

SUMMER 1997 101

Representation

Inference

Control

Domain

Knowledge

DomainExpertise

Management

Know-How

Figure 3. The Knowl edge Must Be Plugged i nt o a Problem-Solvi ng

Archi tecture That Implements Essent ial Reasoni ng Capabil it ies.

A Problem-Solver

Representation

Inference

Control Domain M odel

Problem-Solving

Architecture

Managemen t

Know-How

Doma inExpertise



4/16

I want to distinguish between the problemsolver and what a technologist might call an

application or a functional component. The

diagram (figure 4) is getting a little complicat-

ed; I have to apologize for this. But I want todistinguish here between the inside and the

outside. The inside, if you will, is this prob-

lem solver. Its the pure embodiment ofknowledge to do a task. And now, somehow

this task solution is going to be made func-tional and effective by inserting it into an

environment in which it actually adds value.The di fference here is between what you

might call a pure technical solutionthe

insideand what is required for that to be asolution in any organization, which here you

might call the membrane, or the outside.

There are actually five different interfaces,

architectures in the field. The earliest, say,

were the back-chaining rule-based systems or

the forward-chaining rule-based systems. We

have the blackboard systems, case-based rea-

soning systems, and others.

Each one of these problem-solving archi-

tectures turns out to be extremely important

to us because today we are not generally able

to represent knowledge in a manner that is

independent of the implementation milieu.Knowing something about the architecture

that were going to embed our knowledge in

enables us to do something thats essential to

building an actual system, which is to decide

how to shape or mold or craft, if you will

how to phrase the knowledgeso that it

interoperates effectively with the other bits of

knowledge.

Figure 4. The Problem Solver Operates in a Context of Several Environm ent s and,

Th us, Becomes a Functional Component of a Larger System.

Articles

102 AI MAGAZINE

Functional

Component

Systems and Resources Envt .

Dev

elo

pm

en

tEnv

t.

A Prob lem-Solver

Representation

Inference

Con t rol Domain Model

Problem-Solving

Architecture

Management

Know-How

Doma in

Expertise

User Int erface

Opera

tion

alEnv

t.

Task and Informat ion Envt .



5/16

if you will, which we dont normally focus onwhen were talking about the pure AI tech-nology. And these interfaces, or other envi-ronments that we must interface with, actual-ly consume almost all the effort when you tryto make the pure technology applicable andeffective. The first one, and closest to theproblem solver itself, is the user in terface.Ordinarily, our systems are created to supporthumans, and weve utilized a range ofapproaches to make inference accessible,meaningful, and acceptable to the users. Ofcourse, depending on how the user or opera-tor wants to interact with a system, the prob-lem solver can vary from a nearlyautonomous system to a nearly manual sys-tem, yet the internals need to be approxi-mately the same. There still needs to be a rep-resentation of the problem, a representationof the state of the problem solving, and someway to direct control toward the mostpromising directions. Thus, although the user

interface for many researchers has been anafterthought, from the users point of view,the user interface is the application. It repre-sents a huge percentage of the actual develop-ment cost and certainly cannot really beignored.

Now off in the easterly direction in thefigure, toward the right, let me talk about theoperati onal environment. Consider, as an exam-ple, the problem of scheduling, an area ofextensive research in AI. To produce a usefulsolution, one needs to understand how orga-nizations use schedules. Schedules represent,in the simplest case, agreed-on solutions to

task and resource-allocation problems. In oth-er cases, they represent contracts. They repre-sent in other cases actual production-controlmechanisms. Each of these uses is very differ-ent. You can take the same application andtry to put it into different operational set-tings, and you will find that it will need to beextremely different.

The task and informat ion environm entturnsout to be a major issue because almost all AIapplications require access to a lot of infor-mation. Getting the information in from itssources has often been referred to as a bottle-neck or an integration problem or a knowl-edge-acquisition problem or a learning prob-lem or a maintenance problem. But theinterface between the inside and the outsidethrough this information environment is amajor problem and one for which we haverelatively poor tools today.

On the flip side, we have the systems andresources that this functional component isgoing to exploit, or interoperate with. These

are often computer systems. Theyre oftennetwork systems today, and to a large extent,the applications that we build today have nomodel or view of this environment. We do alot of ad hoc construction to embed or inte-grate the AI solution in various computingenvironments; Ive seen delays of years tomake something work across this barrier.

Finally, on the left edge of the figure, wesee the developmen t env i r onm en t , whichincludes things developers often think about:what are the tools that we have available;what are the languages, compilers, debuggers;and so forth. The field of AI, I think, is afflict-ed by a major problem here. We used to havethe worlds best environments. We wereworking with Lisp environments, for exam-ple, Interlisp and the Massachusetts Instituteof Technology Lisps and their descendants.Nearly all the ventures to improve and propa-gate these environments are gone now. Theones that remain are rather marginal in the

computing field as a whole. Thus, most of thepeople who are worried about integratinginto these other environments have decidedthat they have to go with mainstream com-puting and development technologies, whichare really poor. Theres no question that tech-nically, this decision represents a huge stepbackward. We get small footprints, reasonableperformance, and lousy development envi-ronments. Its a bad bargain.

I want to look at the problem-solving capa-bilities in a context-free way for a moment.Well come back to what I might call the con-text of the problem solver shortly. Table 1 is

an effort to summarize what works. I mighthave omitted a few things, but given myemphasis on symbolic and knowledge-basedtechniques, this is a pretty good summary ofwhat works. We have basically four sorts ofproven technology in the areas of representa-tion, inference, control, and problem-solvingarchitectures. I will allude to each and makesome global assessments without attemptingto provide a tutorial on the various tech-niques here.

In representation, we have some reason-ably good languages. We have a lot of experi-ence in domain modeling, much more thanthe rest of the world thats coming to it now.In fact, having done knowledge engineeringfor a couple of decades, I think we know bestwhat works in this area. I often tell peoplethat I believe that knowledge engineeringalways works if you satisfy the applicabilityconditions. That is, if you can find somebodywho does a reasoning task, does it repeatedly,and does it well, you can ultimately elicit this

Articles

SUMMER 1997 103



6/16

ed messaging for in the first place. Theyhavent gotten to semantics yet. We havedeveloped a variety of ways to develop dis-tributed or cooperative or loosely coupled ele-ments of computing from agenda mecha-nisms, blackboard mechanisms, and so forth.We have a good handle on the representationof search; we have very good algorithms that,when the shoe fits, provide an excellentsolution. Finally, we have a tremendousamount of experience specializing these algo-rithms and applying them, for example, toscheduling problems.

In the architecture area, we have techniquesfor rule-based, object-oriented, frame-based,constraint-based, and blackboard methods aswell as heuristic classification and task-specificshells. This body of methods and techniquesis a significant accomplishment; so, when Ilook at the results and summarize what wasdone over the past few decades, I see that weactually created a large amount of useful andimportant technology.

These important technology results werefalling out, if you will, as a side effect, of con-ducting research in a certain way. In at leastfive areas, shown in table 2, the field hasfocused persistently and with continuousincremental progress. In these areas, we havereached a point where even if we dont havesolutions in hand, we have demonstrablyusable, if somewhat limited, technology. Letsbriefly assess each of these areas.

The first area includes recognition, inter-pretation, and understanding. Today, you canpurchase and employ constrained speech-

knowledge, represent it, and implement it ina machine. This is an amazing accomplish-ment, and I do not mean to be hyperbolic inthis assessment. In doing such knowledgeengineering, we have come with up numer-ous ways to organize knowledge and make itexplicit. These are familiar. They take differ-ent forms and different combinations, andtheyre all useful.

Under inference, the field has demonstrat-ed how to draw valid conclusions, how to doit under resource constraints, and how to doit using uncertain or errorful methods. These

methods work, and theyre in use on a dailybasis in thousands of organizations. Ive listedhere, among the chaining methods,unification, resolution, and other theorem-proving methods, some of the techniques weemploy to simplify knowledge representationand obtain inference for lower costs, such asinheritance or case-based reasoning. Hypo-thetical reasoning is a somewhat esotericapproach. Its extremely useful to organiza-tions, for example, that have to plan underuncertainty.

In the area of control, AI has inventedmost of the metaphors and paradigms thatwill carry computing forward for a significantperiod of time. We know something aboutwhat it means to be oriented in a directionand to organize computing toward some goalor to proceed forward from some data. Mes-saging is taken for granted today, and in mostareas where messaging is being used today bythe computing industry as a whole, they haveno understanding of what the AI field adopt-

Articles

104 AI MAGAZINE

Table of Contents of Proven AI Techn iques

Representation Languages, Domain Modeling, and Knowledge Engineering

Rules, frames, classes, cases, hierarchies, propositions, constraints, demons, certainty

factors, fuzzy variables

Inference Theorem-Proving, Heuristic Reasoning, and Matching Techniques

Forward and backward-chaining, unification, resolution, inheritance, hypotheticalreasoning, constraint propagation, case-based reasoning

Control Goal and data directed, messaging, demons, focus, agenda, triggers, metaplans,

scheduling, search algorithms

Problem-Solving

Architectures

Rule based, object oriented, framre based, constraint based, blackboard, heuristic

classification, task-specific shells

Table 1. A Concise Summary of M any of t he Proven A I Techn iques.



7/16

understanding systems. This is a huge break-through, one that took nearly 30 years ofcontinuous research and is not done yet.Handwriting, for example, which has been

the butt of so much humor in the popularpress, is much easier than speech and wouldhave been solved, in my opinion, very effec-tively by now if it had been pursued continu-ously. Simply to try and transfer what weknow from speech to handwriting is a chal-lenge, but I do not think were very far awayin time from people having commerciallycredible handwriting systems.

In the area of generation of communica-tion, were not as far along toward powerfulsystems for gesture and expression. We haveexcellent automatic speech-generation sys-tems today. The quality is really superb. Iremember vividly the primitive quality ofspeech synthesis when I visited Bell Labs inthe early 1980s. Given the state of the art atthis time, weve come incredibly far.

In the area of what you might call analysisor diagnosis and the recommendation or pre-scription of how to fix problems, we have lit-erally thousands of applications that are inuse today. They manifest two basic approach-es: The first uses what you might call a strongmethod, typically based on some expert sys-tem shell. In this approach, somebody doessome knowledge engineering and figures out

how to classify the problem variants, how tosort users individual situations into one ofthe set of parameterized categories of prob-lem types, how to pull out a standard recipefor problem response, and how to instantiatethe variables in the recipe with the parame-ters identified during the analysis. This way,many problem variants are treated with thesame parameterized solution.

More recent, but in many ways faster-grow-

ing, is the case-based approach, where thesolution typically uses a more interactive,mixed-initiative system. We still face arequirement to organize the knowledge, but

sometimes, knowledge can be accessed bypurely syntactic means. For example, whenyou call a help desk and ask for some help onsome problem, the person whos there mighthave no real understanding but might be typ-ing in what you say as a keyword search toindex into a case and a script of how to walkyou through a problem-solving session.Although the reasoning here is shallow, theorganization and indexing of knowledge iscritical.

In the planning and scheduling area, Ithink its clear that weve made steady incre-mental progress. We have some truly out-standing results. One of the most impressiveis the work of Doug Smith and his colleaguesat the Kestrel Institute in generating sched-ulers using a software generation technology(Smith, Parra, and Westfold 1996). The refine-ment approach they use requires an intelli-gent human software designer to steer aheuristic search method thats using knowl-edge about algorithm transformations to gen-erate a scheduler. In some recent results, theschedulers that theyre producing have liter-ally leapt off the page, being orders of magni-tude faster, for real-world problems. Although

I think there are other examples in this area, Ijust wanted to call this one out.In the area of in tegrated behavior, where we

try to link sensing and planning and actingtogether, we have in some ways just exceededpeoples expectations. I omitted entirelywhen I put table 2 together simple robots,pick and place robots, things like that. I for-got about them because they were an accom-plishment of the AI field many years ago.

Articles

SUMMER 1997 105

Successes of the AI Research Paradigm

Recognition, interpretation, and

understanding

Dozens of constrained speech and message systems

Gesture, expression, and communication Speech generation

Analysis and Prescription Thousands of domain-specific expert systems

Hundreds of case-based assistantsPlanning and Scheduling A few superb domain-specific systems

Integrated behavior Autonomous and teleoperated vehicles

Table 2. A L ist of Prin cipal Frui tf ul Research Areas.



8/16

ments. If we put them in a dynamicallychanging environment, were not really surehow to control all the things that wed haveto think about. In any case, to get on with

building these systems, we prefer to ignoretangential, environmental, or special case fac-tors; so, we put on blinders, which is goodbecause it helps us to be successful.

Fourth, we look for situations where theresa high degree of repetition because havingmade this investment, youd like to get somepayback more than once, which reduces theamount of payback you need each time youuse it. If you look at the applications, theytend to be in highly stereotypic environ-ments that afford dozens, hundreds, or eventhousands of opportunities to use the samegeneric solution repetitively.

Fifth, most of these projects are small. Ihad a lot of trouble deciding what I meant bysmall. I knew it was a project of less than $50million. Now, for a lot of researchers, thatsounds like a lot of money. But I think, as EdFeigenbaum said, thats probably the price ofa wing on one of these modern aircraft! Thesize of the engineering activity that weengage in is truly small scale today. And, infact, one of the problems that we have whenwe go into other environments is that wedont have in our culture a methodology or aset of practices that enable us to say, Oh,having solved small problems we know howto solve big problems. Yet a lot of what peo-ple look to AI to do is to augment humanintelligence by enabling us to put a lot ofunderstanding and know-how into these sys-tems.

Thus, were good at small things. And oneof the reasons were good at small things isthat we can apply really talented people tothem. We do not have many examples today

Although simple robotic control is in ourrepertoire, I was thinking more about thingsthat are more autonomous and more capable.In particular I think about the NAVLAB (see

www.cs.cmu.edu/afs/cs.cmu.edu/project/alv/member/www/navlab_home_page) example,to cite one, which is the work of Chuck Thor-pe and his colleagues at Carnegie Mellon Uni-versity. We have systems today that, becausetheyre integrated, have taught us the valueof looking at the interactions between theway we sense, the way we represent, and theway we plan, which no one would ever haveappreciated if they had worked the problemsin isolation.

If one does a little analysis of why weresucceeding and then tries to pull out some ofthe heuristics or situational characteristics

that have powered AIs successes, a list ofabout a half dozen ideas emerges (table 3).First, we generally narrow the problems. Theideal thing to do is narrow them in bothdimensions if you can, so they have neithertoo much breadth nor too much depth.Preferably, the chosen problem will have justenough scope so that if you go through theexercise of implementing a system, it will bevaluable. The primary obstacle, of course, isthat it is tedious to pull out knowledge andthen recode it in a way that machines canapply. I dont think weve made muchprogress on this problem.

Second, we often decide what were goingto use a system for and then focus narrowlyon the objective. The system, for example,will reduce time or reduce cost or produce abetter design or something like that. Werarely take on multiple objectives at one time.

Third, if were wiseand we almost alwaysrestrict ourselves to thiswe say, These sys-tems really have to be in fairly stable environ-

Articles

106 AI MAGAZINE

Critical Ingredients in the Typical AI Success

Narrow scope Makes knowledge engineering difficult, hence expensive

Focused objective Assures importance of accomplishment

Stabil ity of environment Allows us to put on blinders and limit concerns

High degree of automation & repetition Gives maximum return for fixed development cost

Small project Less than $50 million, usually much less

Lots of custom work by talented

professionals

Not cookie-cutter applications

Table 3. A Li st of H euri stics for Successful AI Research to D ate.



9/16


10/16

make it impossible for our applications tointeroperate and impede our ability to shareand reuse knowledge. Thats a significantproblem. Second, today, the more useful asystem is when we build it, the less general itis. As a consequence, the more useful thingsare less reusable and adaptable. This problemis significant too. We might be able to reapplysome things but not the most useful things.And, conversely, in some sense, the purer theelement, the less valuable it is. One of theother consequences is that, today, the mea-sured part of systems that is AI is small, and Idont think its increasing. And thats a chal-lenge to the survival of AI. In the arena ofrapidly evolving technology components,you have to expand or basically vanish.

Thus, to some extent, our successes to datehave resulted from selectivity and carefulengineering of context requirements. But asIve said, people expect more of us. In fact,people often expect way too much of us, and

some of our best representatives have beenour greatest problems here. I remarked tomyself during the recent chess tournamenthow amazing it was that were at a point inthe mid-1990s where computer chess is actu-ally reaching the grand-master level. However,I think were 30 years late on the Newell andSimon prediction that we will have a worldchess champion.1 Now, thats a half-emptyand half-full kind of glass. In the case of chess,people are probably not holding our under-achievement against us. The people who arewatching realize what great strides have beenmade. But I remember a book by M argaret

Boden about 10 years ago, where she talkedabout interviewing ordinary people in an air-port in London, and they had definitiveexpectations of where AI was and would be(Boden 1986). To a large extent, those expec-tations were that AI, and the world chesschampion, would be here by now, and againstthese expectations, we are late.

What were people expecting we wouldhave by now? Well, we would have intelli-gent machines. I havent met one yet. Ihavent met one thats even close to it. Wewould have superhuman capabilities. Well,we have calculators, and we have symbolicexpression solvers, and I suppose theyresuperhuman. We have routers and sched-ulers, but these are really narrow applications.Wed have machines that we could talk to bynow. I mean, the HAL of2001is only a fewyears away! And, in fact, we can speak tomachines now with a fair amount of training.Ive tried the latest and greatest, and I foundit a little bit daunting. Although weve been

a speech system thats improved by havingsome visual context for whats being referredto. We also cannot combine it with a plannerso that it understands some input and someof the contextual variables. We cannot gener-ally lift out the knowledge from one applica-tion and use it in another. And I think at itsroot, this is a major failing of AI. It is tooexpensive to do knowledge acquisition andknowledge engineering, and we have practi-cally no technology to lift the knowledge outand reapply it.

I think we might be coming to the end ofour research paradigm. This paradigm hasreally worked for us, but its really problemat-ic, and I dont think its going to carry us intothe next century. It certainly is not going toenable us, I believe, to get to the places wherethe patrons think the technology is worthspending money on. Lets talk a little aboutthis paradigm. Im trying to boil down andsynthesize what we do. First, we focus on lit-

tle things, pure things, things that standalone, things that are context independent.We leave as an exercise to others to figure outhow to adapt them and apply them and treatthis adaptation as nonessential. In fact, formany students of the field, we reward onlythe new and the different.

The second point is that, Hey! Were justreally good scientists. I mean the actualheuristics were exploiting work well in otherresearch fields. We should not be surprisedthat they work here too. We have won, tothis point, by taking complex problems anddividing them and treating each of the sub-

problems separately. We specialize within ourfield in functional subareas and get good atthe things we work on. We have noticed, fur-ther, that these subarea technologies alsoneed to be specialized by domain of applica-tion, and this domain-specific approach pro-vides a tremendous source of leverage. In fact,many of the people who might have been atthis AAAI conference are, for example, at SIG-GRAPH right now, applying AI to graphics.Finally, like good scientists, we try to adhereto ceteris pari busas a heuristic: Keep every-thing else, other than the essential, constant,so that you can manipulate the key variables.Unfortunately, this approach leads us basical-ly to ignore or control the environment asopposed to study the environment.

Although these heuristics have worked wellfor us over time, as they have in other sci-ences, I think theyre fundamentally prob-lematic for where we want to go. First, thedifferences among the various contexts inwhich we want to embed our applications

I think wemi ght be

coming to theend of ourresearch

paradigm.

Thisparadigm hasreal ly worked

for us, buti ts reall y

problematic,and I dont

think it s

going tocarry us

intothe

nextcentury.

Articles

108 AI MAGAZINE



11/16

making continuous progress there, were late.And, of course, wed have autonomousmachines. I guess thats what some of usmean by agents, and these would be ubiqui-tous. Now, having all these things in hand,we would be filthy rich as a nation.

The reality is probably almost 180 degreesaway from these expectations. All the currentfeats of AI today have been enormous engi-neering undertakings. We dont have an enor-mous field, we dont have billions going intothe field; so, the results are limited. The scopeof implemented knowledge today is reallysmall. Although we repeatedly demonstratethat the technology works in particular appli-cations, we have little productivity gain fromone application to the next. We have verypoor technology for evolution of the systemsweve built. We have almost no examples oftaking independently developed applications,combining them, and producing somethingthat makes incremental progress toward AI.

Thus, the difference between expectationsand achievements is really large, and this is avery serious public relations (PR) problem.

I think if wed had a PR firm, and wedmanaged it a little more carefully, wed proba-bly have a great reputation because we actual-ly deserve an outstanding reputation. You cango through the syllabus, or the product cata-log, of contemporary computing and trace ahuge fraction of state-of-the-art technologiesto seminal and continuous results in AI:development environments, robots, speech,text, distributed computing, data modeling,computer-aided design, and computer-aided

engineeringall these are built on these basicelements of representation, inference, andcontrol. The field doesnt get much credit forthese contributions because few people areinterested in tracing the origins of currenttechnology.

The application of AI within business hasbeen another major success. Hundreds oforganizations have implemented thousandsof applications using such AI technology ele-ments as expert systems, case-based reason-ing, configuration generators, and machinevision. Multimillion dollar returns on invest-ment are not uncommon because many orga-nizations have situations that are ideal forapplying this highly specialized technology.They have tasks that occur frequently; arerepetitive; have moderate knowledge require-ments; and can be operated in a tightly con-strained, relatively nondynamic, applicationenvironment. In such situations, for example,a system can be built to leverage a service de-partment by reducing errors, reducing time,

saving moneythats a formula for commer-cial success. But, of course, the field aspires toa lot more than such mundane economic suc-cesses. In fact, we have achieved even biggerpractical successes. For example, in the Per-sian Gulf War, one AI application used in tar-geting and scheduling of air flights wasjudged to be enormously valuable. The DODdetermined that this single application pro-duced such significant savings, not to men-tion its effect on accelerating operations, thatit more than compensated for the total DODinvestment, through all time, in the field ofAI. Thus, in some sense, our balance wentpositive for a little while.

Continuing our review of successes, we cansee positive signs even in the hard problemssuch as natural language, vision, domainmodeling, software generation, planning, andintelligent control. In several of these reallyhard problems, we are making steadyprogress, with which you are probably famil-

iar. The fact that we can demonstrate incre-mental progress against such hard and essen-tial problems is a credit to the ingenuity andperseverance of the researchers in the field.Nevertheless, there is a disparity betweenwhat people expect of us and where the tech-nology is today. And we live in a world ofincreasing budget constraints. As a conse-quence, I dont think either our paradigm orour environment allows us to keep going thesame old way.

I recommend a change in the agenda, fromthe familiar problems to what I might callsome twenty-firstcentury issues. I want to

talk about four of these, as listed in table 4.First comes a multifaceted, multiplicity ofcapabilities, followed by a cooperative role forour technology, affordability, and then man-ageability.

I think that actually it was my wife, Bar-bara Hayes-Roth, who coined the phrase mul-ti faceted int ell igencefor some of her applica-tions; so, let me credit her. I think its an aptphrase. What we want in our systems issomething like we expect from people: thatthey can shift from one task to another taskand, in shifting, they dont lose state, andthey dont need to stop the world for a fewyears and recode their knowledge so that itcan be reapplied in the new task context.Thus, a multifaceted intelligence presumablywould have a bunch of different componentcapabilities. It would be able to demonstratecontributions in many different functions. Itcould perform a variety of tasks, and itwould, of course, understand a lot moreabout the world because we would need to

Articles

SUMMER 1997 109

Inthelongterm,AI isessential.



12/16

weak process, employing very weak methods.We dont really know whether were going tobe successful when we start a new kind ofapplication, which is a problem.

I think all these points lead to a newparadigm, which I want to try to illustrate.Figure 6 shows an architectural framework, tobe defined, which would allow us to integratefour different kinds of knowledge, whichcould be acquired independently of oneanother and then applied to support multipletasks, such as vision and planning and con-trol. Moreover, these tasks, such as vision andplanning and control, could be performed inmultiple domains, such as on the highwaysand in a building. This is a goal. If we couldachieve this goal, we would, in fact, have atechnological basis for sharing that wouldproduce economy and efficiency. As previous-ly discussed, with todays state of the art,knowledge needed to do a function in sup-port of a particular task, in a particulardomain, is usually uniquely crafted for anyparticular system environment. Today, we arefar from having the kind of independence ofknowledge modules suggested by figure 6.

Thus, I think if were going to pursue therecommended agenda, we have to have somesustained support for what people might calllong-term or large projects. And here Imaddressing the people in government, whoare the patrons of a lot of this work. Sus-tained pursuit of problems is not populartoday in government, but only the govern-ment is equipped to undertake the impliedchallenges.

communicate with it over a wider range ofrequirements.

Cooperation actually is something that thefield has been pursuing probably for 20 yearsalong a certain set of pathstheyre likethreadsbut I really think its a tapestry thatneeds to be produced now. We have multipleand varied agents, and they need to cooper-ate. Often the agents are people. Often, thepeople need to share state with machines,and this is dynamic state. This is a real chal-lenge because our ability to express where weare in a problem or to have a machine com-municate its state to us is limited by thechannels available, the amount of time werewilling to expend, and the bandwidth of ourinteraction. Usually, in most of the tasks thatrequire cooperation, we are time stressed andare unwilling to convey explicitly all the stateinformation that would be required to alignour agents adequately for significant tasks.

I want to call out as the third focus area theproblem of affordability. I dont have a lot ofsimple solutions to address this requirement.The obvious strategy is to reuse stuff, and itsclear that the critical thing to reuse is the

knowledge that we put into these systems. Tosupport knowledge reuse, not only do wehave to figure out some way to make it becontext adaptable, but we also need to applythe technology directly in support of our ownprocesses for acquisition and maintenance.

Finally, it would be nice, going back to theobservation that its a small-project field, tohave something like manageability or pre-dictability of the process. We have a very

Articles

110 AI MAGAZINE

Critical Ingredients in the Typical AI Success

M ultifaceted intelligence Comprises many components

Exhibits many functions

Supports many tasks

Understands and knows a lot

Cooperation Multiple and heterogeneous agents Comparative advantages

Bandwidth limited

Time stressed

Affordability Knowledge and component reuse

Knowledge-based knowledge acquisition and update

Manageability Estimable, predictable, testable development processes

Tabl e 4. A Suggested Li st of N ew Focus Problems for A I Research.



13/16

What would the recommended strategybe? First, the principal strategy shouldemphasize context-adaptable building blocks,reusable knowledge, high-value componentfunctions, composition architectures for mul-titask systems, and an architecture for seman-tic interoperability. Second, a small numberof domains must be attacked persistently.Third, integrated performance across multipletasks is the first criterion. Fourth, adaptabilityand transferability are the goals. Fifth,research programs should organize appropri-ately.

Wed like to have context-adaptable build-ing blocks, as Ive previously suggested. Wedlike to have some knowledge that we couldmove from one kind of task and function toanother. We would build up a repository ofthese useful functions. Wed have means andarchitectures for composing these and pro-ducing adapted, situated systems that could

do multiple tasks. Of course, underpinningall this is some ability for these componentsto accord with their contextual assumptionor be semantically interoperable.

How are we going to get there? Well, wehave to limit the space, and Ive recommend-ed that we pick a couple of domains; stay inthese domains for a long time; and acrossthese domains, pursue multiple tasks. Whatwere aiming at is the ability to adapt andtransfer our results to new situations.

In the long term, AI is essential. I would notwant to be part of any nation that didnt leadin this area. It will, in fact, have dramaticbenefits, for economic reasons and for nation-al security. But AI is not just a set of little, nar-row applications; this wont get us there. Wemust find a way to integrate across tasks. Wealso have to recognize that for the foreseeablefuture, almost all AI will be in support ofhuman organizations. And cooperation

Articles

SUMMER 1997 111

The New Agenda

Needs an Improved Paradigm

Domain

Mul t i -Task

System

Domain K

Funct ion K

Task K

System K

Domain Doma in

Mul t i -Task

System

Mul t i -Task

System

Integrating Architecture

Figure 6. Futu re Mul ti faceted Intell igent Appli cations W il l Need to Combi ne Several D if ferent Types of Modular and Reusable

Know ledge to Be Abl e to Perform Mul ti ple Tasks in Mul ti ple Contexts.



14/16

to have these systems delegate, accept tasks,and exhibit cooperation. Its essential forthese systems to have a model of their envi-ronment so they can adapt to it. And systemsthat adapt their approach to the resourcesavailable and the time available would also behallmarks of this new approach.

Some of the milestones that Im lookingfor include standardized, reusable representa-tions, which would be domain independentand adaptable to multiple domains; reusable

models for multiple tasks and domains;reusable knowledge bases; some multiple-function systems, or systems that do multipletasks; and some ways to adapt a knowledgebase to a new environment or a changedenvironment rapidly. Finally, I would aspireto have associate systems where we demon-strate that we actually can communicate andtransfer enough problem state to rely on thecorrect interpretation and value-added coop-eration.

I recently reviewed the work that DARPA isdoing and sponsoring in AI. I refer people totheir web site (see yorktown.dc.isx.com/iso/planning/ index.html). It has a number ofprojects that I think are exemplary, and Imasking people to compliment it and encour-age it to go on. One in particular that hasntquite started yet is this joint forces air compo-nent commander (JFACC) AFTER NEXT. Its asystem to support planning, situation assess-ment, real-time reasoning, and multiple func-tions and multiple tasks. This kind of under-

between humans and machines is a differentkind of problem than the field has traditional-ly addressed. Humans are pretty good withambiguity, scruffiness; they are the only oneswho are trusted with values and key decisions;and it would be great if they could delegate afair amount of work. However, today, this isextremely rare. Most AI is about solving prob-lems, which, if theyre successfully done,would allow us to take the corresponding taskoff the typical human agenda. It would be a

new milestone in computer science: systemsthat simplify busy humans lives.

One of the problems is that when you gointo rich-context, multiple-task operatingenvironments, we dont have the time toexplain to our machines all the implicit andexplicit values of organizations that are atwork, an understanding of which makes thedifference between good decisions and poordecisions. I also do not think its going to bea fruitful area for us to invest in.

How would we know we were makingsome progress on this line? See table 5. Idlike to have some systems do more than onetask, and we have a few early examples. Idlike to have some systems that were applica-ble to new domains. Im not aware of anytoday. Today, reapplication technology is lim-ited to tool kits that people use to build mul-tiple systems.

Wed like to have some systems that usethe same knowledge for multiple functions.Again, Im not aware of any. It would be nice

Tabl e 5. Element s of a Proposed New Strat egy and Approach to AI Research.

Articles

112 AI MAGAZINE

Proposed Measures of Progress Som e Potential Key Milestones

Systems do more than one task.

Systems are reapplicable to new domains.

Knowledge adaptation is simplified by

powerful tools.

Systems use the same knowledge for multiple

functions.

The knowledge is economically acquired,

represented, and maintained.

Systems can accept, delegate, and share tasks with

others.

Systems explicitly model their environment and

adapt to variations in it.

Systems can adapt their approach when

constrained by time or resources.

Standardized, reusable representations

Reusable models and knowledge bases

Multifunction knowledge systems

Multitask knowledge systems

Rapidly adaptable and adapting knowledge bases

Associate systems that can

Quickly learn their masters values

Correctly model their masters context and

state

Intelligently execute a variety of delegated

tasks

Extend easily to new tasks



15/16

taking will drive research and technology inthe directions recommended here, towardreusable and component knowledge, taskmodules, human-machine cooperation, andmultiple domains of applicability.

This brings me to the conclusion. I thinkthat AI has accomplished truly amazing resultsin less than four decades. However, were farfrom meeting the expectations that peoplehave for us. I think that the next decades pre-sent us with new challenges. We dont reallyhave the technological foundation to meetthese today. Finally, I think that when onelooks at the areas where weve made majorprogress, you can see that continuous andincremental progress actually is both possibleand probably sufficient. From my point ofview, it matters little if the grand enterprisetakes a few decades more, so long as we achieveand demonstrate incremental progress.

Note

1. As of Sunday 11 May 1997, Herbert Simon's pre-diction of an AI world chess champion was ful-

filled. Although he had predicted the achievement

would take only 10 years and it actually took near-

ly 40, the significance of the result cannot be over-

looked. At this point, it is reasonable to assume

that we can develop world-beating AI capabilities

for any well-defined objective, if only we will stay

the course, commit appropriate resources, and

hunger to succeed! We cannot do everything, but

the field's potential is powerful enough to do prac-

tically anything we choose.

Bibliography

Boden, M. 1986. Arti ficial I ntell igence and N atural

Man. Cambridge, Mass.: MIT Press.Hayes-Roth, B. 1991a. An Integrated Architecture

for Intelligent Agents. SIGART Bull eti n2:7981.

Hayes-Roth, B. 1991b. Making Intell igent Systems

Adaptive. In Archi tectures for I ntell igence, ed. K. Van-

Lehn, 301321. Hillsdale, N.J.: Lawrence Erlbaum.

Hayes-Roth, F. 1995. The JTF ATD Architecture for

Agile, Mobile, Col laborative Crisis Response: A Case

Study in Architecture-Driven Development. In Pro-

ceedings of the First International Workshop on

Software Architecture, ed. D. Garlan, 116126.

Pittsburgh, Pa.: Carnegie-Mellon University.

Hayes-Roth, F. 1994. Architecture-Based Acquisition

and Development of Software: Guidelines and Rec-

ommendations from the ARPA Domain-SpecificSoftware Architecture (DSSA) Program, DSSA-94-01,

Teknowledge Federal Systems, Palo Alto, California.

Hayes-Roth, F. 1993. The Evolution of Commercial

AI Tools: The First Decade. International Journal of

Art ificial Int ell igence Tools2(1): 115.

Hayes-Roth, F., and Jacobstein, N. 1994. The State

of Knowledge-Based Systems. Commun ications of the

ACM37:2739.

Hayes-Roth, F., and Schill, J. 1995. Common Strate-

gy Promises Faster Implementation of Computing

Advances. SIGNAL (October).

Smith, D.; Parra, E.; Westfold, S. 1996. Synthesis of

Planning and Scheduling Software. In Advanced

Plann in g Technology, ed. A. Tate. Menlo Park, Cali f.:

AAAI Press.

Rosenblatt, J., and Thorpe, C. 1995. CombiningMultiple Goals in a Behavior-Based Architecture.

Presented at the 1995 International Conference on

Intelligent Robots and Systems (IROS), 79 August,Pittsburgh, Pennsylvania.

Frederick Hayes-Roth is chair-

man and chief executive officer of

Teknowledge, wh ere he hasworked for 15 years since co-

founding the company. Currently,

he spends most of his time aschief architect for the Defense

Advanced Research Projects Agen-

cy (DARPA) Joint Task ForceAdvanced Technology Demonstration and several

other programs and projects related to this. He has

focused on the architecture and implementation of

distributed intelligent systems for almost twodecades. His work along these lines includes a series

of seminal and successful efforts, such as the

HEARSAY-II speech understanding system and itsblackboard architecture, opportunistic planning

and metacontrol, distributed module-oriented pro-

gramming supported by the ABE development envi-ronment, the real-time architecture for the

PILOTSASSOCIATE, the DICAM architecture for distribut-

ed intelligent control, and the DARPA ISO referencearchitecture.

Prior to cofounding Teknowledge in 1981, he

held faculty appointments at the MassachusettsInstitute of Technology, Stanford University, and

Carnegie-M ellon Un iversity. He directed the

research program in information-processing sys-tems at the Rand Corporation. He was a principal

designer of HEARSAY-II , one of the first 1000-word

continuous speech-understanding systems; the

ROSIE system for programming knowledge systems;

and many knowledge systems for government deci-

sion making. He edited Pattern-Directed Inference

Systems, a reference on rule-based programming,and Buil din g Expert Systems, a text on knowledge

engineering. In addition, he has published numer-

ous articles in such journals as Spectrum , Compu ter,

Comm uni cat ions of th e ACM , Comput i ng Surveys,

IEEE Transacti ons, Art ificial Int ell igence, Cognit ive Sci-

ence, AFCEA Signal , and Systems & Softwa re.

He has served on the editorial boards of the

IEEE Spectrum , IEEE Transacti ons on Know ledge andData Engineerin g, and Expert Systems: the Int ernat ion-al Journal of Knowl edge Engineerin g. He received his

M.S. in computer and communications sciences

and his Ph.D. in mathematical psychology fromthe University of Michigan. He is a fellow of the

American Association for Artificial Intelligence and

a member of DARPAs ISAT review board and haspublished more than 100 technical papers.

Articles

SUMMER 1997 113



16/16

This Is a Pub lication of

The Am erican Association for Artificial Intelligence

This electronic document has been retrieved from theAmerican Association for Artificial Intelligence

445 Burgess DriveMenlo Park, California 94025

(415) 328-3123(415) [email protected]

http://www.aaai.org

(For membershi p inf ormat ion,

consult our web page)

The material herein is copyrighted material. It may not bereproduced in any form by any electronic or

mechanical means (including photocopying, recording,or information storage and retrieval) without permission

in writing from AAAI.
http://www.aaai.org/home.htmlhttp://www.aaai.org/home.html

[Hayes-1997] IA What Works and What Doesn’t

Documents

Transcript of [Hayes-1997] IA What Works and What Doesn’t