Event Templates for Improved Narrative Understanding in Question Answering Systems

Maureen Caudill

SAIC Enterprise Information Systems Division

10260 Campus Point Drive, MS C-2 San Diego, CA 92121

and

Barbara Starr

SAIC Enterprise Information Systems Division

10260 Campus Point Drive, MS C-2 San Diego, CA 92121

ABSTRACT

Understanding narrative text is more than simple information extraction on a sentence-by-sentence basis. To comprehend the true meaning of a narrative requires determining the connections between the sentences and the effect of one event on other events. This story understanding process can be greatly enhanced by the use of event descriptor templates that begin with the basic journalistic questions of who, what, when, where, why, and how but that go beyond these simple basics to address more complex relationships: role playing, context, impact, causality, and interests. Previously, representing story narratives as knowledge representations has required intensive manual effort on the part of trained knowledge engineers to correctly encode the contents of stories into a knowledge base (KB). For large volumes of text, this becomes impractical, limiting the usefulness of KB-based systems in question-answering. This paper describes a means of automating the narrative representation process by using event descriptor templates to elicit critical narrative information to be encoded in a knowledge based system. Keywords: Automating Knowledge Generation, , Knowledge Engineering Bottleneck, Knowledge-Based System Development, Question-Answering Systems, Event extraction, Event understanding, Automatic Event Extraction.

1. BACKGROUND A question-understanding system that can answer questions about historical or current events must have an

understanding of those events to do more than respond to simple look-up questions. If the system is capable of answering complex queries that require reasoning about the situation, its comprehension of events in the real world must substantially exceed that of a simple information extraction system. Information extraction systems typically deal with narrative texts on a sentence-by-sentence basis. That is, they rarely are capable of understanding the relationship between sentences. This limited knowledge makes them into systems that do little more than key word matches when trying to answer questions. In a query about houses, a typical information extraction system looks for the word “house” in sentences. A slightly more sophisticated version might also include synonym searches, and include searches for “apartment,” “condominium,” “condo,” “dwelling,” “abode,” etc. However, none of these techniques can identify sentences that refer to houses indirectly such as the one below: I lived on Main Street when I was a child, in a huge old barn. It had twelve rooms, five fireplaces, and many odd nooks and corners where a child could disappear for hours at a time. In this example, the word “barn” likely would not be identified as a metaphor for “house.” Even if that relationship were identified, the second sentence, referring to the dwelling place only as “It,” would certainly be missed by all but a very few current systems. Thus, if the question is “How many rooms were in the

Maureen Caudill� 4/7/02 9:03 AMDeleted:

house where you lived as a child?” an information extraction system could not answer it, even though the information requested is explicitly included in the sentence. For information that is not explicit, but available only through inferencing, typical information extraction systems have no hope of generating an answer. In knowledge-based question-answering systems, however, the details of the house could be elicited during the knowledge representation process. In previous development of knowledge-based systems, the process of producing these knowledge representations from the content of a narrative has been entirely manual. Trained knowledge engineers and/or experts in a particular domain have been required to manually write the knowledge representations that reflect the contents of documents. This is sometimes called the “knowledge engineering bottleneck.” In the current paper, we describe a method of resolving both these problems. The use of event descriptor templates as models for representing narrative text provides both completeness of coverage and a method of automating the knowledge extraction and representation processes. The following sections describe, first, the types of templates that we have found useful in describing narratives, and then a system design that automatically fills in these event templates from large volumes of text documents without requiring knowledge engineers or domain experts to generate the knowledge representations.

2. BASIC JOURNALISTIC RELATIONS Much of the information needed to correctly respond to user queries is found in the contents of story narratives. These text items range from historical event summaries to news stories. Consequently, narrative text constitutes an important set of data items to be able to automatically process and convert into knowledge representations. Story narratives can be encoded in many ways, going back to the concept of scripts, in which a standardized set of roles and activities occur in a known general environment. The classic example of scripts is eating at a restaurant. There, a person who understands the basic “restaurant script” knows how to interpret such activities as being shown to a table and handed a menu, ordering from a waiter or waitress, having the meal delivered, receiving a check, and paying for the meal. By encoding this and other stereotypical scenarios in a standard knowledge system, then comparing a textual narrative of a specific “eating at a restaurant” incident to the restaurant script, a knowledge-based system can respond

to many complex, non-obvious questions about that event. The concept of prefabricated understanding of specific types of events can, however, be extended to more general event types using an event descriptor template. The initial version of this template first seeks to answer the basic journalistic queries of who, what, when, where, how, and why. When viewing the output from a basic information extraction system, some of these journalistic queries appear to be quite simple to determine. For example, the question of what happened is frequently identified by the verb in the sentence (or in the clause). However, a single sentence can encode more than one event, and multiple sentences can encode the same event. Thus, any automated system to perform story understanding must be able to view all sentences in the story and determine which portions of each refer to which specific events. For example, consider the sentence: [Manila, Philippines] About 36 US Special Forces troops started a month of anti-terrorism training with their Filipino counterparts at a northern army base Tuesday, a Philippine army officer said. (taken from an article on MSNBC.com) The obvious verb here is “started,” as in, the troops started anti-terrorism training. However, the actual main verb of the sentence is “said.” The verb “started” is merely describing what was “said.” Sentences such as this are very common in real-world story understanding. In such cases, both verbs identify events, one of which is an act of starting something (i.e., “starting to train”) and the other is an act of communicating (i.e., “said”). This sentence thus encodes two separate events, the starting event and the saying event. This sentence can thus begin to be encoded by generating the simplest knowledge relations: (defobject Event-E1 (instance-of Event-E1 saying)) (defobject Event-E2 (instance-of Event-E2 starting)) Thus, the verb(s) defines the type of action that corresponds to that event. The next simple journalistic question is who performed the event. Typically, this is identified as the subject of the verb for each verb in the sentence. (Care must, of course, be taken for passive voice and similar non-typical sentence constructs.) Thus, a parse of the sample sentence above can identify the agents involved by identifying the subjects of each of the two verbs. This

enhances the definitions of the two events by adding more relations: (defobject Event-E1 (instance-of Event-E1 saying) (performed-by Event-E1 Philippine-army-officer)) (defobject Event-E2 (instance-of Event-E2 starting) (performed-by Event-E2 us-special-forces-troops) (performed-by Event-E2 filippino-troops)) To get these encodings, however, additional definitions are required to identify the specific agents, since it is unlikely that any general ontology would have the concepts of Philippine-army-officer or us-special-forces-troops already present. Thus, additional prior definitions are added to the knowledge base: (defobject Philippine-army-officer (instance-of Philippine-army-officer soldier) (citizen-of Philippine-army-officer Philippines)) (defobject us-special-forces-troop (instance-of us-special-forces-troop soldier) (citizen-of us-special-forces-troop united-states)) (defobject us-special-forces-troops (group-of us-special-forces-troops us-special-forces-troop) (count-of us-special-forces-troops 36)) The next journalistic query is where does this event take place. This becomes somewhat more difficult to ascertain in many cases. In the case of the starting event, the location is straightforward, and can be added to the event definition in a simple manner: (defobject Event-E2 (instance-of Event-E2 starting) (performed-by Event-E2 us-special-forces-troops) (performed-by Event-E2 filippino-troops) (location-of Event-E2 northern-army-base)) Obviously, this too spawns a definition of the term northern-army-base: (defobject northern-army-base (instance-of northern-army-base army-outpost) (location-of northern-army-base (location-function (direction-fn north) Philippines))) The location of the saying event is far more obscure, however. In this case, with a newswire article that (presumably) has a dateline attached, the assumption

would be made that the location of the saying event is the location specified in the dateline. Thus, the location for that event would be: (defobject Event-E1 (instance-of Event-E1 saying) (performed-by Event-E1 Philippine-army-officer) (location-of Event-E1 city-of-manila)) Should this location not be in the base ontology, it would of course also have to be defined. The next journalistic question is when did the event take place. This is again easier to determine for the starting event than the saying event: (defobject Event-E2 (instance-of Event-E2 starting) (performed-by Event-E2 us-special-forces-troops) (performed-by Event-E2 filippino-troops) (location-of Event-E2 northern-army-base) (took-place-when Event-E2 Tuesday)) Presumably, a newswire article would also have a date attached to it, so “Tuesday” could be more specifically defined as the most recent Tuesday prior to the date of the article. The question of when the saying event took place is again an inference, not something that is clearly defined; however, it can be presumed to be not later than the date of the newswire article itself: (defobject Event-E1 (instance-of Event-E1 saying) (performed-by Event-E1 Philippine-army-officer) (location-of Event-E1 city-of-manila) (took-place-when Event-E1 before(date-of-article))) The two remaining journalistic questions, how and why are not answered by the current sentence; they remain, for the moment, empty relations. And, in fact, the question of why an event occurs is nearly always a more complex relation to implement.

3. COMPLEX RELATIONS A careful reading of the original sentence shows that while the descriptors developed above are accurate, they are sorely incomplete. For example, the “starting” event does not explain what exactly is being started. Furthermore, there’s no information regarding why this event is happening, what impacts it has on other events, or any of a number of other issues that a human reader can be expected to infer easily from the text. Thus, the journalistic event relations are only the starting point for a far more complex set of relationships.

In the case of the example sentence, the first issue is to identify what exactly is being started. The object of the sentence (either direct object or indirect object, depending on the verb) frequently gives insight into what is being used to perform an action or what is being referred to in an event description. The object of the starting clause is the phrase a month of anti-terrorism training. Thus, there must be a way to encode the concept of what is being started. (defobject Event-E2 (instance-of Event-E2 starting) (performed-by Event-E2 us-special-forces-troops) (performed-by Event-E2 filippino-troops) (location-of Event-E2 northern-army-base) (took-place-when Event-E2 Tuesday) (action-performed-on Event-E2 anti-terrorism-training)) This in turn spawns the definition: (defobject anti-terrorism-training (instance-of anti-terrorism-training training) (opposes anti-terrorism-training terrorist-group) (duration anti-terrorism-training (days 30))) Note that the training definition includes the approximate duration of the training. Notice also that because the prefix “anti” means “against,” the relation “opposes” is used. If there were no such prefix, and the phrase was “terrorism training,” the relation “supports” would be used, on the assumption that training generally supports the activity it concerns. The saying event also must be more carefully defined. What exactly did the Philippine army officer say? Basically, the officer described the training event. Thus, a reasonable encoding of the object of the saying event is: (defobject Event-E1 (instance-of Event-E1 saying) (performed-by Event-E1 Philippine-army-officer) (location-of Event-E1 city-of-manila) (took-place-when Event-E1 before(date-of-article)) (action-performed-on Event-E1 Event-E2)) This single sentence does not yield other relations. But there are many other relations that should be considered before abandoning the analysis of the sentence (and the article). These relations can be grouped into categories as shown below with a few examples for each category. Role Relations

• Is-aggressor • Is-defender • Is-protector

• Is-mediator • Is-peacekeeper • Is-victim • Is-innocent-bystander • Is-financial-backer • Is-betrayer, etc.

Causality Relations • Event-causes-event • Interest-causes-event • Contributing-factor, etc.

Impact Relations • Event-opposes-agent <a specific agent> • Event-supports-agent <a specific agent> • Event-opposes-interest <a specific interest> • Event-supports-interest <a specific interest> • Supports-interests-of <an agent and its

interest> • Opposes-interests-of <an agent and its

interest> • Action-performed-on <an object or an agent> • Action-performed-with <an object>, etc.

The relations that form the event descriptor thus provide a rich encoding of the content of a narrative text description. The benefit of this approach to knowledge representation is two-fold. First, it ensures that each event is scrutinized to derive the maximum possible information from the narrative text. Second, it allows multi-sentence descriptions. For example, suppose the next sentence in the text is: The training will provide experience in fighting bio-terrorist attacks in a jungle environment. This sentence can easily be determined to refer to the same event as the “starting” event. The subject of the sentence is “the training.” Since it immediately follows a sentence that refers to starting a training event, it can be inferred that the training referred to is one and the same training. Thus, it is a simple matter to amend the earlier description of the training to assert: (defobject anti-terrorism-training (instance-of anti-terrorism-training training) (opposes anti-terrorism-training terrorist-group) (duration anti-terrorism-training (days 30)) (location-of anti-terrorism-training jungle) (opposes anti-terrorism-training bio-terrorist-attack)) Filling in such detailed event descriptors amounts to pre-answering a number of queries about the narrative. Any question that asks about these events, even in elliptical terms, will quickly generate the correct answer. This removes a great deal of the processing and inferencing burden from the question-answering system and enables it to provide answers accurately, quickly, and efficiently.

4. AUTOMATING THE PROCESS

The remaining question is how can constructing such event descriptors be automated. In essence, this requires constructing a pre-answering system that infers the answer to obvious questions about a narrative and stores those answers in the knowledge base for rapid retrieval. The approach we have taken to this is a modular one for the sake of easy construction, extension, and maintenance. In effect, each relation or logical set of relations has its own miniature knowledge-based reasoner—a knowledge base partition—which knows only how to inspect a parsed sentence and construct a specific relation from that sentence—if and only if that sentence contains the necessary information to construct the relation. These knowledge-based reasoners thus are kept within a small scale, they can be called or not as appropriate for a given sentence, and they construct an appropriate relation (or small number of similar relations) based on the contents of that sentence. Figure 1 illustrates how the architecture operates in the case of the example sentence. The KB Controller is the coordinator for the event descriptor construction. It does not generate event relations itself, but calls whatever other modules are appropriate in a given sentence to determine the relations. It also deals with cross-sentence descriptors by determining that the current sentence probably refers to an event that is already described; in this case it tells the various KB partitions it calls not to define a new event but rather to modify a specific existing event.

Each of the KB partitions has access to the global ontology and lexicon to identify appropriate concepts and lexical terms and to define new ones as needed. These new terms comprise a mini-ontology that is relevant primarily to the current document; however, as appropriate, new terms can be added to the global lexicon, thus enabling cross-document processing of events. Thus, a sequence of events described in multiple documents is an easy extension of this concept. Furthermore, since each relation can be documented with the specific text that generated that relation, it is similarly easy to identify which document events and mini-ontologies correspond to a particular term. For example, if a query asks about US Special Forces Troops in the Philippines, it is very easy to determine exactly which document(s) have information about these troops, since they are mentioned in those documents that include that ontological term. A separate database (not shown in the figure) keeps track of which ontological terms are defined in which document files and mini-ontologies. Thus, if a query arrives asking about a specific concept (such as the U.S. Special Forces Troops in the Philippines), only those knowledge bases that include that concept are added to the answer reasoner’s knowledge base to serve as the basis for determining the answer. This provides a mechanism for automatic partitioning of very large scale knowledge bases as well as increases the efficiency of the answering system. The modular architecture enables each KB partition to focus on only a single problem rather than trying to

Parsed sentence KB Controller KB Ontology

& Lexicon

Calls KB Engines

as needed Etc.

Instance-of KB partition

Performed-by KB partition

Location-of KB partition

When KB partition

Roles KB partition

Causality KB partition

(defobject Event-E2 (instance-of Event-E2 starting))

(defobject Event-E2 (performed-by Event-E2 us-special-forces-troops)) including definitions for “us-special-forces-troops”

(location-of Event-E2 northern-army-base)) including definition of terms

(took-place-when Event-E2 Tuesday)

Figure 1. Architecture for the Event Descriptor system.

generate the entire event descriptor. This makes the entire system easy to construct, easy to extend, and easy to maintain.

5. CONCLUSIONS This paper has outlined a workable methodology for constructing a narrative understanding tool for text narratives that can automatically comprehend stories and answer questions about those stories. The system is under construction and preliminary results indicate that it is flexible, powerful, and highly extensible.

6. REFERENCES [1] Adam Farquhar, Richard Fikes, and James P. Rice. “A Collaborative Tool for Ontology Construction.,” International Journal of Human Computer Studies, 46:707-727, 1997. [2] Peter D. Karp, Vinay K. Chaudhri, and Suzanne M. Paley. “A Collaborative Environment for Authoring Large Knowledge Bases.” Journal of Intelligent Information Systems, 1998. [3] Paul Cohen, Robert Schrag, Eric Jones, Adam Pease, Albert Lin, Barbara Starr, David Gunning, and Murray Burke. The DARPA High Performance Knowledge Bases Project. AI Magazine, Winter, 1998. pp. 25-49 [4] B. Katz, “From Sentence Processing to Information Access on the World Wide Web,” AAAI Spring Symposium on Natural Language Processing for the World Wide Web, Stanford University, Stanford CA (1997). [5] Gentner, D. and K. Forbus “MAC/FAC: A Model of Similarity-based Retrieval,” Proceedings of the CognitiveScience Society. 1991. [6] Forbus, K. and D. Oblinger “Making SME Greedy and Pragmatic,” Proceedings of the Cognitive Science Society. 1990. [7] V. K. Chaudhri, J. D. Lowrance, M. E. Stickel, J. F. Thomere, and R. J. Waldinger “Ontology construction toolkit,” Technical Note Ontology, AI Center, SRI International, 333 Ravenswood Ave., Menlo Park, CA 94025, 2000. [8] Deborah McGuinness “Description Logics Emerge from Ivory Tower”’ Stanford Knowledge Systems Laboratory Technical Report KSL-01-08 2001. In the Proceedings of the International Workshop on Description Logics. Stanford, CA, August 2001. [9] Deborah McGuinness “Conceptual Modeling for Distributed Ontology Environments.”(Word format)

To appear in Proceedings of the Eighth International Conference on Conceptual Structures Logical,Linguistic, and Computational Issues (ICCS 2000). Darmstadt, Germany. August 14-18, 2000. [10] McIlraith, S. and Amir, E. “Theorem Proving with Structured Theories,” Proceedings of the Seventeenth International Conference on Artificial Intelligence (IJCAI-01). pp. 624 -- 631, August, 2001. [11] Amir, E. and McIlraith, S. “Partition-Based Logical Reasoning for First-Order and Propositional Theories,” Submitted for Publication. This work was supported by the Advanced Research and Development Activity (ARDA) as part of its AQUAINT Program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. Government.