TEMPLE: TEMPLate Extension Through Knowledge Acquisition

1USC INFORMATION SCIENCES INSTITUTE EXPECT

TEMPLE:TEMPLate Extension Through

Knowledge Acquisition

Yolanda Gil

Jim Blythe

Information Sciences Institute

University of Southern California

http://www.isi.edu/expect

{gil, blythe}@isi.edu


Acquiring Planning Knowledge Problem: SOF users need to add knowledge to these

planning tools ROEs, commander’s guidance Plan evaluation/critiquing criteria Highlight the information that is important to them Add/extend templates

Approach: provide knowledge acquisition tools to adapt and extend pre-existing planning knowledge Exploit ontologies and background knowledge so users don’t

have to start from scratch KA Scripts guide the user through multiple steps Users manipulate English paraphrases of internal

representations Benefits:

Users can extend the tool’s baseline knowledge for the operation


Prototype for adding plan critiques: Expect’s PSM Tool

Questions formulatedbased on background

knowledge

User adds detailed knowledge through English paraphrases


The next 100 days

Allow users to specify and customise “sentinels” that check for new information and alert planners when needed. Our tools generate Java.

Extend ontologies and background knowledge to handle SOF domain.

Integrate with one of the jumpstart applications, probably the travel planning tool, using InterAcT.


Backup slides

Description of approach, tools and experiment from HPKB project.


Key Technologies

Guiding users through knowledge acquisition scripts [Tallis and Gil 99] that capture typical dialogues that users follow to enter new knowledge step by step

Exploiting domain-independent background knowledge about plan evaluation and critiquing [Blythe & Gil 99] that use background knowledge about plan evaluation and critiquing to guide the dialog

An English-based editor [Blythe & Ramachandran 99] that lets the user add or modify internal knowledge by manipulating its English paraphrase, without having to see or understand the internal formal representation


Architecture of TEMPLETemple

(Server)ConstraintAcquisition UI(Client)

Constraintwizard

Constraintviewer

AcquisitionScripts

Background knowledge

Ontologies¤ constraint types¤ actions and plans¤ proactive

suggestions

Method base

Compiler

Executableconstraints

Templatelibrary

Active Templates Toolkit

SOF

Englisheditor

NaturalLanguageGenerator

Domainknowledge

DomainconstraintsDomainmethods

Domainmodelsandtemplates


Evaluation and Critiquing KnowledgePlan ontology(PLANET)

Ontology ofcritiques

Submethods for checking plan resources

Submethods for checkingplan structure

Reused knowledge(ontologies

and methods)

Domain-specificcritiques

Domain-specific submethods

Domain-specific plan critiquing and evaluation system

Domain-specific knowledge

Ontology ofresources


An Ontology of Plan Evaluation Criteria

Captures general knowledge of how to evaluate plans with respect to standard norms of plan development

Ill-formeddescription

Statementcritique

Linkcritique

Structurecritique

Complete statement

Correct statement

Clear statementako

ako

ako

Correct linkako Does the purpose of supporting effort X

support the main effort?isa

Does <unit> have sufficient combatpower to accomplish its mission?

isa

Complete planako

...

...isa Does the COA include a statement

of the reserve forces?


KA Scripts

Helps the user add new critiques by using a background theory of plan evaluation and critiquing.

KA Scripts guide the dialog with the user about the new critique (wizard-type interaction).

The tool creates some of the needed methods for the critiques, helps the user to create new ones (by suggesting initial templates), and ensures consistency with existing knowledge.


English-Based Editor

Generates automatically English paraphrases of

problem-solving fragments, and presents alternative

text to replace parts of

the paraphrase based

on the ontologies and

background knowledge

NL description of method

Alternatives forselected text fragment


First experiment: An ablation test on the PSM-Based KA Scripts

Hypothesis: the PSM-Based KA Scripts significantly reduce the expertise and the effort required to add a new critique to the knowledge base.

KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check)

Knowledge (and tool) ablation experiment: Two tasks done using PSM-Based KA Scripts, two tasks done without

Subjects: four Army officers, previously trained on EXPECT’s language for a day


Sample Tasks Given to Subjects

Simple critique: Add a critique to check if the COA has a security statement.

Complex critique: Add a critique to check if each

task in the COA has sufficient force ratio. To compute force ratio, divide the sum of

combat powers of the Blue units assigned to the task by the sum of combat powers of the Red units acted on by the task.

(Two other comparable tasks were also used)


Quantitative results: what users could do

Users could complete more tasks using the PSM-based KA scripts

0

5

10

15

20

25

30

35

easiertask 1

easiertask 2

morecomplex

task 1

morecomplex

task 2

All

with PSMToolablated version

LEGEND: indicates total tasks


Quantitative results: speed improvements

Time reduction using the PSM-based KA Scripts

02468

101214161820

Completenesscritique

(simpler)

Resourcecheck critique

(more complex)

ablatedversionwith scripts

Time inminutes


Axiom acquisition rates:Experiment with PSM-Based KA Scripts

Adding small amounts of new knowledge

Adding larger amountsof new knowledge

with PSM-BasedKA Scripts

2.12 ax/min

1.26 ax/min

with ablated version

1.1 ax/min

N/A (users were notable to do tasks)


Summary

Using the PSM-Based KA Scripts significantly reduced the time taken to add a critique

Using the PSM-Based KA Scripts, all four subjects successfully added simple critiques to the EXPECT critiquer; three of them successfully added more complex critiques.

Without the PSM-Based KA Scripts, three out of four subjects successfully added simple critiques and two added more complex critiques.

Comments on the tool usability were positive in all cases.


Second experiment with PSM-Based KA Scripts and English-Based Editor

Hypothesis: the combination of the PSM-Based KA Scripts and English-based editor allows a user with very little training to add new critiques.

Single subject usability test: A subject was briefed in COA critiquer and the KA interface (but not about EXPECT) for 20min and asked to add two critiques using the tool

KA tasks: add two new critiques to the EXPECT COA critiquer (a completeness check and a resource check), same used in the previous experiment

Subject: an Army officer with no EXPECT training


Results

The subject was able to add two new critiques of both low and medium complexity.

The time taken was comparable to that for the other four subjects that had previous training in Expect:

02468

101214161820

Completenesscritique

(simpler)

Resourcecheck critique

(more complex)

new user

Average ofother usersTime in

minutes


EXPECT: A User-Centered Framework for Developing KBSs

Method instantiator

Method instantiator

Knowledge Base

Domain ontologies

and factual knowledge

Problemsolving methods

Domaindependent

KBS

KBS compiler

KBS compiler

Knowledge-BasedSystem

InterdependencyModel (IM)

EXPECT Ontologies and Method libraries

Plans(PLANET)

Evaluations and Critiques

Evaluation PSMs

Resources(OZONE)

COAontologiesCYC/Sensus

Upper

Instrumentation

KA Strategies

KA toolsEMeD

KA Scripts

PSMTool


EXPECT: A User Centered Approach for Knowledge-Based Planning ToolsKnowledge acquisition technology that can guide users to

specify planning knowledge and develop planning tools Expressive representations

– Loom/Powerloom KR&R– EXPECT’s language to represent problem solving knowledge

Powerful reasoners– Loom/Powerloom pattern classifier & reasoners– abstract problem solving through partial evaluation

ex: how to move <a set of units> from a <location> to another <location>

Explicit models of planning knowledge and plan reasoners: – PLANET ontology of plans, OZONE resource ontology– plan evaluation and planning methods

Expectation-based knowledge acquisition tools– Derive interdependencies between individual knowledge fragments– KA Scripts to guide users in completing complex modifications

TEMPLE: TEMPLate Extension Through Knowledge Acquisition

Documents

Transcript of TEMPLE: TEMPLate Extension Through Knowledge Acquisition