Spoken Dialogue Systems and the Spoken Dialogue Systems and the GALAXY ArchitectureGALAXY Architecture

29 October 200029 October 2000

Advanced Technology Laboratories

1 Federal Street • A&E Building 2W • Camden, New Jersey 08102

Jerry FrankeJerry FrankeSenior Member, Engineering StaffSenior Member, Engineering Staffjfranke@atl.lmco.com 856.338.3341jfranke@atl.lmco.com 856.338.3341

September 99

Talk OutlineTalk Outline

• Spoken Language DevelopmentSpoken Language Development

• GALAXY II SystemGALAXY II System

• SUMMIT (speech recognition)SUMMIT (speech recognition)

• TINA (natural language parsing)TINA (natural language parsing)

• GENESIS (natural language generation)GENESIS (natural language generation)

• Turn ManagementTurn Management

• GALAXY II DomainsGALAXY II Domains

• Overview/DemonstrationOverview/Demonstration

September 99

Spoken Language DevelopmentSpoken Language Development

• UniversitiesUniversities

– MIT, CMU, ColoradoMIT, CMU, Colorado

• Basic research labsBasic research labs

– ATT, SRIATT, SRI

• Software developersSoftware developers

– Nuance, SpeechWorksNuance, SpeechWorks

• Domain developersDomain developers

– Lockheed Martin - ATLLockheed Martin - ATL

September 99

GALAXY II SystemGALAXY II System

• Developed by MIT Spoken Language Systems groupDeveloped by MIT Spoken Language Systems group

• Multiple servers performing parts of the dialogue processMultiple servers performing parts of the dialogue process

SpeechRecognition

LanguageUnderstanding

TurnManagement

LanguageGeneration

SpeechSynthesis

Sound

Sound

SUMMIT TINA GENESIS

September 99

SUMMIT (speech recognition)SUMMIT (speech recognition)

• Three elements: vocabulary, language models, acoustic modelsThree elements: vocabulary, language models, acoustic models

• Pause words stripped outPause words stripped out

• AM: segment-based models and boundary-based diphone AM: segment-based models and boundary-based diphone modelsmodels

• LM: Forward Viterbi search with a class bigram model, followed LM: Forward Viterbi search with a class bigram model, followed by a backward A* search with a class trigram modelby a backward A* search with a class trigram model

• Produces N-best list or word graph of possible utterancesProduces N-best list or word graph of possible utterances

• Models trained on domain corpusModels trained on domain corpus

• Models achieve speaker-independenceModels achieve speaker-independence

September 99

TINA (natural language parsing)TINA (natural language parsing)

• Selects from N-best list depending on grammatical parseSelects from N-best list depending on grammatical parse

• Grammars reflect both syntactic and semantic structureGrammars reflect both syntactic and semantic structure

• Result is a semantic frameResult is a semantic frame

• Example:Example:

““Where is the library in Swain Hall?”Where is the library in Swain Hall?”

Clause: LOCATEClause: LOCATE

Topic: PUBLIC-BUILDINGTopic: PUBLIC-BUILDING

Quantifier: DEFQuantifier: DEF

Name: libraryName: library

Predicate: INPredicate: IN

Topic: HALLTopic: HALL

Name: SwainName: Swain

September 99

GENESIS (natural language generation)GENESIS (natural language generation)

• Processes semantic framesProcesses semantic frames

• Embeds semantic frame components into context-dependent Embeds semantic frame components into context-dependent message templatesmessage templates

• Two types of output:Two types of output:

– natural language messagesnatural language messages– messages are sent to some speech synthesis modulemessages are sent to some speech synthesis module

– possibility of output in multiple languagespossibility of output in multiple languages

– keyword-value pairskeyword-value pairs– useful structure for the turn management backenduseful structure for the turn management backend

• Can be used to map between (translate) languagesCan be used to map between (translate) languages

September 99

Turn ManagementTurn Management

• Manages the system’s part of the dialogueManages the system’s part of the dialogue

• Fuses current utterance with dialogue history for full contextFuses current utterance with dialogue history for full context

• Five main tasks:Five main tasks:

– Answer user’s requests (information retrieval)Answer user’s requests (information retrieval)

– Initiate sub-dialogues to clarify the user’s requestInitiate sub-dialogues to clarify the user’s request

– Track progress through the dialogueTrack progress through the dialogue

– Control response to the userControl response to the user

– Provide assistance in using the system when neededProvide assistance in using the system when needed

September 99

GALAXY II DomainsGALAXY II Domains

From MIT:From MIT:

– Jupiter - weather forecastsJupiter - weather forecasts

– Pegasus - airline schedulingPegasus - airline scheduling

– Voyager - Cambridge, Massachusetts city guideVoyager - Cambridge, Massachusetts city guide

– Dinex - Boston restaurant guideDinex - Boston restaurant guide

– Wheels - automobile classified adsWheels - automobile classified ads

– Mercury - airline flight bookingMercury - airline flight booking

From Lockheed Martin - ATL:From Lockheed Martin - ATL:

– DARPA Communicator - airline flight, hotel, car rental bookingDARPA Communicator - airline flight, hotel, car rental booking

– DARPA LCS-Marine/Marine Small Unit Logistics - supply DARPA LCS-Marine/Marine Small Unit Logistics - supply requestrequest

– DARPA LCS-Army - data collection during equipment testsDARPA LCS-Army - data collection during equipment tests

September 99

Overview/DemonstrationOverview/Demonstration

• Booking airline flights (round trip and one way)Booking airline flights (round trip and one way)

• Hotel, car rentalHotel, car rental

• Uses user profile to fill in some information about reservationUses user profile to fill in some information about reservation

• Uses some real-world knowledgeUses some real-world knowledge

• Information retrieval via mobile agentsInformation retrieval via mobile agents

September 99

Try It YourselfTry It Yourself

• CMU 1-412-268-1084CMU 1-412-268-1084

• MIT 1-877-527-8255MIT 1-877-527-8255

• Colorado http://communicator.colorado.edu/Colorado http://communicator.colorado.edu/