Reactive and Responsive Intelligent Environments

L C S

Reactive and Responsive Intelligent Environments

Kevin Quigleyaire groupMIT AI Lab

Oxygen Alliance Annual Meeting — February 25 – 28, 2003Kevin Quigley — MIT Artificial Intelligence Lab

Reactive and Responsive Environments• We are trying to build pervasive, perceptually enabled human-

centered environments

• Such an environment must respond in reasonable ways to high level requests from its users.– It should be up to the system to figure out a reasonable way to implement

the request, translating goals to plans that meet the users needs and that utilize available resources.

– E.g. I might ask to light the room up; the system responds by opening the drapes.

• Such an environment should react to events in the environment even when there is no explicit user request.– E.g. when I walk into my room in the morning, the system should light up

the room.• Reactions and responses should both be contextually

sensitive.• Both must show human levels of adaptivity


5 Challenges

1. Providing a practical level of knowledge representation that enables groups interactions and grounding in the real world of space and time

2. Providing run-time composable services in a multi-user environment that make optimal use of the currently available resources

3. Recovering from equipment failures, information attacks, misestimates of sensors, etc.

4. Coordinating and fusing information from many sensors and modalities

5. Capitalizing on and recognizing context (task, location, personal style & state)

6. Maintaining security and privacy and trading these off against other goals


Challenge 1: Grounding in Real-World Semantics

• We want to build applications that service many individuals and groups of individuals

• These people will move among many physical spaces

• The devices and resources they use change as time progresses

• The context shifts during interactions• The relevant information base evolves as well.

• The system is required to respond dynamically


Research Agenda: Knowledge Representations

• People– Interests, skills,

responsibilities, organizational role

• Organizations– Members, structure

• Spaces– Location– Subspaces– Devices and resources

• Resources

•Information nodes–Topic area, place in ontology, format

•Services–Methods, parameter bindings, resource requirements

•Agents–Capabilities, society, acting on behalf of whom

•Events–E.g. Person identification, motion into a new region of space, gestures–Qualitative Changes in any of the properties in the KR


Challenge 2: Adaptive Resource Management

• In most systems, applications are written in terms of specific resources – (e.g. The left projector in Michael’s office, or worse yet, a physical

address).

• This is in conflict with– Portability across physical contexts– Changes in equipment availability across time– Multiple applications demanding similar resources– Need to take advantage of new resources– Need to integrate mobile devices as they migrate into a space– Need to link two or more spaces

• What is required is a more abstract approach to resources in which no application needs to be tied to a specific device.


An Example of Service Requests

• When I come into my office in the morning it’s dark– The lights are out– The drapes are closed

• I ask the office to light up the room

• It’s a sunny day, it opens the drapes– If I had asked it to turn on the lights, it wouldn’t have opened the drapes

• It’s a cloudy day, it turns on the lights


Responsive, Goal-Directed Processing

Goals

Service Mapper

Plans

Resource Allocator

ResourcePool

Actions

Service mapping is provided by the resource management component


AbstractService

ServiceControl Parameters

User’sUtility

Function

The Binding of Parameters Has a Value to the User

Resource1,1

Resource1,2

Resource1,j

Each Plan Requires Different Resources

The System Selects the Plan Which Maximizes Net Benefit

User Requests A

Service With Certain

Parameters

ResourceCost

Function

The ResourcesUsed by the MethodHave a Cost

Net Benefit

Each Method Binds the Settings of The Control Parameters in a Different Way

Plan1

Plan2

Plann

Each Service Can BeProvided by Several

Plans

Services are Dynamically Mapped to Plans


Challenge 3: Robustness and Recovery From Failures

• Breakdowns are inevitable– Resources sometimes fail while being used– The system acts on sensor data which has uncertainty

• The system renders services by translating them into plans• A plan-monitor watches over the execution of a plan.• Each plan step accomplishes sub-goals needed by

succeeding steps– Each sub-goal has some way of monitoring whether it has been accomplished– These monitoring steps are also inserted into the plan

• If a sub-goal fails to be accomplished, model-based diagnosis isolates and characterizes the failure

• A recovery is chosen based on the diagnosis– It might be as simple as “try it again”, we had a network glitch– It might be “try it again, but with a different selection of resources”– It might be as complex as “clean up and try a different plan”


I need to ask a question of a systems wizard

Plan 1: • Locate a systems wizard in the E21

• Monitor: check that person is still there• Turn on the selected projector

• Monitor: check that projector turned on• Project the message• Done

• Monitor: check that the person noticed the message

I don’t see light on the screen

I see Sally by the screenProjector-1 must be broken.We’ll try again, but using

Projector-3.

PlanBreakdown

The Plan Monitor Manages Recovery From Failures


Making the System Responsible for Achieving Its Goals

Scope of RecoverySelection of Alternative

Localization & Characterization

DiagnosticService

Repair PlanSelector

ResourceAllocator

Concrete Repair Plan

Resource Plan

alerts

PlanMonitor

RollbackDesigner

Enactment

achieves

requires

Step-A

Step-B

Condition-1

Condition-1

prerequisite

ServiceRequest

Plan-for

A plan is a partially ordered collection of stepsEach step achieves a subgoalSome steps establish pre-requisites for others


Self-AdaptiveGoal Directed Processing

Service Mapper

Goals

Plans

Resource Allocator

Actions

ResourcePool

PlanMonitor

Diagnosis& Recovery


Challenge 4: Context Awareness

• The context should influence how the system behaves:– Task Structure– Location– Emotional State– Personal Style

• Perception can help determine the context• The system should choose its reactions to events based on

the context• Perceptual interpretation should be biased by context

– E.g. a person near the White Board, is likely to start drawing• Estimation of utility should be influenced by context


A Reactive System Responds to Events

If somebody enters the roomThen illuminate the room


Structure of REBA• A Reaction maps an event to a goal• Reactions are grouped into behavioral bundles

– Sets of reactions that are always activated as a unit• The Context is determined by the task within a plan

– As well as location, people present, ...• Each context activates a set of behavioral bundles• Contexts have Sub-Contexts, activities that occur within other

activities – Watching a video within a meeting

• The active reactions of a sub-context override the reactions of the parent context

Sub-context-1

Sub-context-2Context Stack

Reactions

Events

Goal1

Goal2


Context Awareness Conditions Reactions

If somebody enters the roomThen illuminate the room But not if a movie is being watched


Context Sensitivity

Events

Context

ReactiveManager

Service Mapper

Goals

Plans

Resource Allocator

Actions

ResourcePool

PlanMonitor

Diagnosis& Recovery

Reactive Processing is provide by the REBA MetaGlue Component


Challenge 5: Perceptual Integration

• We want to separate the implementation of perceptual tasks from the uses to which perception is put

• Some modules advertise the class of “behavioral events” they are capable of recognizing and signaling– These events are organized into a taxonomy– The same event can be signaled by quite different perceptual modules

(e.g. both face and voice recognition can localize a person).• Other modules register their interest in certain classes of

events– Requests at a higher level in the taxonomy subsume lower level events

• Modules which receive low-level events may register for and collate many different classes of events – They combine these and signal higher-level events

• Modules may request perceptual services when they are uncertain of their conclusions


A Dynamic Event Bus For Perceptual Integration

Visual Tracker

Signal Body Motion

VoiceIdentification

Interested in Body MotionSignal the Location of Individuals

FaceRecognition

Interested in Face LocationI Signal the Location of Individuals

White BoardContext Manager

Interested in the Location of IndividualsSignal People Approaching the Whiteboard

FaceSpotter

Interested in Body MotionI Signal Face Location

• A “Blackboard” System• Publishers & Subscribers Are “Knowledge Sources”• Events Are the Blackboard Data Items

• Highly Distributed• Use of Bayesian Techniques


Blackboards: Using Context

phonemic

intent

acoustic

Syllable

sentence

word

R eck Og N I Z

Rec Og IZE

Recognize

Recognize Speech

SP

Context:Discussion of Oil Tanker Crash

Fragility of Environment

Wreck A Nice Beach

Wreck A NICE

Wreck A NIS

It’s not Hard to Wreck a Nice Beach


Integration of Reactive and Goal Directed Processing

Sensory Systems

Blackboard

Events Context

ReactionsService Mapper

Goals

Plans

Resource Allocator

Actions

ResourcePool

PlanMonitor

Diagnosis& Recovery


Summary

• Responsive processing dynamically maps goals to plans– Plans are selected by balancing the benefit to the user against the cost of

resources– Plan monitoring recovers from plan breakdown– Access control is handled as part of the cost benefit analysis

• Reactive processing dynamically maps event to goals– Events are handled within context– Perceptions are maps to events within context


Reactive and Goal Directed Processing

Sensory Systems

Blackboard

Events Context

ReactionsService Mapper

Goals

Plans

Resource Allocator

Actions

ResourcePool

AccessPolicies

PlanMonitor

Diagnosis& Recovery

with Security

L C S

The Metaglue System

Software agents for intelligent spaces


Computational needs

• Smart Environments– Distributed

Components– Dynamic Changes– Highly Varied

Equipment– New Modes of

Interaction– Frequent Failures– Abundance of

Information

• Metaglue – Communication– Resource

Management– Customization– Multi-modal HCI– Agent Recovery– Persistent Storage


The Metaglue System

• What is it?– A communication infrastructure for building systems of

distributed software agents– A software architecture for creating adaptive applications for

Intelligent Environments

• Why for Intelligent Environments?– dynamic adaptation of the applications based on

* the availability of resources (hardware/software) in the current system/environment

* security controls of the participants* preferences of the participants


Metaglue Software Infrastructure

• What do you get with Metaglue?– Multi-modal Human-Computer Interaction

* Spoken Natural Language* Perceptual (Vision)* Direct manipulation (Graphical)

– Persistent storage– Multiple communication paths

* Remote procedure call (Java RMI)* Publish/Subscribe message passing

– Configurable setting for asserting preferences– Start on demand of Agents– Automatic recovery / handling of direct communication errors– Resource Management and Service mapping


What Are Software Agents?

Our definition:• A software agent is any software object capable of

communication by exposing functionality to other agents running within the network.

Metaglue is a Multi-Agent System where agents perform individually specialized, (usually) simple tasks but connect in a web of intercommunication to cooperate on more complex tasks.


More than Java Objects

• Metaglue Agents can be as complex or simple as necessary– Larger programs can be used in Java thru JNI or other interface

* ViaVoice, ASSIST (a sketching tool)– Most Metaglue Agents are simpler and single-purpose

* Projector display or light control– Complex agents are those that control other simpler agents

* ReBa – Reactive Behavioral System* SPIe – Self-adaptive Plan-based Intelligent Environment

• Metaglue provides to agents– identity and occupation– the entity this agent represents– location and the ability to change locations– intrinsic communication to other agents


What are some of the agents in Metaglue?

• Device Control– lights, drapes, fans, sensors– DVD & CD players, MP3 – video multiplexers, projectors– Cameras, TVs, VCRs, Audio

• Data Organization Systems– Blackboard agents – data flow– START system – NLP & KB– Presence & Location agents– Newswall – visual data organization

and presentation

• Agent Systems Applications– ReBa – interactive behaviors– SPIe and Planlet – plan monitoring – Web info display

• Debugging and Logging– Agent testing, Simulated devices– Log & Catalog Monitors– Notification listeners

• Recreation– Checkers, Boggle, RPG,

HexaCheckers, Crosswords, ELIZA clone


System Organization - Societies

• Clusters of agents that operate on behalf of a real-world entity (space or person) are called Societies

• Societies allow the same agent to exist with different customized attributes.

• Agents can talk to agents in other societies as easily as their own society

• Societies look like agents when viewed from the outside– They exposed functionality to higher level resource management through

Hyperglue


System Organization - Catalog

• The Catalog Agent is the central component which knows about all running agents


System Organization - MetaglueVM

• The Metaglue Virtual Machine (MVM) runs as a base platform for all other agents in the Metaglue system.– It handles all registration of the agent with the Catalog Agent– Provides methods for direct communication (RMI) to other agents on the

current catalog

To Catalog


Calling on Other Agents – agent present

1. A reliesOn call by Agent B will create a chain of events to get the RMI stub for Agent A

2. The MVM takes the call and passes it to the Catalog Agent3. The Catalog checks the internal table of agents to see if there is one

matching the requested description1. If the agent exists, the RMI stub for Agent A is returned


Calling on Other Agents – agent not present

3. The Catalog Agent checks the internal table of agents to see if there is one matching the requested description2. If the agent does not exist, it must be started locally (calling MVM)3. The new Agent A registers its stub4. That stub is then returned to Agent B

L C S

Automatic Agent Recovery

Failure recovery through proxies


Without error handling


Error handling with proxy objects


Statistics on Metaglue

• 10 Tons of fun:– There are over 450 agents that exist within Metaglue– Between 50 and 80 agents are running the intelligent room– You are using more than 10 agents just while running the

X10BasicLightControl* Test it! Use agentland.util.LogMonitor

• Metaglue has been in development since 1998• The system is used in several offices and homes including the

office of the AI lab director, Rodney Brooks• There are 2 full spaces at MIT (a 3rd is coming soon!) and one

space in Australia running Metaglue– Why not get your own?

Reactive and Responsive Intelligent Environments

Documents

Transcript of Reactive and Responsive Intelligent Environments