Interaction Techniques for

Ambiguity Resolution in Recognition-based

Interfaces

Jennifer MankoffCoC & GVU Center

Georgia Tech

Acknowledgements

Gregory Abowd & Scott Hudson FCE Group & GVU NSF

Outline

Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusion & Future Work

Ambiguity

am·big·u·ous1 a : doubtful or uncertain especially from obscurity or indistinctness <eyes of an ambiguous color> b : INEXPLICABLE2 : capable of being understood in two or more possible senses or ways

An anathema to computers Normal for humans

Where does ambiguity arise? Web search

user doesn’t know correct answer multiple correct answers

Implicit input user not involved with application

Multiple users System states may not agree

Recognition

Focus: Recognition

Recognition is becoming ubiquitous

Recognition is difficult to use

A range of interface problems result

OOPS toolkit helps solve them

Research Methodology

Motivated by real world, non-CS problems

Evaluators Study individual problems/solutions Design space of possible solutions

Builders Facilitate solutions to sets of problems

(Architectural / toolkit solutions) Design space exploration

Outline

Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work

Definitions Mediation

dialogue between user and computer used for resolving ambiguity

Recognizer interprets user input creates ambiguity

Error mistake from user’s perspective represented with ambiguity

SILK (Landay, 1996)

Burlap

Outline

Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work

OOPS Toolkit (CHI’00)

Toolkit-level support for handling ambiguity in recognition Library of mediators Architectural support

Based on subArctic

Library of mediators

Design space based on survey Generic and re-usable Three major classes

Repetition Choice Automatic

Library of mediators

Design space based on survey Generic and re-usable Three major classes

Repetition Choice Automatic

Library of mediators

Design space based on survey Generic and re-usable Three major classes

Repetition Choice Automatic

if (result is “W.”)reject it

elsedo nothing

Architectural Support

INDEPENDENT of any specific toolkit

Separation of mediators, recognizers, and application

Communication by a common internal model (ambiguous hierarchical events)

Maintains ambiguity indefinitely

Three key pieces

Ambiguous hierarchical events Changes to event dispatch Mediation subsystem

down drag up• • • • • •

s

Ambiguous Hierarchical Events

strokestroke

down drag up• • • • • •

s c

med1med2med3med4medn

rec1rec2rec3rec4recn

Event Dispatch

1. A sensed event arrives

eventInput

handler

2. It is dispatched to all recognizers3. It is mediated

Mediation Subsystem

Ambiguity is identified automatically Presence of multiple interactor leaf nodes

Hierarchy is passed to mediators Recognizers, recipients informed of

accept/reject decisions Accept/reject modifies hierarchy

Application selects mediators from library

Outline

Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work

Problem Areas

Errors & Ambiguity rejection errors target ambiguity

Mediation adding alternatives occlusion

Rejection Errors

Problem: The user’s input is completely missed

Rejection Errors

Problem: The user’s input is completely missed

Solution: Allow the user to tell the system

Rejection Errors

Problem: The user’s input is completely missed

Solution: Allow the user to tell the system

Other applications: Substitution errors

Rejection Errors

Problem: The user’s input is completely missed

Solution: Allow the user to tell the system

Other applications: Substitution errors Any spatial

recognition Requires extended

recognizer API

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: clicking

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: Clicking Solution: Give the

user a choice of all of the targets

Target Ambiguity

Problem: There may be multiple targets of a user action

Example: Clicking Solution: Give the

user a choice of all of the targets

Other applications: Any interface

involving mouse press/release

Requires separation of concerns

Works with all interactors

Occlusion

Problem: A mediator may obscure important information

Occlusion

Problem: A mediator may obscure important information

Solution: Move that information into a more visible location

Occlusion

Problem: A mediator may obscure important information

Solution: Move that information into a more visible location

Other applications: Any crowded

interface that uses an n-best list

Requires extensible mediators

Requires separation of concerns

Adding alternatives

Problem: The correct choice isn’t always present

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

Solution: Allow the user to add choices

Adding alternatives

Problem: The correct choice isn’t always present

Example: word-prediction

Solution: Allow the user to add choices

Other applications: Closely related

choices (e.g. URL prediction)

Requires extensible mediators

Benefits from recognizer API

Outline

Motivation Definitions & Illustration Broad Solution: OOPS Specific Solutions Conclusions & Future Work

Conclusions

Resolution of ambiguity in recognition through mediation

General toolkit architecture (CHI 00) flexible, re-usable support for mediation

separates recognition, mediation, and

applications allows exploration of design space

Next Steps

Implicit input Sensed information about

environment Ambiguous output

Ambient displays Brain-computer interface

Ambiguous, limited, error-prone input

500,000 people worldwide

Locked-In Syndrome

Disease, stroke, accident survivors Completely paraylzed Unable to speak Cognitively intact

Brain-computer interface

Problem: locked-in syndrome No alternative modalities available Need for efficient error handling

Challenge: interpret brain signal in as rich a form as possible

A new hope

Brain signals can be intercepted Implanted electrodes (Schwartz, Chapin et

al.) External sensors (Junker, Wolpaw,

Middendorf, Birbaumer et al., Spencer et al.)

Signals can be produced and controlled by imagined movements

Signals can be interpreted by a computer

A Neurotrophic Electrode

Cone electrode invented in 1987 Animal studies showed it to be stable FDA permission given for human implantation in 1996

Project Goals

High level: Recreate movement Restore communication Turn disability into ability

Low level: Intelligent Interpretation

Recreating movement

Haptic feedbackMuscle stimulation Eventually re-connect nerves

Restore Communication

A basic need

From virtual keyboards to word-prediction

Turning disability into ability:

Supporting Creativity

Converting a the brain signal to music

Example

Two possible experiences Trying to play a piano with ones feet Having an artistic voice no one else can

reproduce

Intelligent Interpretation

Signals are difficult to control Daily variability in signal Patient Endurance

Adaption necessary

Intelligent Interpretation

Raw signal-> mouse Logical control Neural gestures

Further Information

http://www.cc.gatech.edu/fce/errata/

jmankoff@cc.gatech.edu

Alternative Forms of Input

Cirrin (UIST 98) Locked-In Syndrome

(Brain-UI; Current) Cerebral Palsy

(Cursor Activity Recognition; Current)

How general is “general”?

Two Toolkits Two complex applications Library of existing mediators Explorations of 4 problem areas

(UIST 00)

Further generalization

Testing

Further generalization

Testing Implicit input

(CT-OOPS; Current)

Further generalization

Testing Implicit input Arbitrary input devices

Further generalization

Testing Implicit input Arbitrary input devices Ambiguity

Related Themes

Input Output

Ambient Displays (Ten Inch Pixels; 1999)

Is subArctic doing the work here?

No, our minimal requirements are common in today’s toolkits: An event-based toolkit An input-handling module that delivers

events to the appropriate places A library of interactors/widgets Access to source code (OOPS is not just

a library!)

Recognizer

Definition: something that interprets user input generally has a domain (of input) and a

range (of output)

Examples: DragonDictate (speech to text) GDT (strokes to gestures)

Problem areas: Support for correction of errors

Error Definition:

a mistaken interpretation (from the user’s perspective)

Examples: substitution rejection insertion

Problem areas: rejection

Mediation

Definition: a dialogue

between the user and application used to determine the correct interpretation

Problem areas Occlusion Wrong choices

Examples:

Ambiguity Definition

A case where there is more than one potentially correct interpretation of the user’s input

Problem areas target ambiguity

Examples target ambiguity segmentation

ambiguity recognition

ambiguity

Future Work

Testing Implicit input Arbitrary input devices Ambiguity

Conclusions

The problem areas are not intractable

Toolkit-level support allows us to explore them

OOPS allows us to build general, re-usable solutions

Other thesis results

Survey of mediation techniques found in existing interfaces to recognition systems

Two implementations of our architecture

Comparing SILK and Burlap

SI LK BURLAP

Modes Four One Combined

Only Draw Mode ThroughoutMediationSeparate Window I ntegrated

Comparing SILK and Burlap

SI LK BURLAP

Modes Four One Combined

Only Draw Mode ThroughoutMediationSeparate Window I ntegrated

Editing Yes No

Storyboarding Yes No

Architectural support

architecture

interactors

interface

application

Input handler

inters

OOPS Architecture

architecture

interactors

interface

application

recognizers

Input handler

meds

application

interface

stroke

downdrag up• • • • • •

sc

stroke

downdrag up• • • • • •

s

Big Picture

1/5 People have disabilities Everyone has temporary

disabilities Computers can help to overcome

disabilities… … or make them worse

Modified Event Dispatch

1. A sensed event arrives 2a. It is dispatched to all

unambiguous components2b. It is dispatched to all recognizers3. It is mediated4. Any accepted leaf nodes are

dispatched to all unambiguous components

Comparison with GUI

event event

interactors

interface

application

Input handler

recognizers meds

application

interface

Input handler

inter

recognizers mediators