Enabling Natural Interaction

Randall Davis, Howard ShrobeAaron Adler, Christine Alvarado, Mark Foltz,

Tracy Hammond, Mike Oltmans, Metin Sezgin,Olga Veselova

MIT Artificial Intelligence Lab

Randall Davis

Consider This Device...

Randall Davis

Natural Interaction?

• What if we had to deal with one another as we deal with our software?

Randall Davis

Let’s Talk

Randall Davis

Our Vision

Randall Davis

Our Model

Randall Davis

Rationale Capture

• Rationale capture doesn’t happen.– Because it’s too much trouble

• Rationale capture tools don’t work– Because they’re too much trouble

• The key: it needs to be less trouble than it’s worth.

• The key to less trouble: break down the barriers

Randall Davis

Sketch Understanding

Randall Davis

Why Sketching?

• Informal

• Succinct

• Natural

• High in information content

• Preserve topological and spatial properties

Randall Davis

Sketch Understanding

• Requires intelligence– at the signal level: early processing– at the interpretation level: handling ambiguity– at the engineering level: intended behavior

Randall Davis

Other Research

• Additional domain: software diagrams

• Multi-modal interaction

• Describing and learning new icons

• Generating efficient recognizers

Randall Davis

Sketching Software

Randall Davis

Sketching Software

Randall Davis

Sketching Software

Randall Davis

Sketching Software

Randall Davis

Sketching Software//Source file: c:\\video\\Deck.javapublic class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** * @roseuid 3C21348C0257 */ public Deck() { }}

//Source file: c:\\video\\Dealable.javapublic interface Dealable {}

//Source file: c:\\video\\Game.javapublic class Game { public Graphics theGraphics; public Deck theDeck; public Player thePlayer; /** * @roseuid 3C21348C02E3 */ public Game() { }}

Randall Davis

Multi-Modal Interaction“Three identical, equally spaced pendulums…”

• Graphical vs verbal communication: What’s best said, what’s best sketched?

Randall Davis

New Icons

• A shape description language

• A drawing sequence description language

Define AndGateline L1 L2 L3arc Asemi-circle A1orientation(A1, 180)vertical L3parallel L1 L2same-horiz-position L1 L2connected A.p1 L3.p1connected A.p2 L3.p2 meets L1.p2 L3meets L2.p2 L3

Randall Davis

Sketch Understanding Architecture

Mechanical

Engineering

devices

simple compositions

primitive objects

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical

Engineering

devices

simple compositions

primitive objects springs, joints

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical

Engineering

devices

simple compositions ratchet, latch

primitive objects springs, joints

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical

Engineering

devices mechanisms

simple compositions ratchet, latch

primitive objects springs, joints

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical Software

Engineering Engineering

devices mechanisms

simple compositions ratchet, latch

primitive objects springs, joints objects, methods

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical Software

Engineering Engineering

devices mechanisms

simple compositions ratchet, latch hierarchies

primitive objects springs, joints objects, methods

geometry rectangles, curves

Randall Davis

Sketch Understanding Architecture

Mechanical Software

Engineering Engineering

devices mechanisms modules

simple compositions ratchet, latch hierarchies

primitive objects springs, joints objects, methods

geometry rectangles, curves

Randall Davis

Design Rationale Capture

• Make it less trouble than it’s worth– Break down the barriers: eliminate

interfaces– Natural interaction is enabled by

intelligence• About sketching, gesturing• About the domain• About the task