Henry Lieberman MIT Media Lab Programming by Example Henry Lieberman Media Laboratory Massachusetts...

Henry Lieberman • MIT Media Lab

Programming by Example

Henry LiebermanMedia Laboratory

Massachusetts Institute of Technology

Cambridge, MA, USA

[email protected]

http://www.media.mit.edu/~lieber

PBE Web site: http://www.media.mit.edu/~lieber /PBE/


Your Wish is My Command:Programming by Example

Morgan Kaufmann, 2001


Programming should be like teaching

User plays the role of a teacher

Computer plays the role of a student


Programming by Example

(Also called "Programming by Demonstration")

Agent "watches what you do" in an interactive interface

Records sequence of operations, data

Generalizes program so that you can use an analogous procedure in new examples

"Macros on Steroids"


Programming in the User Interface

The same interface used by end-users can be used for programming

Empowers end-users to do programming

No switching between user and programmer interfaces

Empowers "visual thinkers"


PBE and the "Intelligent Agent" movement

There's been a recent movement toward "intelligent agents" -- proactive, assistive, (possibly anthropomorphic) interfaces

PBE can be used to make agents instructible

Agent personalities:

Butler/Servant/Secretary

Teacher/Student, Master/Apprentice


Issues in Programming by Example

The Generalization Problem (= Data Description)

How do I describe actions and data?

How similar do the actions/data have to be next time?

Feedback

How does the system show the user what it's learned?

Advice

How can the user influence or modify what the system learns?



How should we record the user's actions?

At what granularity should things be recorded?

Can the user interact with the agent during recording? If so, how?

Do we represent the agent directly in the interface?

Do we anthropomorphize the agent?

Can users give advice during recording?

Turning recording on and off



How should the recorded program be represented?

Should the user see the recorded program?

What should it look like?

Should the user interact directly with the recorded program? If so, how?

How to represent examples?

How to represent generalizations?



How should the user edit an already-recorded program?

How do we keep the representations consistent during editing?

Can the user edit previous generalizations?

How can users debug programs?

Can users step/trace/break programs?


Machine learning in Programming by Example

Lots of work in machine learning

Careful: Watch for hidden assumptions

Few examples, learns fast, explainable

•Explanation-based Generalization

•Case Based reasoning

Lots of examples, slow but more robust

•Neural networks, PAC learning, data-mining


Users of the world, unite!

You have nothing to lose but your error messages!

Almost "political" message of empowerment of users

Users should not be at the mercy of software providers

User procedures and data should not be held hostage by software companies


"Poor man's"Programming by Example

Macro recorders in applications, AppleScript

Emacs keyboard macros

Interface builders / Scripting languages

Quickeys / OneClick …


Hooking PBE systems to conventional applications

PBE systems would like to use conventional interfaces as tools rather than re-implement apps

Conventional interfaces aren't set up to be operated by an external program

Issues: Access to application data, sharing the interface, parallelism

Article: Lieberman, IUI 98 "Attaching Interface Agents to Conventional Applications"


Watch What I Do

Allen Cypher, ed. MIT Press 1993


A Small Matter of Programming

Bonnie Nardi, MIT Press, 1994.

End-User Programming

Scripting languages

Application-specific languages

"Gardeners"


Eager - Allen Cypher

Programming by example for Hypercard

Notices patterns in user actions

Anticipation highlighting of predicted actions

Anthropomorphization of learning agent

Example: Collect subject lines from messages


Eager


Mondrian: An Instructible Graphical Editor

Mondrian =

•An object-oriented graphical editor +

•An agent that records user interface actions +

•Learning and generalization heuristics

Arch Near Miss


Mondrian: An Instructible Graphical Editor


Wizard of Oz Experiment

Human simulates agent

What kinds of communication between human and agent are plausible?

What kind of knowledge does the agent need to know to perform the task?


Wizard of Oz experiment task

Reformat a Bibliography

•Lieberman, H., The One True Path to Intelligent Agents, International Conference on Buzzwords and Hype, Tahiti, 1996.

• [Lieberman 96] The One True Path to Intelligent Agents, Int. Conf. on Buzzwords and Hype, Tahiti, (1996).

•Oz, Wizard Of, Pay No Attention to the Man Behind the Curtain, Journal of Smoke and Mirrors, August 1942, Emerald City, Oz.


Wizard of Oz results

Users can make mistakes

Information can be incomplete

Agents can ask the user if they encounter an error situation

Agent could ask the user to supply missing information

Agent could ask the user to edit the example

Skip steps that don't work


Wizard of Oz experiment method

User performs task once

• “Thinking aloud”: goals and methods

Agent (secretly) constructs description of actions

• In terms of simple “implementable” operations

Agent offers help

Agent performs help according to action description

Agent gives user feedback

Agent accepts advice from user


Survey of "Your Wish" systems

Astonishingly wide variety of application domains:

Kids, Games, CAD, GIS, Text editing, Graphical editing, Web browsing, Animation, Databases, etc.

Illustrates variety of design decisions in PBE systems

Different reps for programs, agents, etc.

How much inference, generalization, initiative?


Stagecast Creator - Smith, Cypher

Programming for kids, grid game world

If you see this pattern of squares, replace it with that pattern of squares

Viz rep of rules, editor/stepper

Generalization on object types, introduce variables

Commercially available, success with thousands of users


A Stagecast Creator Rule


A Stagecast Creator Game


Toontalk - Ken Kahn

Programming language is a video game

Most radical programming model

Concurrent logic programming

Abstraction by removing detail

Commercially available, success with kids in schools


Toontalk's Visual Language


Generalizing by Removing Detail


Toontalk Sort program


How much intelligence?- Brad Myers, Rich McDaniel

CMU group one of most active and consistent in PBE

Multitude of systems, "gems" of PBE

Peridot, Lapidary, Jade, Gilt, Tourmaline, C32, Pursuit, Gold, Marquise, Katie, Turquoise, Topaz, Gamut

Experience with range of conservative to radical PBE systems

"Spectrum of Instructibility"

Tradeoff between conveience and control


Pursuit's Visual Language


Selecting Generalizations in Marquise


Internet Scrapbook - Sugiura

Everybody's favorite application area - the Web!

Automates Browse & Cut & Paste applications

How do you describe a Web page selection?

Partial matching, Web design heuristics

Commercially available - preloaded on NEC PCs


Internet Scrapbook


TriAS - Bauer, Meyer, Dengler, Paul

Automating Browse & Copy & Paste, like Scrapbook

Collaborative, mixed-initiative dialog

At any time, agent or user can propose selections, generalizations

What Next? Can I Stop?

Integrates wrapper description language HyQL, InfoBeans dynamic Web sources


Collaborative Wrapper Definition


PBE for GIS: CSprl - Traynor, Williams

Study of how end users use Geographical Information Systems

"Gardeners", "Surrogate Users"

Comic-strip PBE language


A Geographical Info Systems PBE Program


Visual Language for Geographic Information Systems


PBE for Computer Aided Design -Girard, Pierra

True "Industrial Strength" PBE, probably the champ

CAD users are sophisticated in application domain, willing to specify a lot

Provides complete editing and debugging

Key issues: Naming, Ambiguity in spec/solutions, general control structures: conditionals, loops


Example-Based Programming in Computer-Aided Design


Demonstrating "Hidden Features" - McDaniel, Myers

PBE for animated computer games

WYSIWYG - "What you get is all you have"

Demonstrate "hidden features" by reifying them

Guides, e.g. Arrows for directions

Sophisticated heuristics, lots of inference

Stop This!

Do Something!


Demonstrating Hidden Features


Report Generation - Masuishi

Mundane, but hey, could save a lot of time

Format generalization

Application-specific iteration

Distributed to wide user community


Report Generation by Example


Dynamic Macro - Masui

Notices loops in text editing commands

Simple, automatic: No start/stop recording

Wide community of users

Also: POBox predictive input for PDAs, phones, etc.


Dynamic Macro


SmartEdit - Lau, Domingos, Wolfman, Weld

Automates repetitive text-editing procedures

Predictive, anticipatory, loop detection

Uses version-space learning


SmartEdit Version Space


SmartEdit Screens


Grammex: "Grammars by Example"- Lieberman, Nardi, Wright

Pick out "needles" of structured information in "haystack" of unstructured information

First direct-manipulation user interface to powerful grammar and parsing technology

Show example strings, explain components recursively

Iterative: System parses, you explain


Defining Recursive Grammars


Visual representation of regular expressions - Blackwell

SWYN: "See what you need"

Representation of positive and negative examples

Anticipation of selections updated dynamically

Representation of grammars as Finite-State machines

User study of understandability of representations


Visual representation of Regular Expressions


APE: PBE for Programming Environments - Ruvini, Dony

Combining predictive interfaces with PBE

Conventional programming environment: Smalltalk

Automates coding, debugging procedures

What-set, When-set use different learning algorithms

C4.5, new algorithm IDHYS can learn non-consecutive patterns


Repetitive code in protocols


Familiar - PBE across applications - Paynter

PBE for Applescript

Application-independent, no mods to applications, uses commercial OS scripting language

Limited by shortcomings of Applescript and application implementations


Familiar arranging files


Tatlin - Gaxiola, Lieberman

Examinability: Some applications will let you look at their data even if you can't record them or use API

PBE by comparing states and inducing transformations rather than recording and generalizing actions


Tatlin - PBE via Examinability


Demonstrating What vs. When- Wolber, Myers

Saying "when" you want something to happen

Stimulus-Response PBE

Pavlov: Animation scripting system like Macromind Director


Pavlov driving game


Programming by Analogous Examples - Repenning, Perrone

Agentsheets - Grid world like Stagecast Creator

If-then rules by example

Make generalizations by analogy to generalizations of previous examples


Programming by Analogous Examples


Visual Generalization - St. Amant, Zettlemoyer, Lieberman, Potter

Actually parse the screen!

Simulate human user's visual system

Finesses data access problem, hooking to applications

Fragile - if viz rep changes, you're screwed

Permits unique visual generalizations not possible by accessing data


Step 1 - Select URL text field:

Step 2 - Start mouse drag to select deepest directory:

Step 3 - Finish mouse drag:

Step 4 - Press backspace to delete the selection:

Triggers - "Just the pixels, ma'am"


Solitaire by "Parsing the screen"


“Short order programming” exercise

Task: Add similarity-based learning to an action-based learning system

Ingredients: Graphical editor, action recording, relation inference


Things to think about

How could we integrate the paradigms of Mondrian and Eager?

What sort of information do we need to get from the graphical editor?

What should the learning algorithm learn?

How should the result of learning affect the interface?

Henry Lieberman MIT Media Lab Programming by Example Henry Lieberman Media Laboratory Massachusetts...

Documents

Transcript of Henry Lieberman MIT Media Lab Programming by Example Henry Lieberman Media Laboratory Massachusetts...