Retrospective and Challenges for Model-Based Interface Development Pedro Szekely University of...

Retrospective and ChallengesRetrospective and Challengesfor Model-Based for Model-Based

Interface DevelopmentInterface Development

Pedro SzekelyPedro SzekelyUniversity of Southern California,University of Southern California,

Information Sciences Institute, USAInformation Sciences Institute, USA

http://www.isi.edu/isd/Mastermindhttp://www.isi.edu/isd/Mastermind

6/5/96 2DSV-IS'96

AcknowledgmentsAcknowledgments

Mastermind Project Mastermind Project USC/ISIUSC/ISI

Pedro Szekely, Project LeaderMartin Frank, Computer Scientist

Pablo Castells, Visiting Scientist (U Madrid)Ewald Salcher, Graduate Student (U Graz)

Mastermind Project Mastermind Project Georgia TechGeorgia Tech

Jim Foley, Project LeaderSpencer Rugaber, Project Leader

Kurt Stirewalt, ProgrammerNoi Sukaviriya (now at IBM)

Funding provided byDARPA

USA Defense Advanced Research Projects Agency

Allen Sears, Program Manager

6/5/96 3DSV-IS'96

OutlineOutline

High quality interfaces

Model-Based User Interface Development

Analysis of model based tools

New directions

Mastermind

6/5/96 4DSV-IS'96

User Interface Tools GoalUser Interface Tools Goal

Quality

DevelopmentEffort

High Quality InterfacesHigh Quality Interfaces

6/5/96 6DSV-IS'96

Interface QualityInterface Quality

Task support Actively supports the tasks users need to perform

Presentation Show information needed for tasks in an easy to assimilate format

Dialog Efficient, easy to learn, natural for task

Context sensitivity Sensitive to platform characteristics, user profile, usage history

Multi-modal interaction Speech, language, pen in addition to mouse and keyboard

6/5/96 7DSV-IS'96

Task SupportTask Support

Systems with the following capabilities Understand the tasks the user is performing Understand its own capabilities Actively assist users in performing their tasks Show information needed to perform the tasks Anticipates subsequent tasks Prevents errors, provides error recovery

MS wizards: a step in the right direction

6/5/96 8DSV-IS'96

Task Support Example: MS WordTask Support Example: MS Word

MS Word DocumentMS Word Document

6/5/96 9DSV-IS'96

Task Support Example: MS WordTask Support Example: MS Word

1) Invoke answer wizard1) Invoke answer wizard

MS Word DocumentMS Word Document

2) Fill in question2) Fill in question

3) Select topic3) Select topic

4) Ask for advice4) Ask for advice

6/5/96 10DSV-IS'96

Task Support (contd.)Task Support (contd.)

5) System shows instructions5) System shows instructions

6/5/96 11DSV-IS'96

Task Support (contd.)Task Support (contd.)6) System show which menu to use6) System show which menu to use


6/5/96 12DSV-IS'96

Task Support (contd.)Task Support (contd.)6) System show which menu to use6) System show which menu to use


7) System shows dialogue box7) System shows dialogue box

8) System explains what to do8) System explains what to do

6/5/96 13DSV-IS'96

Task Support: The Correct AnswerTask Support: The Correct Answer

Wizard is brittleWizard is brittle

The answer I was looking forThe answer I was looking for

6/5/96 14DSV-IS'96

High Quality PresentationsHigh Quality Presentations

Show the information that users need to perform their tasks

… in a format that makes it easy to assimilate information

… sensitive to the amount and kind of data

… using appropriate graphic design principles

… taking into account cultural conventions

… and capabilities and handicaps of user

6/5/96 15DSV-IS'96

Easy To Assimilate Presentations Easy To Assimilate Presentations

Presentations generated with SAGE (Steve Roth, CMU)

Input: database tables User: select columns to view System: designs presentation

6/5/96 16DSV-IS'96

Context SensitivityContext Sensitivity

Platform sensitivity Displays that adapt to screen size, color depth, resolution

Interaction that adapts to network bandwidth and latency

User sensitivity Interaction and displays that adapt to user experience and skill

Interaction sensitivity Interfaces that remember previous interaction

Remember window placement, sizes and options

Remember user choices

6/5/96 17DSV-IS'96

Multi-Modal InteractionMulti-Modal Interaction

Pen

Gesture, handwriting

Speech input

Limited natural language

Deictic input

E.g. “put that there”

Phil Cohen, OGI

Development ToolsDevelopment Tools

6/5/96 19DSV-IS'96

HCI Development ProcessHCI Development Process

Developers

Guidelines

Past Experience

User Requirements

Task Analysis

DesignSpecifications

User

6/5/96 20DSV-IS'96


Developers

Guidelines

Past Experience

User Requirements

Task Analysis


ExecutableSpecification

User

Libraries

Design Knowledge

Context

6/5/96 21DSV-IS'96


Developers

Guidelines

Past Experience

User Requirements

Task Analysis

User Testing Data


ExecutableSpecification

User

Analytical Evaluation

Libraries

Design Knowledge

Context

Advice

6/5/96 22DSV-IS'96

Domains Of ExpertiseDomains Of Expertise

DevelopersDevelopers Application domain Users Tasks Work context

Graphic design HCI

High level decisions

ToolsTools Book keeping Search Repetitive tasks

Application independent domains Chart generation Style rule application and

checking

6/5/96 23DSV-IS'96

ToolsTools

Specification tools Word processors, drawing programs Layout tools, visual programming

Transformation tools E.g., task model to Petri nets

Advice/design exploration tools E.g., TRIDENT interaction object selection

Generation tools E.g., Adept, Humanoid, Trident, ...

Evaluation tools NGOMSEL, model checkers

6/5/96 24DSV-IS'96

Informal Definition of Formal Informal Definition of Formal SpecificationsSpecifications

A formal specification is used by a program to:

derive useful information for developers

generate an executable specification

generate another formal specification

6/5/96 25DSV-IS'96






Informal specifications Text Drawings

Formal specifications Logic Semantic net Rules Programs

6/5/96 26DSV-IS'96






Informal specifications Text Drawings

Formal specifications Logic Semantic net Rules Programs

Model-Based Tools:use formal specifications as input

6/5/96 27DSV-IS'96

Knowledge Sources of ToolsKnowledge Sources of Tools

Knowledge Kind Application Specific Library/Knowledge Base

Run-time Context

Who Creates It Applicationdevelopers

Tool builders Automatic

Who Uses It Tools Developers,Tools

Application

Examples TaskPresentationDialogUser profiles

Interactionobjects

IO selection rulesStyle rulesCases

Interactionhistory

Platformcapabilities

User preferences

6/5/96 28DSV-IS'96

Ease Of Use Issues In Model-Based Ease Of Use Issues In Model-Based ToolsTools

Modeling problem Tools need detailed application knowledge Difficult for application developers to formalize Requires expressive notations

Reuse problem Developers need to find and understand the relevant portions of

tool knowledge bases

Control problem Developers need to understand how their inputs affect the

observable behavior of tools Need to understand “what to say” to obtain desired behavior

6/5/96 29DSV-IS'96

Expected Benefits of Expected Benefits of Model-Based ToolsModel-Based Tools

Tools Quality Impact Cost ImpactSpecification Reduce effort to construct

specification

Transformation Enable the use of other tools

Interface generation Rapid prototyping enablesmore iterations

Implementation derivedautomatically fromspecification

Advice Based on HCI theory andexperience

Reduces design time

Evaluation Detect design problemsautomatically

Errors found early arecheaper to fix

6/5/96 30DSV-IS'96

Classification Of Model-Based ToolsClassification Of Model-Based Tools

Purpose: Specification Transformation Interface generation Advice Evaluation

Features: Task support Presentation Dialog Context sensitivity Multi-modal

6/5/96 31DSV-IS'96



Language semantics: Mathematical Defined by tools


Application domain: Narrow General

Extensibility: Open Closed

6/5/96 32DSV-IS'96


Ease of use: Modeling Reuse Design control






Model contents: Task Application Presentation Dialog User Platform

AnalysisAnalysis

6/5/96 34DSV-IS'96

Successful Model-Based ToolsSuccessful Model-Based Tools(High Quality, Lower Cost)(High Quality, Lower Cost)

ITS (Charlie Wiecha, IBM) Style rules to define presentation Small set of style rules for large

number of windows Easy to adapt to different devices

mouse, touch screen small, large displays

Easy to apply policy changes to many windows

Less work than defining each window individually

Industrial strength tool Seville World Fair Kiosks many others ...

6/5/96 35DSV-IS'96

Successful Model-Based ToolsSuccessful Model-Based Tools(High Quality, Lower Cost)(High Quality, Lower Cost)

ITS (Charlie Wiecha, IBM) Style rules to define presentation Small set of style rules for large

number of windows Easy to adapt to different devices

mouse, touch screen small, large displays

Easy to apply policy changes to many windows

Less work than defining each window individually

Industrial strength tool Seville World Fair Kiosks many others ...

Sage (Steve Roth, CMU) Automatic presentation generation Presentations are of high quality Deep knowledge of chart design Narrow domain Model

Developer models data Simple way to control design

6/5/96 36DSV-IS'96

Why Are ITS and SAGE Successful? Why Are ITS and SAGE Successful?

Do one thing well Extensible

Can’t model it, program it Specification language

Accessible to developers, even though not easy (ITS)

Model covers surface aspects ITS: presentation & dialogue

SAGE: presentation Designer control

Automation, … but developer has last word






6/5/96 37DSV-IS'96

Best Automatic Generation Tool: Best Automatic Generation Tool: TRIDENTTRIDENT

Basic approach High level specification

Application & Task models Generate

Presentation & Dialogue Use rich knowledge bases

Style guides for choosing interaction objects

Use sophisticated algorithms Decision tree for IO selection Metric based layout Task-based window allocation



Extensibility: Open Closed ???



6/5/96 38DSV-IS'96

Lessons For Automatic GenerationLessons For Automatic Generation

TRIDENT lessons Narrow domain enables

progress Automatic interface generation

is a VERY hard problem Results are good … lots remain to be done

How to make it better?

Window allocation, IO selection & layout cannot be done independently Space influences everything Choice of IO influences space,

=> window allocation & layout Layout influences IO selection,

window allocation

Modeling burden? Modeling? Developers specify what they want?

6/5/96 39DSV-IS'96

Evaluation ToolsEvaluation Tools

Logic-based Still in infancy Useful in special situations

Medicine, factory control Modeling burden is too large Asking the right questions is

difficult

Too hard!

Task based (e.g., GOMS) Less powerful than logic based,

Predictions execution time and learnability

Much easier to do Similar to established evaluation

techniques cognitive walkthrough

GOMS models automatically derived from task models used for generation

6/5/96 40DSV-IS'96

Advisor/Critiquing ToolsAdvisor/Critiquing Tools

Analogy to word-processing Spelling checkers

Detect errors Suggest corrections Custom dictionaries

Grammar checkers Marginally useful No correction

Content checkers Non-existent

6/5/96 41DSV-IS'96

Advisor/Critiquing ToolsAdvisor/Critiquing Tools

Analogy to word-processing Spelling checkers

Detect errors Suggest corrections Custom dictionaries

Grammar checkers Marginally useful No correction

Content checkers Non-existent

Interface design Style guide checkers

(e.g., Gerhard Fischer) Like grammar checkers Detect lots of “violations” Little on correction

Layout metrics

Low semantics advisors? Like spelling checkers

6/5/96 42DSV-IS'96

Summary of Current Model-Based Summary of Current Model-Based WorkWork

Generation tools Generation from task and application/data models Automatically design presentation and dialog Progress being made (TRIDENT)

Successes Generation: very narrow domain, sophisticated algorithms

(SAGE) Not so narrow, but developer designs presentation and dialog

(ITS) Payoff is in the features it enables: platform retargetting, reuse

New Directions New Directions

6/5/96 44DSV-IS'96

Shift FocusShift Focus

Automatic Generation

Automatic Evaluation

Interface Quality Task awareness/support Intelligent presentation Context-sensitivity Multi modal interfaces

6/5/96 45DSV-IS'96

Interface TrendsInterface Trends

Technology Interface Paradigm Interface Tools

CRT 24x80Keyboard

Command lineForms

4GLDOSCurses (UNIX)

14” Color monitorMouseKeyboard

WIMPDirect manipulation

Interface buildersVisual Basic4GL

2” - 21” MonitorPhonePenMulti-mediaNetwork

WIMP +GestureHand-writingLimited speech3D

???

??? Task supportCoachesIntelligent

presentation

???

1984

1995

2000

6/5/96 46DSV-IS'96

Avoid Being One Avoid Being One Generation BehindGeneration Behind

Interface Paradigm

Command lineForms

WIMPDirect manipulation

WIMP +GestureHand-writingLimited speech3DTask supportCoachesIntelligent presentation

1984

1995

2000

WIMP era is ending

Model-based tools are just maturing for the WIMP era: Generation of WIMP Evaluation of WIMP


Tools to construct multi-modal interfaces

6/5/96 47DSV-IS'96

New Challenge:New Challenge:Platform ScalabilityPlatform Scalability

WIMP era: single platform

New era: many platforms Large monitors Laptops PDAs Powerful pagers Networks

Technology Interface Tools

CRT 24x80Keyboard

4GLDOSCurses (UNIX)

14” Color monitorMouseKeyboard

Interface buildersVisual Basic4GL

2” - 21” MonitorPhonePenMulti-mediaNetwork

Model-BasedTools

??? ???

1984

1995

2000Tools to construct interfaces that work

across platforms


6/5/96 48DSV-IS'96

Task SupportTask SupportIntelligent PresentationIntelligent Presentation

Task support Embed more semantics in the

interface Understand tasks Cooperate with user to map

goals to system capabilities Prevent & recover from errors Streamline task performance

Task models Use them for support, … not only for generation

Intelligent presentation Help users absorb the

information they need

Task models: The context needed to design

presentations at run-time

Direct manipulation +

multi-modal interfaces are not enough

6/5/96 49DSV-IS'96

SummarySummary


Interfaces that reason about tasks about information about platform capabilities about user behavior

6/5/96 50DSV-IS'96

SummarySummary


Interfaces that reason about tasks about information about platform capabilities about user behavior

Model-Based InterfacesModel-Based Interfaces

MASTERMINDMASTERMIND

Showcase of capabilities in the context of querying logistics databases

05/28/96 52MASTERMIND

Query Interfaces: Query Interfaces: User’s ProblemUser’s Problem

Example queryExample query “Which radio/TV/com equipment is in less than condition code A or B in AR4”

User questionsUser questionsHow do I find radio/TV/com equipment?How do I specify condition codes?How do I specify that the item must be in AR4?What information can I get back?How do I know that my query will work?


Query Interfaces: Query Interfaces: User’s ProblemUser’s Problem


SolutionSolutionDisplays that show the relevant terminologyDialogue that matches the user tasks

Find main concepts (radio/TV/com equip)Specify restrictions (condition code)Specify info requested (nomenclature, price)

User questionsUser questionsHow do I find radio/TV/com equipment?How do I specify condition codes?How do I specify that the item must be in AR4?What information can I get back?How do I know that my query will work?


Presentation ModelPresentation Model

Banner

Category Category Category

Banner Relation Relation

Banner Type In

Banner

BannerRefinements Refinements Refinements

Restrictions Restrictions Restrictions

Outputs Outputs Outputs

Banner

Button Button Button

C Label

S Label

S Label

C LabelC Label

S LabelS Label

S LabelS Label

Developer View Derived Implementation


Presentation ModelPresentation Model

Layout scheme based on graphic design theory Produce visually pleasing

displays Pink elements contain LAD-

specific behavior Reused in alternate designs Recursively defined using

same technique Novel replication mechanism

Supports dynamic behavior without programming

Banner

Category Category Category

Banner Relation Relation

Banner Type In

Banner

BannerRefinements Refinements Refinements

Restrictions Restrictions Restrictions

Outputs Outputs Outputs

Banner

Button Button Button

C Label

S Label

S Label

C LabelC Label

S LabelS Label

S LabelS Label


Task AwarenessTask Awareness


““Specify restriction” taskSpecify restriction” task

Choose attribute to restrict

Choose operator

Specify restriction value

Specify restriction information


Task AwarenessTask Awareness


““Specify restriction” taskSpecify restriction” task

Choose attribute to restrict

Choose operator

Specify restriction value

Specify restriction information

1. What are the restrictable attributes of my object?2. How do I specify the one I want?3. Does the attribute match any keywords?4. Is it restricted already? To what?5. Can I disable the restriction without deleting it?

6. Which widget invokes the restriction display?7. What happens if the restriction display is already active?8 What else can be done while specifying a restriction? e.g., specify an output?

9. Which operators are valid for this attribute?

10.What values are legal for the selected attribute given a choice of operator ?11.What happens when an invalid combination of operator & value is entered?

Questions that task centered interfaces must answerQuestions that task centered interfaces must answer

Mastermind provides facilities to enable interface to address these questionsMastermind provides facilities to enable interface to address these questions


Facilities For Task AwarenessFacilities For Task Awareness

1. What are the restrictable attributes of my object?2. How do I specify the one I want?3. Does the attribute match any keywords?4. Is it restricted already? To what?5. Can I disable the restriction without deleting it?

6. Which widget invokes the restriction display?7. What happens if the restriction display is already active?8 What else can be done while specifying a restriction?

e.g., specify an output?9. Which operators are valid for this attribute?

10. What values are legal for the selected attribute given a choice of operator ?

11. What happens when an invalid combination of operator & value is entered?

Choose attribute is_a Primitive Choice Task(1) choices: <<link to interface state model>>

Specify restr info is_a Dialogue Task

Specify restriction(7) reentrant: False

(2) interactor is_a Choice Inter(6) where: <<link to presentation model>>action: activate

Done is_a Done Dialogue Task(10) preconditions:(11) recovery:

Cancel is_a Cancel Dialogue Task

Modeling language provides terminology to express the answers

Raises the issues Forces developers to think solutions

Reusable components contain answers

... choose operator, specify restriction value


Summary of LeverageSummary of Leverage

Task model makes interface “task aware” Computes and maintains the info that users need to perform tasks Defines the sequencing of steps to complete the tasks

Presentation model manages information presentation Easy for developers to define & change presentations appropriate to task

Modeling is more productive than programming Models are much smaller than equivalent code Model checkers ensure generation of correct source code

Model enables adaptation of interface Platform scaleable adaptation of LAD interface done in 1 day 2nd version of LAD interface done in 3 days


Analysis of InterfaceAnalysis of Interface

Presentation model success Presentation model is small Model checker prevents common

Amulet errors Generated Amulet is large

Interface state is complex Coded by hand LAD Largest piece of code Currently being modeled

Update code Coded by hand LAD Small, but longest debugging Change manager will automate it

LAD Interface Code Breakdown

0

2000

4000

6000

8000

10000

12000

14000

16000

KB

KB

Par

ser

Qu

ery

Tcp

Ip

UI

Sta

te

Pre

sen

tati

on

Act

ion

s

Up

dat

e

Inte

ract

ors

Wo

rds

Application Interface

Presentation model

Generated code


Evolution:Evolution:Version 1 to Version 2 Version 1 to Version 2

One new component(based on old one)

• New layout defined with a new grid• Change of color scheme• Interface state of outputs and restrictions merged


Platform AdaptationPlatform Adaptation

Workstation InterfaceWorkstation Interface



• Refinements become pop-up

• Components scale appropriately


Preserve structure of interfacePreserve structure of interfaceto facilitate user transferto facilitate user transfer


Laptop InterfaceLaptop Interface



• Refinements become pop-up

• Components scale appropriately

• Title disappears• Keywords move• Constraints

collapsed




PDA InterfacePDA Interface

Laptop InterfaceLaptop Interface


Adaptation:Adaptation:Workstation, Laptop & PDAWorkstation, Laptop & PDA

Workstation to PDA: Defined thresholds on grid:

When space smaller than threshold, make it zero

When component disappears, add button to pop it up

Defined conditional settings for Position of keyword type-in Font-sizes

Interface state modifications to control contents of pop-up

Pop UpPop Up

Retrospective and Challenges for Model-Based Interface Development Pedro Szekely University of...

Documents

Transcript of Retrospective and Challenges for Model-Based Interface Development Pedro Szekely University of...