1 Informal Tools for Multimodal, Context-based User Interface Design James A. Landay July 7, 1999...

1

Informal Tools for Multimodal, Context-based User Interface

Design

James A. LandayJuly 7, 1999HCC Retreat

http://guir.berkeley.edu

2

Natural Tides of Innovation

Time

Integration

Innovation

Log R

Mainframe

Minicomputer

Personal ComputerWorkstationServer

7/99

??Universal

Computing

3

Universal

1 : including or covering all or a whole collectively or distributively without limit or exception

2 a : present or occurring everywhere b : existent or operative everywhere or under all conditions <universal cultural patterns>

3 a : embracing a major part or the greatest portion (as of mankind) <a universal state> <universal practices> b : comprehensively broad and versatile <a universal genius>

4 a : affirming or denying something of all members of a class or of all values of a variable b : denoting every member of a class <a universal term>

5 : adapted or adjustable to meet varied requirements (as of use, shape, or size)

4

Powerful, personal capabilities from specialized devices* small, highly mobile or embedded in

environment “Intelligence” + immense storage and

processing in the infrastructure Everything connected

Laptops, Desktops

SpecializedDevices

Away From the “Average Device”

5

HCI Challenges

Universal computing devices will not have the same UI as “dad’s PC”* a wide range of devices

- often w/ small or no screens & alternative I/O

+ e.g., pens, speech, vision

- special purpose to particular applications

+ “information appliances”

* lots of devices per user- all working in concert

6

HCI Challenges (cont.)

Design of good appliances will be hard* single device design is easy* hard to design the same “application” in a consistent

manner across many devices- e.g., calendar app.: one speech based & one GUI based

* hard to make different devices work together- which device is used when?- multiple UIs & modes/device, which to “display”?- building awareness of context of use into design is the key to

some of these issues

* multimodal input is assumed, but little design support for creating multimodal interfaces

7

Our Approach

Build* novel applications on existing appliances

- e.g., NotePals on the Palm PDA & CrossPad

* new information appliances Evaluate appliances in realistic

settings Iterate

* use the resulting experience to build - more interesting appliances - better design tools & analysis techniques

8

Outline

HCI Challenges for Universal Computing Multimodal interaction Why is building MM UIs hard? Best practices for designing GUIs Proposed approach to building MM UIs Using context to our advantage

9

Multimodal Interaction

When communicating with people we use more than one mode at a time!* gesture & speak

* sketch & speak* etc.

Computers would be easier to use and more useful if they also worked like this

10

Benefits of Multimodal (MM) Interaction on Heterogeneous

Devices Computers could be used in

* more situations- when hands are full or vision used for something else

* more places- e.g., walking, car, factory floor, etc.

Interfaces would be easier to use* use innate perceptual, motor, & cognitive skills

More people would be able to use computers* including users w/ vision or motor impairments

MM UIs likely to become predominant

11

Why is building MM UIs hard?

Often require “recognition” technology* speech, handwriting, sketches, gesture, etc.

Recognition technology is immature * finally “just good enough”* single mode toolkits just appearing now* no prototyping tools

Hard to combine recognition technologies* still requires experts to build systems* few toolkits or prototyping tools!

This was the state of GUIs in 1980

12

Best Practices for Designing GUIs

Iterative design

Prototyping tools are key to this success

Design

Prototype

Evaluate

13

Early Stage UI Design

Brainstorming* put designs in a tangible form* consider different ideas rapidly

Incomplete designs* do not need to cover all cases* illustrate important examples

Present several designs to client or design team

No need at this stage for “coding”

14

Prototyping Tools for Multimodal UI Design Should Support

Iterative design methodology

Informal techniques designers currently use in the early stage of UI design* sketching

15


Iterative design methodology

Informal techniques designers currently use in the early stage of UI design* sketching* storyboarding

16


Iterative design methodology Informal techniques

designers currently use in the early stage of UI design* sketching* storyboarding* “Wizard of Oz”

Landay Dictation Machine

|Um…. I’ll see you in the morning.

17

Our Approach: Sketches, Models & Context-aware Design Tools

Infer models from design “sketches”or other informal representations* model is an abstraction of apps UI design* model for representing contexts & UI implications

Use models to* semi-automatically generate UIs on diverse

platforms* dynamically adapt a particular appliance UI to

changing context of use

18

How to Specify Events

We have a good idea how to for visual UIs* visually!

But how about speech or gestures?

19

Specifying Non-Visual Events

How do designers do this now?* speech

- scripts or grammars (advanced designers only)- flowcharts on the whiteboard- “Wizard of Oz” -> fake it!

* gestures- give an example & then tell programmer what it does

We can do the same by demonstration (PBD)* demonstrate example of act (e.g., speech)* demonstrate result* system infers program

- just a prototype, so doesn’t have to be too general

20


result of demonstrating a pen gesture for delete

21

Combining the Visual & Non-Visual

How do you see what the system inferred?* necessary for editing * generate a visual representation

- flowchart seems like a start (common in speech UIs)

+ appropriate? what should it look like?

22


Type: System ResponseMode: Speech

“Please name the departure city”

Type: User StimulusMode: Speech

DEP_CITY


“DEP_CITY departures arriving to which city?”

Type: User StimulusMode: Speech

ARR_CITY

Type: Computation

Lookup ARRIVAL_TIME in “flight times” using DEP_CITY, ARR_CITY


“The flight from DEP_CITY to ARR_CITY will arrive at ARRIVAL_TIME”

A flowchart representing inferences made from the demonstration of a flight arrival time application

23

Combining the Visual & Non-Visual

How do you see what the system inferred?* necessary for editing * generate a visual representation

- flowchart seems like a start (common in speech UIs)+ appropriate? what should it look like?

Combining visual & non-visual events* e.g., end-user dragging truck while saying “fast”* use a visual language that combines visual

storyboard of GUI w/ flowchart for non-visual- VL better be simple...

24

Supporting Heterogeneous Devices

Consider sketches as an abstraction Infer a “model” from the sketches Use methods from model-based UI tools to

* generate UIs for multiple devices* generate alternative modes for a single spec. on

one device Hard problems

* how to abstract?* how do you generate a “good” UI?

- keep the designer in the loop

25

Take Advantage of Context: Monitor Environment & Actions to Improve

Interaction Which devices are present & available?

* there is a wall display -> use it for my wearable

- device discovery

26

Take Advantage of Context: Monitor Environment & Actions to Improve

Interaction

What is the state of the user?* hands using tools -> use speech I/O & visual Out

- tangible tools or vision processing

Solution: UI design tools that understand context as well as multiple devices & modalities

Which devices are present & available?* there is a wall display -> use it for my wearable

- device discovery

What is occurring in the environment?* people are talking -> don’t rely on speech I/O

- speech sensing

27

Design Goals

Let designer’s rapidly produce “rough cuts”* doesn’t need to handle all cases

Allow end user testing & fast modification Generate code that can help start UI for

multiple devices* designer adds more detail & improves

interaction* programmers add necessary code

28

What We’ve Accomplished So Far

Informal tools for UI design* sketch-based tools for GUI / Web design

- built & tested 1st generation, building next gen. now

29




* informal tool for speech UI design- designed & implementation in progress

30




* informal tool for speech UI design- designed & implementation in progress

Automatic generation of simple control UIs First cut designs for multimodal

* UI design tool & appliance (SpeechCorder w/ ICSI)

Experience w/ appliances & simple context* NotePals

31

Take Home Ideas

Universal Computing is about supporting people

Success will require the design & evaluation of new appliances (device + app + UI) that* take advantage of natural modes of input

- especially multimodal input!

* take advantage of context * are used in realistic settings

Experience, new architectures, and new tools will make this design problem easier

32

Informal Tools for Multimodal, Context-based User Interface

Design

James A. LandayJuly 7, 1999HCC Retreat

http://guir.berkeley.edu

33

Research Plan Finish implementation of informal tools

* study usage (especially of speech UI design)* use results to design multimodal design tool

Develop algorithms for extracting app model Build context-aware applications w/o tools

* two testbeds to create & study- wirelessly networked PDAs in classroom/learning- extraction of tacit context using social networking

* build taxonomy of contexts- how they should effect UI?

34

Research Plan (Cont.) Implement tool for multimodal UI design

* extract model & generates UI for 2 diverse platforms

* uses simple context ques Develop algorithms for capturing context Evaluate usage (apps & tools) in target

settings Extend multimodal UI design tool

* generate multi-platform UIs that dynamically adapt- allow context to be fully integrated in decisions

35

HCI Goals of Universal Computing

Some of the roots of Universal Computing are in the ideas of Mark Weiser* Ubiquitous/Pervasive/Invisible/Calm

Computing- “... does not live on a personal device of any sort

[PDA or dynabook], but is in the woodwork everywhere.”

- “you don’t want personal technology, you want personal relationships”

36

HCI Goals of Universal Computing

Some of the roots of Universal Computing are in the ideas of Mark Weiser* Ubiquitous/Pervasive/Invisible/Calm Computing

- “... does not live on a personal device of any sort [PDA or dynabook], but is in the woodwork everywhere.”

- “you don’t want personal technology, you want personal relationships”

Universal Computing is about* supporting people’s tasks

- most often includes working with other humans

* making people’s lives easier- just creating ubiquitous technology does not solve this

37

Computers Support Human-Human Communication (HHC)Designideas

PresentationsE-mail Reports

38

Force translations to formal representations* sometimes we want this (e.g., conference slides)* sometimes we don’t (e.g., creative tasks)

Traditional Software Interfaces

39

Traditional Representations

Rigid and unambiguous* hard to mix (e.g., few tools support rough

sketches)

* warp perceptions of the viewer and user Increase time

* encourage precision Inhibit creativity

* “tunnel vision”“Put me in a room with a pad & a pencil and set me up against a hundred people with a hundred computers -- I’ll outcreate every goddamn sonofabitch in the room.” -- Ray Bradbury, Wired 6.10

40

Informal Communications Styles

Speaking

Writing

Gesturing

Sketching

Informal UIs do not immediately translate naturalinput, allowing users to work more naturally

1 Informal Tools for Multimodal, Context-based User Interface Design James A. Landay July 7, 1999...

Documents

Transcript of 1 Informal Tools for Multimodal, Context-based User Interface Design James A. Landay July 7, 1999...