Permafrost in Canada's Subarctic Region of Northern - T-Space
The subArctic Input System and Extensions for Handling Inputs with Ambiguity
description
Transcript of The subArctic Input System and Extensions for Handling Inputs with Ambiguity
![Page 1: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/1.jpg)
1
The subArctic Input System
and
Extensions for Handling Inputs with Ambiguity
![Page 2: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/2.jpg)
2
subArctic A Java-based GUI toolkit that I
(along with Ian Smith) built and distributed in 1996-97
Goal: highly extensible allowing support for lots of cool new interaction techniques– Emphasis on making new and strange
widgets / components / interactors easy to create
– “High ceiling”
![Page 3: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/3.jpg)
3
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
![Page 4: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/4.jpg)
4
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
Most toolkits target support
here
![Page 5: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/5.jpg)
5
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
By moving work up (into reusable library)
![Page 6: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/6.jpg)
6
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
But typically don’t help much here
(assume a fixed library)
![Page 7: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/7.jpg)
7
subArctic Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
![Page 8: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/8.jpg)
8
subArctic Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
Input system is a big part of that
![Page 9: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/9.jpg)
9
Schema for pretty much all GUIs
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
![Page 10: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/10.jpg)
10
Schema of a GUI
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
Event Record – recording of the relevant facts about
some occurrence of interest (i.e., user has manipulated
an input device)
![Page 11: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/11.jpg)
11
Schema of a GUI
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
Send (“dispatch”) the event to the object(s) that want it and/or know
how to respond to it(e.g., widget/component/interactor)
![Page 12: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/12.jpg)
12
Event dispatch All the work happens here Typically delegated to interactors
– E.g., buttons know how to respond to press and release like buttons should
– Each object keeps track of its own state
... but which interactor gets it Toolkit “event dispatch” process
![Page 13: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/13.jpg)
13
Event dispatch policies
Two primary ways to decide which interactor gets an event
What are they?
![Page 14: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/14.jpg)
14
Event dispatch policies Two primary ways to decide which
interactor gets an event– Positional dispatch
Based on where mouse is pointingExamples…
– Focus-based dispatchDesignated object always gets inputExamples…
![Page 15: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/15.jpg)
15
Pop quiz
Should input for dragging be dispatched via positional or focus?
![Page 16: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/16.jpg)
16
Pop quiz
Should input for dragging be dispatched via positional or focus?
Answer: No! (both)
![Page 17: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/17.jpg)
17
subArctic input policies
subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects”– Manages bookkeeping (e.g., picking)– Extensible set
Turns out there are other useful policies (e.g., for modal dialogs)
![Page 18: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/18.jpg)
18
When interactors get events…
… they typically respond to them with the equivalent of a simple finite state machine
Press
Move
Release
![Page 19: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/19.jpg)
19
subArctic has lib of common FSMs Move a lot of input handling work
typically done by interactor programmer up into the toolkit
One (highly parameterized) FSM for all – Brad’s “interactor” model (awful terminology :-)
Many customized FSM (extensible set)– subArctic input model
![Page 20: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/20.jpg)
20
FSMs moved to toolkit object
“Dispatch agent”
Translates low level input into higher level terms
![Page 21: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/21.jpg)
21
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
![Page 22: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/22.jpg)
22
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
Defined by Java interface
![Page 23: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/23.jpg)
23
Set of dispatch agents is extensible E.g., can subclass for
specialized kinds of drag such as “drag_within_box” or “snap_drag”– Can create custom for one interface– Once created can reuse
![Page 24: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/24.jpg)
24
How it all goes together
Focus Policy
Positional Policy
Etc…Events
Press
Click
Rollover
Etc...
Etc...Text
Move drag
Grow drag
Etc...
![Page 25: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/25.jpg)
25
How does interactor indicate it wants / can handle some type of input? “… implements input_protocol”
– Where “input_protocol” is interface with calls like drag_start(), etc.
For positional that’s it! For focus-based must also ask
for focus
![Page 26: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/26.jpg)
26
Example: Hypertext for all User (Ken Anderson) wanted to
add hyperlinks to all objects– Hold down the control key and click– His external hyperlink database
would take over and map interactor id to hyperlink target
– But… how do you change every interactor to do this?
![Page 27: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/27.jpg)
27
Example: Hypertext for all In Swing, Motif, etc. this is
essentially impossible
In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down– About 15 lines of code– Works for interactors written later!
![Page 28: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/28.jpg)
28
Questions about the SA input system?
![Page 29: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/29.jpg)
29
Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input
See: http://doi.acm.org/10.1145/354401.354407and http://doi.acm.org/10.1145/332040.332459
Jennifer Mankoff, Gregory Abowd
Georgia Institute of Technology
Scott HudsonCarnegie Mellon University
![Page 30: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/30.jpg)
30
Motivation Recognition-based input offers
the promise of naturalistic input modalities, BUT…
![Page 31: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/31.jpg)
31
Motivation Recognition-based input offers
the promise of naturalistic input modalities, BUT…
Recognizers are imperfect– affects users– breaks current system models
New interfaces & mechanisms
![Page 32: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/32.jpg)
32
Example Interaction
From Newton
Handwritten text
![Page 33: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/33.jpg)
33
Example Interaction
From Newton
Handwritten text automatically replaced with best recognition result
test
![Page 34: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/34.jpg)
34
Example Interaction
Double-tap to get a correction interactor
test
![Page 35: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/35.jpg)
35
Example Interaction
Correction interactor (mediator)
testtextteatted
N-Best List
Keyboard
Character Correction
Revert to Strokes
![Page 36: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/36.jpg)
36
Example Interaction
Works well, but…– Not reusable or customizable– Hard to grow your own
Basically we don’t have toolkit support for recognition based UI
![Page 37: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/37.jpg)
37
Motivation (cont.) At much the same stage we
were at for GUIs in 1983– No common model for input– No re-use
Infrastructure“widget library”
![Page 38: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/38.jpg)
38
Goals of This Work Robust, reusable infrastructure Reusable library Integrate with convent. toolkit
– Don’t throw out the baby with the bathwater
![Page 39: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/39.jpg)
39
Talk Roadmap
Requirements for handling uncertain input
Extending toolkits to handle it Interaction techniques for
ambiguity Implementation
![Page 40: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/40.jpg)
40
Invoking Application Actions Action often done by callbacks
– Direct procedure call to application
Hierarchical events are alternate approach– Delivered to app as well as toolkit
![Page 41: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/41.jpg)
41
Hierarchical Events Low-level events contribute to
production of higher-level events
[Green TOG ‘86; Myers & Kosbie CHI ‘96]
User Inputcircle
stroke
down drag up• • • • • •
Corresponding Events
![Page 42: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/42.jpg)
42
Implicit Assumption of Certainty Implicit in all this is the
assumption that the events really happened as reported
Problems arise when this isn’t true– E.g., brittle dialogs
![Page 43: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/43.jpg)
43
Needed to Handle Uncertainty:
Allow for (and explicitly model) multiple alternatives– alternative higher level events– in recognition context: interpretations
Detect conflicting interpretations Mediation of conflicts
![Page 44: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/44.jpg)
44
Needed to Handle Uncertainty:
Lexical feedback about uncertain events – split “feedback” from “action”
Library of mediators
![Page 45: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/45.jpg)
45
How do we do this...
![Page 46: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/46.jpg)
46
Extended Event Model
Uncertainty results in multiple interpretations
interpretation graph
Uncertain Input
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
Certain Input
![Page 47: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/47.jpg)
47
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Now delivered to recognizers,
interactors, and application objects
Button
Checkbox Menu
Recog
![Page 48: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/48.jpg)
48
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Objects initially only produce
(reversible) feedback, no actions
Button
Checkbox Menu
Recog
![Page 49: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/49.jpg)
49
Another Change:
Interface Appearance Uncertain Event Hierarchy
circlebox
stroke
down drag up• • • • • •
Events dispatched to all who might use it
![Page 50: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/50.jpg)
50
Details: Arranging for Mediation Identify any conflicts Look for a mediators
– Pluggable list of them in toolkit Mediator chosen by meta-
mediator Mediator can:
“Pass”, “Pause”, “Accept”
![Page 51: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/51.jpg)
51
Doing Mediation
Example:User selects interpretation
circle
box
circle
![Page 52: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/52.jpg)
52
Doing Mediation (cont.)
Mediator prunes interpretation graph to tree
– App informed of accept & reject
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
![Page 53: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/53.jpg)
53
Mediation Strategies
Many mediation strategies– e.g., Automatic vs. user involvement
Toolkit is fully “pluggable” (!)– Library of mediators provided, but– Can extend/build new ones as needed
Research goal:Finding new ones
![Page 54: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/54.jpg)
54
Providing a Library of Mediators
![Page 55: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/55.jpg)
55
Providing a Library of Mediators
Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99]
– Automatic – User Involvement
Repetition & repair strategiesChoice strategies
![Page 56: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/56.jpg)
56
Automatic Mediation Techniques Probability modeling and
thresholding
Historical statistics
Rule-based
![Page 57: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/57.jpg)
57
User Involvement: Repetition & Repair Strategies Undo and repeat
Change of modality
Partial repair
![Page 58: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/58.jpg)
58
User Involvement:Choice Strategies Variations on N-best “lists”
– Presentation form– Instantiation time– Contextual information– Interaction – Feedback
Many techniques via “parameterization”– Ripe for toolkit support
![Page 59: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/59.jpg)
59
Implementation
Added to subArctic toolkit– Reusable– Fully “pluggable”– Full existing library still works as is (!)
Small library of mediators Also working on non-GUI toolkit
![Page 60: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/60.jpg)
60
Experience
Major example: Burlap– Smaller version of SILK [Landay]– For sketching
UI designs and turning them into functioning interfaces
![Page 61: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/61.jpg)
61
Conclusions Reusable infrastructure to
support ambiguous input– Reduces difficulty of creating UIs– Easier to explore new design space
Done by modifying a toolkit, not a separate mechanism– Integrated with conventional input – Other support from toolkit still useful
![Page 62: The subArctic Input System and Extensions for Handling Inputs with Ambiguity](https://reader035.fdocuments.us/reader035/viewer/2022062811/56815fc9550346895dcec08d/html5/thumbnails/62.jpg)
62