Post on 19-Feb-2016
description
1
The subArctic Input System
and
Extensions for Handling Inputs with Ambiguity
2
subArctic A Java-based GUI toolkit that I
(along with Ian Smith) built and distributed in 1996-97
Goal: highly extensible allowing support for lots of cool new interaction techniques– Emphasis on making new and strange
widgets / components / interactors easy to create
– “High ceiling”
3
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
4
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
Most toolkits target support
here
5
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
By moving work up (into reusable library)
6
Parties involved with a toolkit Toolkit designer (me)
Interactor designer
Interface programmer
User
But typically don’t help much here
(assume a fixed library)
7
subArctic Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
8
subArctic Toolkit designer (me)
Interactor designer
Interface programmer
User
SA tries to move work for many kinds of interactors into toolkit infrastructure
Input system is a big part of that
9
Schema for pretty much all GUIs
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
10
Schema of a GUI
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
Event Record – recording of the relevant facts about
some occurrence of interest (i.e., user has manipulated
an input device)
11
Schema of a GUI
init();for (;;) {
evt = wait_for_next_event();dispatch(evt);if ( damage_exists() ) redraw();
}
Send (“dispatch”) the event to the object(s) that want it and/or know
how to respond to it(e.g., widget/component/interactor)
12
Event dispatch All the work happens here Typically delegated to interactors
– E.g., buttons know how to respond to press and release like buttons should
– Each object keeps track of its own state
... but which interactor gets it Toolkit “event dispatch” process
13
Event dispatch policies
Two primary ways to decide which interactor gets an event
What are they?
14
Event dispatch policies Two primary ways to decide which
interactor gets an event– Positional dispatch
Based on where mouse is pointingExamples…
– Focus-based dispatchDesignated object always gets inputExamples…
15
Pop quiz
Should input for dragging be dispatched via positional or focus?
16
Pop quiz
Should input for dragging be dispatched via positional or focus?
Answer: No! (both)
17
subArctic input policies
subArctic encapsulates these “ways of dispatching inputs” in “dispatch policy objects”– Manages bookkeeping (e.g., picking)– Extensible set
Turns out there are other useful policies (e.g., for modal dialogs)
18
When interactors get events…
… they typically respond to them with the equivalent of a simple finite state machine
Press
Move
Release
19
subArctic has lib of common FSMs Move a lot of input handling work
typically done by interactor programmer up into the toolkit
One (highly parameterized) FSM for all – Brad’s “interactor” model (awful terminology :-)
Many customized FSM (extensible set)– subArctic input model
20
FSMs moved to toolkit object
“Dispatch agent”
Translates low level input into higher level terms
21
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
22
Dispatch agent example: move_drag
Translated to calls in input protocol:– drag_start(); – drag_feedback(); – drag_end();
With useful parameters (e.g. new pos)
Press
Move
Release
Defined by Java interface
23
Set of dispatch agents is extensible E.g., can subclass for
specialized kinds of drag such as “drag_within_box” or “snap_drag”– Can create custom for one interface– Once created can reuse
24
How it all goes together
Focus Policy
Positional Policy
Etc…Events
Press
Click
Rollover
Etc...
Etc...Text
Move drag
Grow drag
Etc...
25
How does interactor indicate it wants / can handle some type of input? “… implements input_protocol”
– Where “input_protocol” is interface with calls like drag_start(), etc.
For positional that’s it! For focus-based must also ask
for focus
26
Example: Hypertext for all User (Ken Anderson) wanted to
add hyperlinks to all objects– Hold down the control key and click– His external hyperlink database
would take over and map interactor id to hyperlink target
– But… how do you change every interactor to do this?
27
Example: Hypertext for all In Swing, Motif, etc. this is
essentially impossible
In SA, just insert a new subclass of the “click” dispatch agent that checks for the control key down– About 15 lines of code– Works for interactors written later!
28
Questions about the SA input system?
29
Providing Toolkit Level Support for Handling Ambiguity in Recognition-Based Input
See: http://doi.acm.org/10.1145/354401.354407and http://doi.acm.org/10.1145/332040.332459
Jennifer Mankoff, Gregory Abowd
Georgia Institute of Technology
Scott HudsonCarnegie Mellon University
30
Motivation Recognition-based input offers
the promise of naturalistic input modalities, BUT…
31
Motivation Recognition-based input offers
the promise of naturalistic input modalities, BUT…
Recognizers are imperfect– affects users– breaks current system models
New interfaces & mechanisms
32
Example Interaction
From Newton
Handwritten text
33
Example Interaction
From Newton
Handwritten text automatically replaced with best recognition result
test
34
Example Interaction
Double-tap to get a correction interactor
test
35
Example Interaction
Correction interactor (mediator)
testtextteatted
N-Best List
Keyboard
Character Correction
Revert to Strokes
36
Example Interaction
Works well, but…– Not reusable or customizable– Hard to grow your own
Basically we don’t have toolkit support for recognition based UI
37
Motivation (cont.) At much the same stage we
were at for GUIs in 1983– No common model for input– No re-use
Infrastructure“widget library”
38
Goals of This Work Robust, reusable infrastructure Reusable library Integrate with convent. toolkit
– Don’t throw out the baby with the bathwater
39
Talk Roadmap
Requirements for handling uncertain input
Extending toolkits to handle it Interaction techniques for
ambiguity Implementation
40
Invoking Application Actions Action often done by callbacks
– Direct procedure call to application
Hierarchical events are alternate approach– Delivered to app as well as toolkit
41
Hierarchical Events Low-level events contribute to
production of higher-level events
[Green TOG ‘86; Myers & Kosbie CHI ‘96]
User Inputcircle
stroke
down drag up• • • • • •
Corresponding Events
42
Implicit Assumption of Certainty Implicit in all this is the
assumption that the events really happened as reported
Problems arise when this isn’t true– E.g., brittle dialogs
43
Needed to Handle Uncertainty:
Allow for (and explicitly model) multiple alternatives– alternative higher level events– in recognition context: interpretations
Detect conflicting interpretations Mediation of conflicts
44
Needed to Handle Uncertainty:
Lexical feedback about uncertain events – split “feedback” from “action”
Library of mediators
45
How do we do this...
46
Extended Event Model
Uncertainty results in multiple interpretations
interpretation graph
Uncertain Input
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
Certain Input
47
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Now delivered to recognizers,
interactors, and application objects
Button
Checkbox Menu
Recog
48
Toolkit Extensions
Toolkit’s job is still to deliver events to objects– Objects initially only produce
(reversible) feedback, no actions
Button
Checkbox Menu
Recog
49
Another Change:
Interface Appearance Uncertain Event Hierarchy
circlebox
stroke
down drag up• • • • • •
Events dispatched to all who might use it
50
Details: Arranging for Mediation Identify any conflicts Look for a mediators
– Pluggable list of them in toolkit Mediator chosen by meta-
mediator Mediator can:
“Pass”, “Pause”, “Accept”
51
Doing Mediation
Example:User selects interpretation
circle
box
circle
52
Doing Mediation (cont.)
Mediator prunes interpretation graph to tree
– App informed of accept & reject
circlebox
stroke
down drag up• • • • • •
circle
stroke
down drag up• • • • • •
53
Mediation Strategies
Many mediation strategies– e.g., Automatic vs. user involvement
Toolkit is fully “pluggable” (!)– Library of mediators provided, but– Can extend/build new ones as needed
Research goal:Finding new ones
54
Providing a Library of Mediators
55
Providing a Library of Mediators
Survey of existing techniques [Abowd & Mankoff GVU Tech Report 99]
– Automatic – User Involvement
Repetition & repair strategiesChoice strategies
56
Automatic Mediation Techniques Probability modeling and
thresholding
Historical statistics
Rule-based
57
User Involvement: Repetition & Repair Strategies Undo and repeat
Change of modality
Partial repair
58
User Involvement:Choice Strategies Variations on N-best “lists”
– Presentation form– Instantiation time– Contextual information– Interaction – Feedback
Many techniques via “parameterization”– Ripe for toolkit support
59
Implementation
Added to subArctic toolkit– Reusable– Fully “pluggable”– Full existing library still works as is (!)
Small library of mediators Also working on non-GUI toolkit
60
Experience
Major example: Burlap– Smaller version of SILK [Landay]– For sketching
UI designs and turning them into functioning interfaces
61
Conclusions Reusable infrastructure to
support ambiguous input– Reduces difficulty of creating UIs– Easier to explore new design space
Done by modifying a toolkit, not a separate mechanism– Integrated with conventional input – Other support from toolkit still useful
62