Interaction: gestural input, popup widgets, interaction techniques, and

hardware input devices

Prof. Michael McGuffin

Gestural Input

• Gestures can be executed rapidly, and can be used as symbols to activate commands or select objects

• Sketches– can be used to quickly enter text or diagrams, without

requiring the user to switch from the pointing device to something else

– Have an informal, loose, implicit structure– Can carry much more (implicit) information than typed text

• Gestures as characters to recognize– Graffiti, Unistroke

Gestural Input

Kurtenbach and Buxton (video)• Example gestures: circle

objects, drag to move, or drag + draw a "C"  to copy

• Such a gesture selects the noun (objects) and verb (operation) in a single action, which may be more natural, fluid, or faster than doing selection + Ctrl-C/Ctrl-X + Ctrl-V

Gestural Input

"Teddy" (Igarashi et al. 1999).

See Java applet at Takeo Igarashi’s website.

Phun (≈2008)• A free game/physical simulator that uses gestural input• http://www.acc.umu.se/~emilk/

Phun

• Alvarado and Davis 2001• http://www.youtube.com/watch?v=NZNTgglPbUA

• LaViola 2007 http://doi.acm.org/10.1145/1281500.1281558

Observation

• Most user interfaces that use gestural input require the user to first learn the gestures before they can be used. This can be a significant obstacle for novice users.

• In the following slides, we will see a technique called Marking Menus that eliminates this problem.

Modes, and some examples of interaction techniques using gestural

input or other input

Interaction Techniques

• “are ways to use input devices to enter information”

• "are made up of single input-device actions.” (Foley, et al. “Computer Graphics: Principles and

Practice”)

Verbs(actions,commands,tools,operations) Nouns

(objects,locations)

Verbs in a pulldown menu

Nouns(objects,locations)

Nouns(objects,locations)

Verbs ina popup menu

Summary• Modes can lead to mode errors, where the user thinks

they are in one mode but are really in another• Visual feedback (example: changing cursor shape) to

indicate the current mode is good, but often not enough to avoid mode errors

• Popup menus help…– Avoid mode errors

(via transient modes, and kinaesthetic feedback where the user feels pressure through their finger from holding down a button)

– Increase screen space devoted to content, because the menu is only displayed when desired

– Decrease the distance travelled by the cursor, because there is no back-and-forth motion between the workspace and peripheral interface elements

– Create a tighter integration, or fusion, of the selection of noun and verb, leading to better mental chunking (Buxton 1986)

Given that popup menus have all these advantages, can we design

other widgets that are even better?

Radial Menu (or “Pie Menu”)

Demo: Radial Menu in SimplePaint

An interface that uses, effectively, radial menus

Yatani et al., CHI 2008

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

• Directions are easier to rememberand reproduce than distances

Hierarchical Radial Menu

"Mouse Gestures" for Firefox

Marking Menu

• “Scale invariant recognition”: The recognition of the gestures (marks) does not depend on the length of the segments; only the angles matter. Hence, the marks can be drawn small and quickly, or ballistically.

• A user who knows what mark to draw doesn’t even need to see the menu popup.

The set of gestures is discoverable and “self-revealing”, contrary to other gestural interfaces

Improved graphical presentation

Marking Menus

• Video/Demonstration

Marking Menus

• Video/Demonstration

Marking Menus

• Video/Demonstration

Transition from novice to expert

Traditional menus:Pointing versus shortcuts

Marking Menus:Transition is gradual and natural !

Summary

Marking Menus• Can be operated faster than linear menus

(ballistic, directional, scale-invariant marks)• Have a self-revealing gesture set• Enable a smooth transition

from novice use to expert use• Can be used to select both a noun and verb• Are limited to 8 items per submenu,

and limited to a depth of about 3 levels

Marking Menus• Allow for faster selection than with linear

menus(directional, ballistic movements)

• Can be used without looking at the screen(“eyes-free operation”)

• Have a discoverable set of gestures• Allow for a gradual and natural transition

from novice to expert• Can be used to select a noun and verb• Are limited to approximately 8 items per

submenu, and a depth of about 3 levels

Summary• Temporary modes, held by holding down a button or key with kinesthetic

feedback …– Help avoid mode errors– Example: a key that must be held down to maintain a temporary mode

change, such as the Shift key on keyboards– Example: popup widgets

• Help combine noun+verb selection• Only require screen space when in use

• One popup widget with many advantages: Marking Menus– Allow for ballistic gestures– Allows users to gradually transition from novice to expert

Other examplesof modes, tools, etc.

• video: "Selection and Positioning tasks", Bill Buxton 1983

Some observations• Techniques shown in the video:

– 1. Drag-and-drop– 2. Modal toolbar– 3. « Moving menu » (a kind of popup menu)– 4. « Moving menu » with memory

• Questions: of the 4 techniques, which ones …– Are not modal (or only have a temporary mode with kinesthetic feedback), and so

help to avoid mode errors ? (Answer: 1 and 3)

– Avoid back-and-forth movements between the workspace and the toolbar, and do not have screenspace consumed by a toolbar ? (Answer: 3 and 4)

– Have visible affordances (toolbar), therefore showing the possible operations to a novice user ? (Answer: 1 and 2)

– Allow the same shape to be created many times repeatedly, very quickly ?(Answer: 2 and 4)

– Ensure that each shape is always selected with the same dragging gesture, allowing the user to learn the gestures by heart and then execute them rapidly ? (Answer: 1 and especially 3.)

Can we extend popup menus/widgetsfor other uses?

Other popup menus and widgets

• Hotbox (Kurtenbach et al., 1999),Control Menus (Pook et al., 2000),Flow Menus (Guimbretière et Winograd, 2000),FaST Sliders (McGuffin et al., 2002),Tracking Menus (Fitzmaurice et al., 2003),Trailing Widget (Forlines et al., 2006)Hover Widgets (Grossman et al., 2006),PieCursor (Fitzmaurice et al., 2008)

• These widgets and interaction techniques are designed for:– A large number of commands– Controlling continuous variables– Input of text and numbers with gestures– Use with a stylus (pen), for example, on a “tablet PC”

FlowMenus (Guimbretière et al., 2000)

2D manipulation with FlowMenus(Guimbretière et al., 2000)

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

Demo of a more recent kind of Hotbox (McGuffin and Jurisica 2009) used for manipulating a network visualization

Tracking Menu (copy of video athttp://www.dgp.toronto.edu/~gf/videos/tm.avi

at George Fitzmaurice’s website)

Hover Widgets (copy of video at http://www.dgp.toronto.edu/~ravin/videos/chi2006_hoverwidgets.mov

at Ravin Balakrishnan’s website)

Toolglass: bimanual input

Click-through: Simultaneousselection of verb and noun!

Bimanual 3D Navigation

Erick Velazquez-Godinez and Michael McGuffin (unpublished, 2010)http://www.youtube.com/watch?v=ggoLLnxHwhAHaving the 3D position of each hand is enough to control the 6 degrees-of-freedom of a 3D camera: translate hands together to translate the camera, rotate hands around each other to rotate the camera, move hands toward or away from each other to zoom.

Tracking 2 or more fingers with a Nintendo Wiimote

• See online videos by Johnny Chung Lee http://johnnylee.net/projects/wii/

Other uses of two hands?

Two-Handed (Bimanual) InputPotential uses:

• Dominant hand (DH) on mouse,non-dominant hand (NDH) on keyboard

• Two mice, two cursors, symmetrical:– Rapid clicking by alternating between hands?– Simultaneous rotation+scaling+positioning in 2D

or rotation+zooming+panning in 2D

• Two mice, asymmetrical:– NDH for camera, DH for selection/manipulation– NDH for tool palette, DH for clicking-through

(Toolglass)

Kinematic Chain Model(Yves Guiard 1987)

• NDH (non-dominant hand) performs slower and less precise movements than the DH (dominant hand)

• The NDH moves before the DH

• The NDH establishes a frame of reference within which the DH performs work

Input Devices

Keyboards

http://world.std.com/~jdostale/kbd/SpaceCadet1.jpeg

Foldable keyboard for Palm Pilot Flexible keyboard; 24 $ at http://www.thinkgeek.com/computing/input/5a7f/?

cpg=ab(price from 2008)

Projected Keyboard

http://en.wikipedia.org/wiki/Projection_keyboard

160$ at http://www.thinkgeek.com/computing/input/8193/(price of 2008)

Optimus

http://computer.howstuffworks.com/keyboard.htm/printable

Optimus Maximus

Each of the 113 keys contain a 48x48 pixel screen.

1864 $ US ! (price of 2008)

http://www.artlebedev.com/everything/optimus/

Optimus Maximus

Configured for English

Optimus Maximus

Configured for Russian

Optimus Maximus

Configured for the game "Half-life"

Optimus Maximus

Configured for "Photoshop"

Bottons on a tape recorder

The status quo:

Uniform buttons

Buttons with different forms: allow use without looking (“eyes-free operation”)

Piano Keyboard

naïve design

modern design

Apple iPhone Keyboard

Scholes Keyboard versus Dvorak Keyboard

Designed in 1873 !

Still the standard :-(

Still not in widespread use

Notice that all vowels are under one hand in the midlde row.

Comparison of QWERTY and Dvorak

http://infohost.nmt.edu/~shipman/ergo/parkinson.html

“Half Keyboard” from Matias Corp.

http://www.halfkeyboard.com/

« Septambic keyer »or « chorded keyboard »

4 x 24 – 1 = 63 possible "chords"

Pointing Devices

The first mouse

• 1968• Douglas Engelbart• Stanford Research Institute• Two rollers for x and y

40+ years later

Touchpad

Rotation sensing

Rockin’ Mouse

Haptic feedback(or “force feedback”)

Other kinds of 2D pointing devices

• Digitizing tablet / graphics tablet with a stylus (pen) and/or puck (mouse)

• Touchscreen• Light pen• Touchpad• Joystick

– isometric (rigid, senses pressure, for example: Trackpoint)– elastic (like isotonic, but returns to center when released)– isotonic (can be moved freely)

• Trackball• Eye tracker

TrackPoint (isometric joystick)

Touchpad

Logitech Trackball

Graphics tablets

Wacom Bamboo5.8x3.7 inches; 80$

Wacom Cintiq 21UXwith integrated screen21.3 inches (17x12.75); 2500$

Wacom Intuos312x19 inches; 750$

(prices of 2008)

Input devices for tablets

Styluses can have• a pressure sensor on their tip• a button on the tip to click• a button on the side ("barrel button") • a scroll wheel that can be rotated

IntuPaint (Vandoren et al. 2008)and FluidPaint (Vandoren et al. 2009)

Properties of Pointing Devices

• Absolute vs relative sensing– Which is more general ?

• Direct vs indirect– Which is more “intuitive” ?– Which is more predictable ?– Which is more relaxing ?– Which leaves visual feedback more visible ?– Which do artists prefer ?

• Discrete vs continuous– Which is more general ?

• Position control vs velocity control– Which is more general ?– Which typically requires less movement/smaller footprint ?

Velocity control:

Velocity control pointing devices, for 3D

SpaceballSpacemouse

Spaceball 5000

Magellan SpaceNavigator (60$)3dconnexion.com(in 2008)

Other products from 3dconnexion.com(in 2008)

Taxonomy of input devices (Buxton)M: Mechanical intermediaryT: Touch

3-State Model (Buxton, 1990)

State 0: no (x,y) coordinatesStates 1 and 2: (x,y) position is known

Examples:• Graphics tablet: states 0, 1, 2• Mouse: states 1, 2• Touchscreen: states 0, 1

TouchMouse(Hinckley and Sinclair 1999)

States 0, 1, 2

TouchMouse(Hinckley and Sinclair 1999)

“Pop Through” Buttons(Zeleznik et al. 2001, http://doi.acm.org/10.1145/502348.502384 )

PreSence (Rekimoto et al. 2003)

Haptic Pen

Lee et al., "Haptic pen: a tactile feedback stylus for touch screens", UIST 2004

Handheld Projector and Pen

Cao et Balakrishnan, "Interacting with Dynamically Defined Information Spaces using a Handheld Projector and a Pen", UIST 2006

PenLight

Song et al., "PenLight: combining a mobile projector and a digital pen for dynamic visual overlay", CHI 2009