Jörg Müller, Michael Nischt [email protected] Deutsche Telekom Laboratories, TU Berlin...

Jörg Müller, Michael [email protected]

Deutsche Telekom Laboratories, TU Berlin

Visualization and Interaction Techniques for Small Displays

Mobile and Physical Interaction, WS 2010/2011

Mobile and Physical Interaction 2WS 2010/2011Jörg Müller, Michael Nischt, T-Labs

Preview

• Showing content on small displays• Locating off-screen objects• Behind-the-device interaction• Improving touch screen accuracy

Showing Content on Small Displays


Web Pages Don’t fit on Small Screens


Web Pages Don’t fit on Small Screens

• Solution approaches– Device-specific authoring– Automatic re-authoring– Client-side navigation

• Tap to zoom into region– Uses HTML DOM model

• iPhone zooms into columns– Double tap


Overview + Detail

• Show details around cursor region

• But...– Detail region takes up

valuable screen space– Overview region not

readable– How to know what to

highlight?


Browsing Web Content

• Reorganize content in narrow vertical strip– Avoids horizontal scrolling

• 1-D browsing• Narrow layout

– Width = display width– Compact layout– Original layout

destroyed– Little overview


Context in Focus Display

• Nokia MiniMap– Overview is semi-transparently

shown in detail view while scrolling

– Content scaled but original layout preserved

• Even overview does not show whole page

Detail view

Indication ofdetail area

Overview


Nokia Symbian Browser

• Overview embedded in detail view • Semi transparent detail view• Site-specific adaptations


Android Browser


Collapse-to-Zoom

• Zoom into arbitrary rectangular areas• User’s knowledge about relevance of areas

relevant zoom in irrelevant collapse

Baudisch, Xie, Wang, Ma: Collapse-to-zoom: Viewing web pages on small screen devices by interactively removing irrelevant content. UIST '04.


collapse-cell

expand-cell

collapse-column

expand&zoom

expand

collapse

an

mai

n

com

ds m

Collapse-to-Zoom Gestures

• Similar to marking menus• Browser gestures


Thumbnail view: unreadably small

Collapse-to-Zoom Walkthrough


User collapses “archive” column



User collapses “menu” column



Content area expands moreand is now readable



User can follow links directlyfrom within overview



Expand-and-zoom gesture…




Leads to fully readable detail view


Collapse-to-Zoom

• Placeholders for collapsed areas• Bookmarking of collapsed state

Video


Summary Thumbnails

unreadablereadable

Lam, Baudisch: Summary Thumbnails: Readable Overviews For Small Screen Web Browsers. CHI 2005.

Video


Summary Thumbnails – Process HTML

• For each object on the web page– Count # of lines– Increase font size– Reduce text to preserve # of lines

• Text reduction strategies– Remove words from the end – Remove most commonly occurring

word (frequency dictionary)

reduce text to fit

scale font up

original page


Summary Thumbnails – Benefits

• No horizontal scrolling• Preservation of original page layout• Readable text• But: Missing text• Impact?• User study hypotheses

– Preserving layout is important to locate information

– Readable text reduces the need to zoom– Effects of text reduction are acceptable


Summary Thumbnails – User Study

• Design– Within subject

• Interfaces– Thumbnail– Summary thumbnail– Single-column– Desktop

• Dependent variables– Task time and accuracy– Amount of zooming– Amount of scrolling


Ready To Start

Task scenario

Task summary


Ready To Start

Thumbnail interface


Ready To Start

Summary thumbnail interface


Ready To Start

Single column

interface


Ready To Start

Desktop interface


Summary Thumbnails – Study Results

• Task times– Summary thumbnails significantly

different from single column

• Number of zooming events– Significantly different

0

5

10

15

20

25

Thumbnail Summary Th. Single Column Desktop

Me

an

ta

sk

tim

es

(s

)

0

2

4

6

8

10

12

14

Thumbnail Summary Th.

# of

zoo

min

g ev

ents


Summary Thumbnails – Study Results

• Error rates

• User preference

Thumbnail summary

Thumbnail

Single Column

# o

f p

arti

cip

ants

0

2

4

6

8

10

Thumbnail Summary Single Thumbnail Column

0

2

4

6

8

10

12

14

16

Thumbnail SummaryThumbnail

Single Column Desktop

% e

rro

r


DateLens (Bederson et al., 2004)

• Calendar with fisheye view and semantic zoom

Bederson, Clamage, Czerwinski, Robertson: DateLens: A Fisheye Calendar Interface for PDAs. ACM TOCHI, 2004.

Locating Off-Screen Objects


Halo (Baudisch & Rosenholtz, 2003)

Source: Patrick Baudisch

Baudisch, Rosenholtz: Halo: A Technique for Visualizing Off-Screen Locations. CHI 2003.


Streetlamp Metaphor

• Aura visible from distance• Aura is round• Overlapping auras aggregate• Fading of aura indicates distance




Arrow-Based Techniques

• Games, maps• Cinematography

– Partially out of the frame halos

• Requirements for “Partially out of the frame”

– Mentally fill in missing parts– Convey position in space


Shipley and Kellman 1992


Gestalt Laws: Perceptual Completion


arc/arrow fading off

same selectable size

halo ring distance from display border

legend


User Study: Halos vs. Arrows


click at expected location of off-screen targets Source: Patrick Baudisch

1. Locate Task


click arrow/arc or off-screen location closest to car Source: Patrick Baudisch

2. Closest Task



3. Traverse Task

click on all targets in order, so as to form the shortest delivery path, beginning at the car


click on hospital farthest away from traffic jams Source: Patrick Baudisch

4. Avoidance Task


Task Arrow interface Halo interface

Locate 20.1 (7.3) 16.8 (6.7)

Closest 9.9 (10.1) 6.6 (5.3)

Traverse 20.6 (14.1) 16.8 (8.7)

Avoid 9.2 (4.7) 7.7 (5.8)

0

5

10

15

20

25

Locate Closest Traverse Avoid

Arrow interface

Halo interface


Task Completion Time


Task Arrow interface Halo interface

Locate 23.5 pixels (21.6) 28.4 pixels (33.8)

Closest 22% (42%) 21% (41%)

Traverse 97.4 pixels (94.7) 81.0 pixels (96.7)

Avoid 15% (35%) 14% (34%)

0

5

10

15

20

25

30


Arrow interface

Halo interface


Error Rate

• Participants underestimated distances by 26%• Participants saw ovals• To compensate: width += 35%


0

1

2

3

4

5

6

7

8

9


Arrow interface

Halo interface


Subjective Preference


Limitation of Halo: Clutter

• Clutter from overlapping or large number of halos• Wedge: Isosceles triangles

– Legs point towards target– Rotation, aperture

• No overlap– Layout algorithm adapts

rotation and aperture

Gustafson, Baudisch, Gutwin, Irani: Wedge: Clutter-Free Visualization of Off-Screen Locations. CHI 2008.


The Wedge

• Degrees of freedom– Rotation– Intrusion– Aperture


Avoiding Overlap


User Study: Halos vs. Wedges

• Locate

• Avoid


User Study: Halos vs. Wedges

• Closest

• Subjective preferences

Behind-the-Device Interaction


LucidTouch

• Behind-the-device multitouch input• Pseudo transparency

– Enabling back of the device pointing– 3 states + visual feedback

= land-on selection

• Form-factor– Enabling multi-touch

with all ten fingers

Wigdor, Forlines, Baudisch, Barnwell, Shen: LucidTouch: A See-Through Mobile Device. UIST'07.

Video


Why Behind-the-Device Interaction?

• Avoid occlusion• “Fat finger” problem


Pseudo Transparency

• Show finger “shadows” as cues

physical see-through camera see-through


LucidTouch Camera See-Through

• Finger shapes and positions tracked by camera– Hovering state

• (Multi-)touch detected by pad– Tracking state


Out of Range Tracking Dragging

State0

State1

State2

LiftMouse

ReplaceMouse

DepressButton

ReleaseButton

Buxton’s Three-State Model of Input

Buxton: A Three-State Model of Graphical Input. In Proc. of INTERACT ’90.


PassiveTracking Dragging

State2

State0

Contact

ReleaseContact

Two-State Model for Touch Input


Out of Range Tracking Dragging

State0

State1

State2

Contact

ReleaseContact

FingerBehind Back

RemoveFinger

LucidTouch: Three Input States


LucidTouch Applications


Back-of-Device Interaction Works for Very Small Screens

• Jewelry, watches, etc.

• Pseudo transparency– Capacitive touch pad– Clickable touch pad

Baudisch, Chu: Back-of-Device Interaction Allows Creating Very Small Touch Devices. CHI'09.


Pressure-Sensitive Two-Sided Multitouch Interaction

• Metaphor– Holding an object between thumb and index finger

• Common in everyday interactions– Grabbing, sliding, twisting, turning

• Local input with high expressivity– Precise pressure– Thumb - index finger positions


iPhone Sandwich: Three Input States

Out of range

Tracking

Dragging

State0

State1

State2

Release

contact

Contact Increase pressure

Decrease

pressureState

2State

2State2+

Increase pressureIncrease pressure

Decrease pressure

Buxton: A Three-State Model of Graphical Input. In Proc. of INTERACT ’90.


Pressure-Sensitive Map Zooming

Essl, Rohs, Kratz: Squeezing the Sandwich: A Mobile Pressure-Sensitive Two-Sided Multi-Touch Prototype. Demo at UIST’09.


Virtual Trackballs for 3D Object Rotation x,y,z-axis rotation z-axis rotation

x,y,z-axis rotationx,y,z-axis rotation

θθθθ

z-axis rotationz-axis rotation


Extension of Virtual Trackball to Back of Device

x

y

z

o

• Full sphere operated from both sides instead of hemisphere operated from front


Gaussian Curve Avoids Discontinuity

Gaussian curve avoids discontinuity

Gaussian curve avoids discontinuity Extension to back side Extension to back side


Simultaneous Thumb and Index Finger

• Simultaneous interaction with thumb and index finger

– Shift center of virtual sphere

• Works for larger display sizes– Specific control widgets


User Study

Time: Nasa TLX:

front rear tilt front rear tilt

• Task: Count number of labels• Independent variables

– Method: front, rear, tilt– Number of labels (3±1, 9±1)– Scene complexity (2x2, 4x4)


Side-of-Device Interaction: SideSight

• Useful if device is placed on table• Distance sensors along device edges

– Multipoint interactions

• IR proximity sensors– Edge: 10x1 pixel “depth” image

Left and right “depth” images

Butler, Izadi, Hodges: SideSight: Multi-“touch” Interaction Around Small Devices. UIST’08.

Improving Touch Screen Accuracy



Small Displays Small Targets


Stylus vs. Direct Finger Input

• Stylus or pen– Grabbing stylus takes to

long for short interactions

• Bare finger input– Unclear contact point,

imprecise– Finger occludes target


Disadvantages of this “software stylus”

1. No visual feedback until contact, need to estimate offset

2. Makes some display areas unreachable

3. Unexpected offset affects walk-up-and-use scenariosSource: Patrick Baudisch

Offset Cursor (Potter et al., 1988)

Potter, Weldon, Shneiderman: Improving the accuracy of touch screens: an experimental evaluation of three strategies. CHI 1988.



• Shifting the whole screen is distracting• Disorients users, negatively impacts performance

Shifting the Whole Screen



Shift Callout (Vogel & Baudisch, 2007)

• Only shift callout• Enough context around target• 26mm circular shape occluded area under finger


no offset, click on the target itselfSource: Patrick Baudisch

Shift Needed Only for Small Targets


callout can go anywhere, no edge problemsSource: Patrick Baudisch

Corners and Edges


by default: dwell time (300 ms)

extension: larger target longer dwell time

extension: shift learns dwell times

vs.Source: Patrick Baudisch

When to Show the Callout

ST = target size

SF = finger occlusion size



Experiment: Shift vs. Offset vs. Touch

• Independent variables– 3 techniques: shift, offset, touch– 2 finger styles: nail, tip– 6 target sizes

(a) (b) (c)


Results: Error Rate

96482418126Square Target Size (px)

100

80

60

40

20

0

Mea

n E

rror R

ate

(%)

Touch Screen

Offset Cursor

Shift

Tip Nail



Results: Time



Shift + Zoom

• Increased callout diameter to compensate for less context


Escape: A Target Selection Technique Using Visually-cued Gestures

• Problem: Selecting a target on a touch-screen that is surrounded by other selectable objects

• Solution: The icons in “Escape” indicate directions. A finger tap followed by a movement in this direction enables disambiguation.

Yatani, Partridge, Newman: Escape: A Target Selection Technique Using Visually-cued Gestures, CHI 2008


Escape: A Target Selection Technique Using Visually-cued Gestures

• The icons in “Escape” indicate directions. A finger tap followed by a movement in this direction enables disambiguation.

• Can handle 2.3 icons per square centimeter• Find an assignment that separates gestures

– Similar to graph coloring NP-complete– “Escape” uses heuristic algorithm

http://www.youtube.com/watch?v=x3NeZswKkKw


Experiment: Escape compared to Shift

• Independent variables– Technique: Escape or Shift– Target size: 6, 12, 18, 24 pixels– Exposure (fraction of target visible): 0.25, 0.5, 0.75, 1

• Results– Escape

significantly faster than Shift

– No significant difference inerror rate


Experiment: Influence of Color and Beak Occlusion

• Independent variables– TargetSize: 6, 9, 12 pixels– Exposure: 0.25, 0.5, 0.75, 1– Direction: 8 directions– Color: gray or colored by dir.– Beak occlusion: yes, no

• Results– Significant effect for

beak occlusion– 1-colored icon selection

as fast as 8-colored icon selection


Escape: Advantages and Disadvantages

• Advantages– If target big enough, no

need to gesture– No need to be 100%

accurate– No need to wait for

something to appear

• Disadvantages– Each icon takes up more

space than original target– At screen edge limited

gestures directions

Jörg Müller, Michael Nischt [email protected] Deutsche Telekom Laboratories, TU Berlin...

Documents

Transcript of Jörg Müller, Michael Nischt [email protected] Deutsche Telekom Laboratories, TU Berlin...