interacting with large screens

of 119 /119
interacting with large screens patrick baudisch microsoft research visualization and interaction research

description

interacting with large screens. patrick baudisch microsoft research visualization and interaction research. large screens. building a large display focus plus context screens interacting using the mouse high-density cursor mouse ether interacting using pen & touch drag-and-pop - PowerPoint PPT Presentation

Transcript of interacting with large screens

Page 1: interacting with large screens

interacting with

large screens

patrick baudisch

microsoft researchvisualization and interaction research

Page 2: interacting with large screens

large screensbuilding a large display

focus plus context screens

interacting using the mouse

high-density cursormouse ether

interacting using pen & touch

drag-and-poptablecloth

(halo) return

Page 3: interacting with large screens

large screens and multimon

Page 4: interacting with large screens

... are coming

[Jon Peddie ResearchDec, 2002 N=6652]

No Multimon30%

Plan to Use Multimon

38%

Use Multimon32%

• information mural[Guimbretière, Winograd]

• on large screens optical flow helps navigation [Tan 2001]

• large screens help productivity tasks [Czerwinski 2003]

• multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]

Page 5: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

focus plus context screens

conclusions

Page 6: interacting with large screens

• Hardware– At least one hi-res

display– At least one larger low-

res display

• Software– scaling of the display

content is preserved– resolution varies

Page 7: interacting with large screens

skip to dnp

Page 8: interacting with large screens

setup

Page 9: interacting with large screens

application scenarios

video

Page 10: interacting with large screens

field study: users & tasksSubject’s task Document/view

Sta

tic do

cum

en

ts

Web designer Page: 800 pixel

Mechanical engineer Polybot segment: 5cm

Graphic designer Poster: 1m

Architect in remodeling Building: 50m

Photogrammetry (2) Highway 2 miles

Geographic info. system County: 80km

Chip designers (2) Wafer: 12cm

Dyn

am

ic

Air traffic ctrl. tool builder Zone: 50km

Ego shooter gamer Surrounding: 360º

Submarine ROV op. Surrounding: 360º

Strategy gamers (2) Map: 30k pixel

Smallest detail Ratio

Table detail: 1 pixel 800

Clearance: 0.03mm 2,000

Align: 0.5mm 2,000

Accuracy: 1cm 5,000

Accuracy: 1 inch 100,000

Land boundaries: 0.5m 160,000

Grid: 0.5m 240,000

Plane distance in 25m steps 2,000

Aiming: 0.1º 3,000

Use arms: 1mm/0.05º 8,000

Aiming: 1 pixel 30,000

Page 11: interacting with large screens

focus plus context screen VisualizationSame # of pixels

fisheye

55

overview plus detail

44

Display technology

homogeneous resolution

44wall-size, hi-res display

44

What participants

used

What participants

used

Available to½ of participants

Available to½ of participants

… andcurrentsolutions

Page 12: interacting with large screens

experiment 1:• 3 interfaces:

• focus plus context screen• overview + detail• homogeneous

• 2 tasks

• 12 subjects from Xerox PARC• Within subjects, counter-balanced• Same number of pixels

Page 13: interacting with large screens

task 1: closest hotel

8 maps per interface

F+C screen and O+D use same magnification factor

Page 14: interacting with large screens

task 2: verify connections

Verify a different set of 24 connections on the board

Page 15: interacting with large screens

0

100

200

300

400

500

600

700

Map task Board task

zooming + panning

overview+detail

focus+context screen

resultsA

vera

ge

ta

sk c

om

ple

tion

tim

es

in s

eco

nd

s

21% faster21% faster 36% faster36% faster

manually zooming

takes time

manually zooming

takes time

visually switching reorientation

visually switching reorientation

visually more ambiguous

visually more ambiguous

Page 16: interacting with large screens

experiment 2:driving simulation

120 sec sequence100 fields of nails; 30 rocks; tradeoff

Page 17: interacting with large screens

resultsM

ea

n n

um

be

r o

f co

llisi

on

s su

bje

cts

cau

sed

• Sweet spot:flight simulation, unmanned vehicles…

0

5

10

15

20

25

run-over nails rocks hit

overview+detailfocus+context screen

Error rate only 1/3 of two-

monitor setup

Error rate only 1/3 of two-

monitor setup

Subjects preferred thef+c interface

Subjects preferred thef+c interface

Page 18: interacting with large screens

observations

• low-res periphery is often ok

• on a wall-size display: If periphery was hi-res I still could not read it due to distance

focus-plus-context interaction

toc

Page 19: interacting with large screens

• mouse • pointer moves too fast, users lose track of it• pointer behaves in unpredicted ways when

crossing screen boundaries

• touch/pen input• users cannot reach their content anymore

ktop?ktop?we observed: lots of input problems

conclude

Page 20: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

high density cursor

conclusions

Page 21: interacting with large screens

problem: mouse across long distances

• large screen longer distances

higher mouse acceleration

• temporal aliasing: 500 pixels jumps

• lack of visual continuity & weak stimulus users lose track of the pointer

Page 22: interacting with large screens

the problem will get worse

• cursor update is limited by screen refresh rate

• screen refresh rate has actually decreased (LCDs)• larger screens + lower refresh rate status quo• future: even larger screens problem will get worse

Page 23: interacting with large screens

how it works

previous cursorposition

current cursorposition

mouse

motion fill-in cursors

current framefill-in cursors

previous frame

inserts cursor image between actual cursor positions the mouse cursor appear more continuous

Page 24: interacting with large screens

this is not the mouse trailthe windows

mouse trail…• makes mouse

trail last longer• drawback: cursor

images lag behind

...is not high-density cursor

• hd cursor makes mouse trail denser

• lag-free: mouse stops => cursor stops

video

Page 25: interacting with large screens

benefitsprevious cursor position

mouse

motion

current cursor position

fill-in cursors

current framefill-in cursors

previous frame

mouse

motion

1. mouse cursor appear more continuous easier to track the cursor

2. higher “visual weight” easier to re-acquire the cursor

Page 26: interacting with large screens

designs alternatives

acceleration

• reference: exponential acceleration

a

b

d

e

f

c

h

g

frame

Page 27: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• motion blur with higher weight

Page 28: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frame acceleration

• temporal super-sampling vs. motion blur

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

chose discreet version1. latest cursor position is always shown blur-free and in full opacity2. appearance that users are familiar with today3. computationally less expensive

Page 29: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• density = detectability vs. intrusiveness

Page 30: interacting with large screens

transfer function

mouse speed

distancebetweencursorimages

onset threshold(configurable)

cursor trail provides no speed cues

hd c

urso

r has

no

effe

ct

transfer function(configurable)

Page 31: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• optional cursor growth

Page 32: interacting with large screens

user study• conducted pre-study to define interface candidates• interfaces: control vs.

high-density cursor (conservative, tripleDensity, plusScaling) • fitts’ law task• triple-mon: button located at 5” to 40” distance• participants: 7 external participants, 5 coworkers

• hypotheses• high-density cursor faster• the greater the distance the

greater the effect• tripleDensity and plusScaling

faster than conservative

Page 33: interacting with large screens

resultsti

me %

rela

tive t

o r

eg

ula

r cu

rsor

90

92

94

96

98

100

102

target distance (mm)125 250 500 750 1000

regular mouse cursor

high-density cursors

speedupup to 7%

conservative

+ scale

+3-dense

short distance

Page 34: interacting with large screens

subjective satisfaction

Condition Liked Most Liked Least

control 0 7

HD_conservative 2 3

HD_tripleDensity 2 0

HD_plusScaling 6 0

• most participants did not notice that cursor was different!“did that condition use a different mouse acceleration?”…

Page 35: interacting with large screens

• lesson we learned:display frame rate is not a hard limit

toc

Page 36: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

mouse ether

conclusions

Page 37: interacting with large screens

mouse ether

target

start

Page 38: interacting with large screens

tasks with incorrecttasks with incorrectvisual feedbackvisual feedback

Page 39: interacting with large screens

mouse across multimonproblem 1: different dpi

targetstart

low dpihigh dpi

Page 40: interacting with large screens

mouse across multimonproblem 2: the gap

target

start

Page 41: interacting with large screens

contents

• problem: why the cursor gets warped

• eliminating warping with mouse ether

• calibration

• user study: up to 28% faster in acquisition task

• conclusions: need to test fitts’ law

Page 42: interacting with large screens

why the cursor gets warped

targetstart

low dpihigh dpi

target

start

system oblivious of• dpi differences

system oblivious of• gaps

Page 43: interacting with large screens

mouse ether

target

start

targetstart

low dpihigh dpi

target

start

targetstart

low dpihigh dpi

with mouse ether

now users can acquire targets on direct path

now users can acquire targets on direct path

Page 44: interacting with large screens

calibration

step 1step 2step 3

Page 45: interacting with large screens

the math

screen i

screen j

(cx,cy)

(ex,ey,)

on in

it

+(dx,dy)every frame

(cx,cy)

(ex,ey,)’ ’

’ ’

Page 46: interacting with large screens

off-screen travel

target

start

with mouse ether

Page 47: interacting with large screens

more off-screen travel

target

start

blocked

same dpi

user’s perspective system’sperspective

target

blocked

high dpilow dpi

start

start

target

target

start

same dpi

with mouse ether

target

high dpilow dpi

start

start

target

name ether

Page 48: interacting with large screens

taming off-screen travel

• iteration 1:restrict to convex hull

• iteration 2:allow entering ether only for the purpose of transit• towards another screen• at sufficient speed

• otherwise screen boundaries “hard”

start

target

Page 49: interacting with large screens

user study

• 2x 18” screens• 1280x1024 and

800x600 resolution• eight participants• WinFitts software• 72 movements

(=4 trials x 9 paths x 2 directions)

Page 50: interacting with large screens

results

High-Low Low-High0

200400600

8001000

120014001600

1800

Mov

emen

t Tim

e (m

s ±S

EM

)

Direction of Movement

Control

MouseEther

with mouse ether up to 28% faster

with mouse ether up to 28% faster

Page 51: interacting with large screens

discussion

• multimon offers incorrect visual cues• this hinders successful target acquisition

• mouse ether improves acquisition speed by matching pointer behavior to visual cues

• off-screen travel allows for direct target acquisition

• next steps: investigate target acquisition under contradictory visual feedback ( Fitts’ law?)

toc

target

start

Page 52: interacting with large screens

pen/touch input on walls-size displays

• mouse input can be mapped to any screen geometry

• not so for “direct” input technique, such as touch and pen input

Page 53: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

drag-and-pop

conclusions

Page 54: interacting with large screens

a technique for people with limited reach…

• user may not be able to reach content

• user may not be able to read content

Page 55: interacting with large screens

drag-and-pop…

• let’s focus on a specific case for a moment: extend basic actions drag-and-drop and picking

Page 56: interacting with large screens

scenario 1: long distances

dragging is designed for small screens…… but becomes time-consuming on large screens

Page 57: interacting with large screens

scenario 2: dragging + bezels

dragging across bezels in display wall is no problem for the mouse…

…but a big problem when using pen/touch input

Page 58: interacting with large screens

drag-and-pop: demo• users starts

dragging icon towards a distant folder or application

• icons of compatible type come towards mouse cursor

• user drops icon with minimal motion

• targets retractdrag-and-pop works across bezels

Page 59: interacting with large screens

• drag-and-pop generalizes direct manipulation• bring content to the user• let the user interact with it• send content back

Page 60: interacting with large screens

scenario 1: long distances

dragging on large screens

Page 61: interacting with large screens

scenario 2: dragging + bezelsdragging acrossbezels in display wall

Page 62: interacting with large screens

the displays we used…

Page 63: interacting with large screens

design

Page 64: interacting with large screens

selecting candidates

• initialize• all icons are candidates

• filter• eliminate icons with non-matching file types• eliminate icons that are too close• eliminate icons outside target angle• if necessary, restrict to some hard limit

Page 65: interacting with large screens

preserving layout

• snap to grid• eliminate empty

rows and columns

• translate back• place center of

bounding box in front of user

• closer for experts

Page 66: interacting with large screens

the rubber band

• animationdid not work

• “frozen”motion blur

• narrow midriff• suggests elasticity• clue for distance

• simplified version

Page 67: interacting with large screens

getting it out of the way

• to rearrange icons on the desktop (overloaded):• any mouse motion moving away from the

“popped-up” icons de-activates drag-and-pop• introduce flick gesture into mouse motion

Page 68: interacting with large screens

pre-study

• 3 layouts for study: sparse (11), frame (28), cluttered (35)

• 15 single, 6 dual, and4 triple monitor users

• overall resolutions 800,000 pixels to 3,900,000 pixels

• (= 66% more than the display wall used in the experiment).

Page 69: interacting with large screens

user study

• participants: 2 female, 5 male• dynaWall

• 3 Smartboard• 15’ long (4.5m)• 3 x 1024x768 pixels

• native code not stable enough Macromedia Flash version

• task: drag icons into matching folder• highlighting disappeared when started• each desktop: 11-35 icons + 10 icons to be filed

Page 70: interacting with large screens

results

• faster with drag-and-pop• error rate higher with drag-and-pop• most of the effect caused by the bezels

0 1 20

2

4

6

8

10

12

14

Number of Bezels Crossed

Drop

Pop

Control

Drag-and-pop

3.7 times3.7 timesspeedupspeedup

0

10

20

30

40

0 500 1000 1500 2000 2500Target Distance (pixels)

Drop

Pop

Control

Drag-and-pop

Page 71: interacting with large screens

subjective satisfaction

• > 6 (out of 7)• “I liked using drag-and-pop”• “I always understood what was happening when drag-and-pop

was on”,• “I would use drag-and-pop for large displays.”

• < 3 for• “It took a long time to get used to drag-and-pop”• “It was hard to control what the targets did when drag-and-pop

was on.”

• drag-and-pop interface causes less manual stress and fatigue than the control interface

• lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)

Page 72: interacting with large screens

drag-and-pick

problem• launch app or open file

drag-and-pick• user drags “background”• all icons in that direction

move to the cursor• user drags % releases

mouse over it• target is activated

Page 73: interacting with large screens

current work: push-and-pop

Page 74: interacting with large screens

outperforms pantograph-based techniques

0

1

2

3

4

5

6

7

8

9

10

11

12

198 455 480 504 920 1075 1123 1173 1651 1695 1850 1897Distance (pixels)

Tim

e (

s)

drag & drop

push & throw

pick & drop

acc. push & throw

drag & pop

push & pop

Page 75: interacting with large screens

0

1

2

3

4

5

6

7

8

9

10

260 586 678 712 1363 1592 1624 1660 2476 2505 2734 2766

Distance (pixels)

Tim

e (s

)

drag & drop acc. push & throw

push & throw pick & drop

drag & pop push & pop

insert halo

outperforms pantograph-based techniques

Page 76: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

tablecloth

conclusions

Page 77: interacting with large screens

tableclothgeneralizing drag-and-pop to arbitrary interactions

Page 78: interacting with large screens

for users withouttable manners

Page 79: interacting with large screens
Page 80: interacting with large screens

object I would like to interact with

window panning endswhen window edgereaches edge of screen

tablecloth panningallows panning any screenobject to any screen location

Page 81: interacting with large screens

design challenge

• how to invoke tablecloth?• while dragging an object

Page 82: interacting with large screens

1st design: crossing menu

Page 83: interacting with large screens

dragging

2nd design: dual touch

jumpback

panning

pan

jump

dragging

video

Page 84: interacting with large screens

focus-plus-context interaction

the distinction of screen space into focusand context regions is always there(focus plus context screens only emphasize it)

• focus-plus-context interaction• bring contents to user• interact• send contents back

toc

halo

Page 85: interacting with large screens

discussion

• distance to peripheral space and human peripheral vision make peripheral content hard to read

• making periphery all hi-res does not solve this problem

• we need to support users using appropriate interaction techniques

Page 86: interacting with large screens

building one

mouse acrossbezels

reach icons

reach anything

mouse acrossdistances

read more

UIST 2001CHI 2002

INTERACT2003

CG&A 2005

INTERACT 2003GI 2005

CHI 2004

patrickbaudisch.com/projects

Page 87: interacting with large screens

thanks!• try it out www.patrickbaudisch.com

• thanks to: • focus plus context screens: victoria

bellotti, nathan good, paul stewart, pamela schraedley

• high-density cursor: ed cutrell, george robertson

• mouse ether: ed cutrell, ken hinckley, rob gruen

• drag-and-pop: & ed cutrell, dan robbins, peter tandler, ben bederson…

• tablecloth: george robertson

Page 88: interacting with large screens
Page 89: interacting with large screens

can we push this further?

• (unless the app has fixed focus of attention) bring content to the user on demand

• that’s great—so maybe the periphery does not need to be full resolution after all

• can we push this even further? can we create a visual periphery without any peripheral screen space?

Page 90: interacting with large screens

halo

store content in periphery

Page 91: interacting with large screens

<view halo section is the small screen deck>

Page 92: interacting with large screens

access off-screen content

• make halo arcs (or city lights) clickable to pan display to that location

vs.

• preserve users’ spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen

Page 93: interacting with large screens

Seamless integration of displays

a b

Page 94: interacting with large screens

Context

No reflections on focus screen

Focus

Page 95: interacting with large screens

If I had to commercialize today…

• Build all-analog immersive video link• Immersive telepresence,• Remote operated vehicles, drones• Immersive VR• Remote medicine

Page 96: interacting with large screens

ContextWall

Page 97: interacting with large screens

Extra

Page 98: interacting with large screens

(a) locate (b) closest

(d) avoid(c) traverse

Page 99: interacting with large screens

How does it work?

Page 100: interacting with large screens

The scaling software

• Display image on two display units of different resolution• Similar to two-headed display• but display units are overlapping• and one of them has to be scaled down

• (Related work “Flux capacitor”[Dr. Emmett Brown, 1985])

Page 101: interacting with large screens

Linux/VNC

app

focus

contextinput

server

clip

scale

viewer

viewer

Page 102: interacting with large screens

Image viewer

context

Photoshop

scale

ACDsee

ACDsee

.gif

.gif

mousefork

input

focus

Page 103: interacting with large screens

related work• acquiring distant targets

• move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999)• flick snaps cursor to target (Dulberg et al. (1999)• sticky icons capture cursor (Swaminathan and Sato, 1997)• throwing gets across long distances (Geißler, 1998) • expanding targets save space on screen (McGuffin and Balakrishnan, 2002)• drag-and-pop (baudisch et al 2003)

• enhance detectability of the mouse cursor• <ctrl> for radar animation (Microsoft, Steve Bathiche)• cursor growth (Kensington Mouseworks 2001)• mouse trail for slow response LCDs (e.g. MS Windows)• liveCursor points in the direction of its motion (Ben Bederson)

• motion blur and temporal supersampling• reduce temporal aliasing, such as stroboscope, e.g. wheel spokes• rendering a scene multiple times (Dachille and Kaufman, 2000)• improve the perceived responsiveness of graphics apps (Conner and Holden, 1997)• help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)

Page 104: interacting with large screens

design goals

• for users who track the cursorenhance the predictability of the cursor path• enhanced trail density/continuous blur• smooth interpolation of the cursor path• preservation of trail density as a cue for cursor speed.

• for users who reacquire the cursorincrease the detectability of the cursor (visual weight)• enhanced trail density• enhanced cursor opacity• and cursor scaling.

• preserve responsiveness

Page 105: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• distance between cursor images as cue for mouse speed

Page 106: interacting with large screens

designs alternativesa

b

d

e

f

c

h

g

frameacceleration

• smooth interpolation

Page 107: interacting with large screens

bezier interpolation

cursor position

1. linear interpolation

2. attraction point

3. interpolate

Page 108: interacting with large screens

pre-study

• goal: define interfaces for user study• participants: 14 coworkers

• informal procedure• try out high-density cursor• try out different settings (density, onset…)• choose “favorite” setting

• resulting interface parameters• 12-17 pixels/frame vs. 35 pixels/frame• distance = sqrt(n)• cursor growth on or off

Page 109: interacting with large screens

touch/pen input breaks

touch/pen-input + multimon

touch and pen input renaissance• PDAs• Tablets• Liveboards /

Smartboards

multi-display systems• DynaWall, iRoom

Smartboard wall• connect tablet to

external screen• …

Page 110: interacting with large screens

scenario 1: tables + screen

tablet users scribble with pen… but filing iconsinto folder on external monitor requires mouse

Page 111: interacting with large screens

scenario 1: tablet + screen

filing icons into folder on external monitor

Page 112: interacting with large screens

related work• techniques for transferring information

• drag-and-drop avoids hidden clipboard (e.g. Xerox Star)• hyper dragging (Rekimoto, 1999)• pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001)

• overcome large distances• magic pointing (Zhai et al., 1999) requires an indirect input device• gesture input techniques (Rubine, 1991)• throwing (Geißler, 1998) for reliable target acquisition?• laser pointers to acquire targets on a Smartboard (Myers et al. 2002)

• mouse-based interaction techniques• lodestones and lay lines (Jul, 2002)• flick (Dulberg et al., 1999)• sticky icons (Swaminathan and Sato 1997)

Page 113: interacting with large screens

related work

• driving directionsvs. route planning aids

• overview-plus-detail• focus-plus-context

• pointing into off-screen space

Page 114: interacting with large screens

inside applications…

• drag-and-pop workseven if target is• occluded• clipped• closed (folder)

• use the concept tofile emails?

Page 115: interacting with large screens

intrusion border

handle

space for arcs…

and for corner arcs

reserve space for content

Page 116: interacting with large screens

arc length = distance

Page 117: interacting with large screens

handling many objects

• find best (restaurant): relevance cut-off

• see all (dangers): merge arcs

Page 118: interacting with large screens
Page 119: interacting with large screens

a first attempt… city lights